Setup questions?


#21

Following the install instructions (on Ubuntu 18.04.1) I was able to get to the point where systemd service vespene was running but when browsing to port 8000 I was only getting

This page isn’t working 192.168.0.68 didn’t send any data.
ERR_EMPTY_RESPONSE

Adding --bind 0.0.0.0:8000 to supervisord.conf fixed that for me! So thanks @psadm!


#22

Should add a note to whitelist 8000/tcp in firewalls as well I think.


#23

Hi. I'm interested in connecting Vespene to a LDAP server but after reading the documentation I'm confused on how I'd go about doing that. Can anyone offer any pointers?


#24

Hi @dncrash

Ok, so I'll be completely honest that I haven't tried that yet, and I am likely to not try this in the immediate future, but here's the basic instructions for converting any Django app to LDAP below.

My philosophy with Vespene is to not reimplement extra features that Django does stock, so there is nothing Vespene specific to swap out the authentication per se - which is GOOD in this case. The instructions for LDAP'ing any Django app should be 100% the same here.

It's going to need this installed:

https://django-auth-ldap.readthedocs.io/en/latest/

Here are the configuration instructions:

https://django-auth-ldap.readthedocs.io/en/latest/install.html

You should pay special attention to how you can (and should) tell LDAP how to decide who is a superuser per something like this:

AUTH_LDAP_USER_FLAGS_BY_GROUP = {
    "is_active": "cn=active,ou=groups,dc=example,dc=com",
    "is_staff": "cn=staff,ou=groups,dc=example,dc=com",
    "is_superuser": "cn=superuser,ou=groups,dc=example,dc=com"
}

This way not every LDAP user can log in and superusers are defined from LDAP. Read the existing Vespene security guide about who should have superuser access - I would be very inclined to make an extra LDAP user group for Vespene admins than giving too many people in the organization admin access.

If you're reasonably comfortable with Python (and especially Django configuration) this should "just work", but I will say getting the parameters right for LDAP is sometimes (often!) a frustrating experience. The cli command 'ldapsearch' can help make sure the queries are right.

I'd be very interested in your experiences getting this working! Can you perhaps start a new thread and link to the thread here?

Once you have a working configuration this would be an excellent example to add the sidebar of the main docs page.


LDAP authentication
#25

Hi Michael, Thanks for the great job as always.
I have some doubts about the setup scripts.

  1. Why did u choose to break setups in 7 scritps? any specific motivation?
  2. Using custom directories are a problem in running the APP? (are directories path hardcoded?)
  3. I'm making a full-setup script (Interactive and no interactive) for vespene, pull request for this are welcome?

[]'s


#26

Hi @rmedivh

Vespene is going to be most frequently installed in a cluster where there are M headnodes and N workers - where each worker may be based on different images.

The user may host their own database or may wish to use something like RDS.

Vespene contains a tutorial configuration for trying it out that will likely only be used once.

Further, while I would like OpsMop to be the future, everyone uses different automation tools. Some people use container clouds.

As such, it makes sense for Vespene to adapt to everyone’s environment.

0_common isn’t really a script - it’s a set of choices.

The purpose of the rest of the installation scripts being split is so people can learn what makes up a Vespene install, how it is put together, and how to manage it.

Because I don’t see that changing (except possibly to also have some OpsMop in the future maybe?) I don’t really want to maintain a copy of someone’s parallel installation preferences - and especially not a merged setup that assumes a single node. I also do not believe in interactive installation, having of course created Ansible in the past :)

So I’m always interested in patches to those setup scripts but fully expect everyone to develop their own install automation that matches their site specific requirements for their Vespene cluster.

I am of course always interested in fixes to those scripts and instructions/docs, and inclusion of support for other platforms as well!

Hope that answers the question - I don’t want to discourage anyone trying to make a site specific install for them but also don’t think there is a one size fits all approach either.

Which is why the installation scripts that are here are about teaching what the production setup may look like.


#27

To answer your hardcoding question - yes and no.

I see nothing wrong with assuming /etc/vespene as a convention among all installs, and it seems to increase the pain of supporting all the user questions if everyone chooses their own.

That being said Vespene is a standard Django app and you can override anything in settings with /etc/vespene/settings.d or local_settings.py next to settings.py in the tree.

The build roots are configurable there, and that is one of the variables in 0_common.sh

Other than /etc nothing should be hard coded. The default supervisor configs reference /var/log for logrotation and it wants to create some systemd unit files on Linux.


#28

Hi @mpdehaan, thanks for the quick answer,
I would greatly appreciate your response about the above questions, your experience is really welcome to my understand of Vespene and Software Management at all. Thanks and sorry for any misunderstand of mine.

I truly understand whats the point about that people should know how everything works in Vespene setup, but if the deal is to encourage people to understand and make your own installation set of script (or modify the exist one) i think that the general information about how everything works (including dependencies and how to work with that) in Docs is ok (not so necessary to have everything high explained in the setup script, cuz it englobe a large set of environments), what do you think?

I'm sorry if i don't made myself clear, but when i said a full-setup (merged) it should include options to provide only APP Setup, Database, Headnodes, Workers, etc. Including some features to be more accurate in SO detection, handle errors in execution, SO configuration (APP user, internet connection, ensure directories, etc).
About Interactive Mode:
it's just provide options for people who want to execute the setup with parameters like --setup-node (Running local or executed by other applicartions), what do you think about this?

i tried to setup vespene on a simple fresh new CentOS 7 VM and the 3_application.sh didn't work well, syntax errors in python execution (psycopg2.ProgrammingError for example) and it's not clear how to handle this, at least for people who don't know python a lot (my case).

[]'s


#29

Can you post the specific exception you recieved? That’s new to me.

I’ll reply to other points in a bit later...


#30

Personally, that doesn't work for me. Mostly because I dislike it when documentation has 57 steps and step 5 is "go read this install guide" and step 10 is "go read this install guide". Which is why we want to have some really good ways to get this installed in just a few minutes.

Think of the scripts as executable documentation. Rather than saying 'type this' (which will rapidly grow out of date), we've got scripts that show the same things.


#31

I'm always open to seeing other proposals, I can't stay offhand whether I'd merge ideas from it, but if you have something on a branch or gist somewhere I'm open to seeing it and talking about it.

I do think conceptually having them in smaller pieces makes some sense, but there's nothing saying those couldn't just be functions.

If I ever decide to throw out the bash it would be for OpsMop and not anything else, and that's mostly from a desire to not do dual maintaince and definitely a desire to not keep another automation system in my head :)


#32

Hey, thanks for the responses, i have your point now.
I'll keep working in the setup bash script and probably open a PR.
Thanks for all,

[]'s


#33

I have a couple of observations about the tutorial.

I haven't yet got the dev tutorial build to work.
First hurdle was that I needed to configure my sudo user in Worker Pools. That's kind of ok and didn't take much thinking about having read the vespene concepts. I'm ok with a little learn through failure.
Second was that my cranky Centos 7.2 box required tty for sudo until I figured out that I needed to visudo and fix that (I believe its not the default behaviour upstream anyway). Easy enough to fix with a little searching and likely not that many people will hit it anyway.
Third failure was because the project I was trying to build had defaulted to 'git'
The error message here could perhaps be more helpful to noobs. Message was
"0.00m | add one or more SSH keys to the project or use a http:// or https:// URL" Perhaps add something like . "Check your Project configuration and ensure your Scm type is set correctly in the Repository tab".

(having some kind of 'next action on error advice' mechanism would a nice future enhancement, but lets just get the tutorial slick first).

This could perhaps be avoided by tweaking the configuration of the tutorial-project1-build (but at the expense of a little learning)?

Having set 'Scm type' to none for the project, next error I hit was
'0.00m | sudo: a password is required'
I've re-entered the sudo password in my worker pool, but still get the same when I re-run.

I'll dig some more but this is where I am stuck at the moment. Hope some of the above is useful to others.


#34

This is super useful and exactly the kind of info I want, thanks so much.

A few thoughts/comments:

Second was that my cranky Centos 7.2 box required tty for sudo until I figured out that I needed to visudo and 
fix that (I believe its not the default behaviour upstream anyway). Easy enough to fix with a little searching and 
likely not that many people will hit it anyway.

Hmm, I didn't hit that on my CentOS 7 install. We should definitely put that in setup notes at least.

Third failure was because the project I was trying to build had defaulted to 'git'
The error message here could perhaps be more helpful to noobs. Message was
"0.00m | add one or more SSH keys to the project or use a http:// or https:// URL" Perhaps add something like . "Check your Project configuration and ensure your Scm type is set correctly in the Repository tab".

I can definitely look at that. Do you mean the project you were trying to build was a SVN project or that the URL was blank? If you can maybe send me a screenshot of the project config that would probably help.

Having set 'Scm type' to none for the project, next error I hit was
'0.00m | sudo: a password is required'
I've re-entered the sudo password in my worker pool, but still get the same when I re-run.

That sudo password is entered on the worker pool config for sure. Maybe it's not the right password?

If you can send me what your sudoers file looks like that might also be helpful in recreating this.

The user I was using was probably set to sudo nopassword, but I need to verify.

The user I have been using has been "vespene_build", which is a user less privileged than vespene.

If you want to get past that quickly, you might want to turn off the sudo password for vespene_build short term?


#35

Thanks for this.

I found this bug report about the 'default requretty' in sudoers https://bugzilla.redhat.com/show_bug.cgi?id=1196451
Possibly the Centos box is a red herring for some of this as I'm not the admin of that machine so possibly there is local configuration applied that may be breaking things.

I was trying to build the tutorial project called 'tutorial-project1-build'. The script looks like a very bare bash script, but I think when I checked the Repository tab, the 'SCM type' was set to 'git' instead of 'none'. I just checked and that's not the case in either of the 2 vespene installations I have for any of the 4 same projects installed by the tutorial, so I'm doubting myself now.

I was just trying to use the sudo for my regular login so I think I typed it right.

Sudoers file (non comment lines) from Centos is as follows

Defaults requiretty
Defaults !visiblepw
Defaults always_set_home
Defaults env_reset
Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"
Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin
root ALL=(ALL) ALL
%SMCORP+Domain\ Admins ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
SMCORP+jhawkesworth ALL=(ALL) ALL

I also tried adding the following after my user, but still wouldn't run

Defaults:SMCORP+jhawkesworth !requiretty

On my Ubuntu bionic box it looks like its probably stock (non comment lines as follows:

Defaults env_reset
Defaults mail_badpass
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:$

root ALL=(ALL:ALL) ALL
%admin ALL=(ALL:ALL) ALL
%sudo ALL=(ALL:ALL) ALL

There's only a README in /etc/sudoers.d

So I guess I could do with some instructions on setting up a locked down user like your vespene_build - I'll go hunting for something.

I'd kind of like to get the sudo isolation working as I'd like the option not to have to make use of containers.


#36

Hmm, IDK what the problem might be, but I would benefit from seeing specific errors. You can run the worker process directly from the command line "python manage.py worker " instead of from supervisor if the logs aren't being too helpful.

I've done most of my 'vespene_build' testing on OS X, where I am sudoing with a password. Is it happy with a clean CentOS VM?