I cannot get a build to work


#1

So, I followed initial setup, and got vespene running. Clicked on the initial test build, and it failed.


    0.00m | mkdir -p /opt/vespene-builder/16
    0.00m | chmod 770 /opt/vespene-builder/16
    0.00m | ----------
Pre hooks...
    0.00m | running build asunder user: vespene-builder
    0.00m | ----------
Building...
    0.00m | chmod -R 777 /opt/vespene-builder/16
    0.00m | chmod a+x /opt/vespene-builder/16/vespene_launch.sh
    0.00m | see 'Output'
    0.01m | build failed with exit code 1
    0.01m | ----------
Post hooks...
0.01m | sudo: a password is required
0.01m | Failed

I have vespene-builder user setup on ubuntu, no sudo pass needed in sudoers file, its home dir is /opt/vespene-builder
The worker pool is using it, why is it still getting this sudo issue? I put sudo password into worker pool through ui as well and still this issue.

Whats the deal?


#2

Hmm...

The "a password is required" message does not seem to be coming from Vespene as that code is over here:

(no messages about that)

Can you perhaps paste the sudoers file?


#3

So despite having a worker pool, where sudo-user and sudo password is set, the build-root is created by the user vespene, and not the worker pool. Then some files are created in the directory by vespene again, and then the worker-pool user is finally used.

total 8.0K
-rwxrwxrwx 1 vespene vespene 55 Dec 6 14:58 vespene_launch.sh
-rwxrwxrwx 1 vespene vespene 231 Dec 6 14:58 vespene.json
-rw-r--r-- 1 vespene-builder vespene-builder 0 Dec 6 14:58 yourself


#4

So for my setup, I need to either use /tmp/ or give the vespene user access to the build_root


#5

Hey a few quick questions - are you saying this resolves your issue or this is a different one? I was going to say try adding a password just to have something in that field and see if that helped.

 Then some files are created in the directory by vespene again, and then the worker-pool user is finally used.

The worker pool by this definition is the process that runs "python manage.py worker "

The build root is created by the worker pool process, so what you say seems normal to me. Though I think they do need to be chowned to the build user.

Permissions look a little too open to me, did you 777 them?

Is the file "yourself" something the build script created?


#6

Now that I understand that the build-root / builder# dir is created by the vespene user, I switched back to /opt/vespene/tmp/ (because ubuntu does wierd lvm things and I decided to make /opt/vespene the big lvm)

mkdir tmp, chowned it with vespene:vespene, kicked off a new build.

drwxr-xr-x  3 vespene vespene    16 Dec  6 15:23 tmp
root@vespene:/opt/vespene# cd tmp
root@vespene:/opt/vespene/tmp# ls
24
root@vespene:/opt/vespene/tmp# ls -ltrh
total 0
drwxrwxrwx 2 vespene vespene 67 Dec  6 15:23 24
root@vespene:/opt/vespene/tmp# cd 24
root@vespene:/opt/vespene/tmp/24# ls -ltrh
total 8.0K
-rwxrwxrwx 1 vespene         vespene          55 Dec  6 15:23 vespene_launch.sh
-rwxrwxrwx 1 vespene         vespene         231 Dec  6 15:23 vespene.json
-rw-r--r-- 1 vespene-builder vespene-builder   0 Dec  6 15:23 yourself

My builds work now, the build script is touch yourself I was being cute =). the 777 on the files in there, and on the build# dir appear to be how vespene does it.

I am no longer blocked.


#7

Ah, it's been a little while since I wrote this and forgot:

The permissions come from

chmod = self.build.project.worker_pool.permissions_hex

Which is not exposed in the UI at the moment - it should be.

I think this is something I ran into because I didn't want to worry about creating a group that included both vespene and the build user in the setup scripts and thinking that no other users should really be installed on the build workers.

I think I should expose this in the UI so people can set it to something else if both users share a group and can open a ticket on that.

Do you know what the sudo problem was?


#8

#9

I did a bunch of nonsense trying to fix it. Turns out vespene user needs sudo access without a password for this to work.

sudo:x:27:jmainguy,vespene-builder,vespene
root@vespene:/opt/vespene/tmp/24# grep sudo /etc/sudoers
# This file MUST be edited with the 'visudo' command as root.
# Please consider adding local content in /etc/sudoers.d/ instead of
# See the man page for details on how to write a sudoers file.
# Allow members of group sudo to execute any command
%sudo  ALL=(ALL) NOPASSWD:ALL
# See sudoers(5) for more information on "#include" directives:
#includedir /etc/sudoers.d

If I remove vespene from the sudo group, and then bounce vespene, builds fail again with.

    0.00m | chmod 770 /opt/vespene/tmp/25
    0.00m | ----------
Pre hooks...
    0.00m | running build asunder user: vespene-builder
    0.00m | ----------
Building...
    0.00m | chmod -R 777 /opt/vespene/tmp/25
    0.00m | chmod a+x /opt/vespene/tmp/25/vespene_launch.sh
    0.00m | see 'Output'
    0.00m | build failed with exit code 1
    0.01m | ----------
Post hooks...```

```    0.00m | sudo: a password is required
    0.01m | Failed```

since there is no where to place the vespene user password anywhere, not sure you can get around this currently without adding vespene user to sudo group, and possibly locking it down to mkdir and chmod commands.

adding user back to sudo group fixes builds.

#10

There is a sudo password field on the worker pool config. Were you saying if the sudo password matches it doesn't work for you?

If you remove the ability of the vespene user to sudo that definitely shouldn't work, but I think it probably does warrant reiterating that this needs to be set up.

The setup install doesn't really manipulate the sudoers file so I can see that being a source of confusion for sure.

Thanks for the heads up and I'll also file a ticket on that part, but if you can test sudo with a password real quick I'd appreciate it - that SHOULD work if you have the password configured on the worker pool in the UI.


#11

Added a ticket to write more about needing to edit the sudoers file - https://github.com/vespene-io/vespene/issues/97


#12

worker pool lets you set a sudo user, and a sudo password.

I set this to vespene-builder, with vespene-builders password. However it fails, saying a sudo password is required during a build. I believe this is because the vespene user requires a password, but there is no where to set it, I set vespene, and vespene-builders password the same on the command line, and that didnt fix it either. Changing sudoers back to NOPASSWD fixed the issue.

I was using chrome to interact with ui, and the OS vespene is installed on is ubuntu bionic beaver.


#13

The user doing the sudo'ing is the vespene user (that started the worker process), so the only password you need to add to the worker pool dialog is the password needed to sudo into the vespene_build user, which should be their password if a password is required.

It could be that there is some issue with the way I am calling sudo with expect - I didn't adjust my requiretty settings at all on Bionic, though I did test this, so I'll need to do it again soon to make sure.

Just saying "set sudo nopassword" is generally good advice for those machines though, we should probably just recommend that regardless because if there was any problem that makes it go away pretty nicely.


#14

Thanks for all this feedback BTW - very helpful and much appreciated!


#15

=) glad I could offer something. yeah I like sudo without password, Sudo's purpose, in my mind is merely to track the commands the user is running, sudo is easy to break out of, you generally needed the password to login as the user anyway.


#16

If vespene needs a sudo password, were would you add it at?


#17

Just to make sure I'm not confusing anyone else, "Vespene" the program and "vespene" the user the Vespene processes run as are of course different :)

The "vespene" user does not need a password because the worker should be started under that user by systemd (which runs the workers and web server through supervisord).

The sudo isolation plugin for build security is one of two supported modes configured on the worker pool in the UI, and there's a sudo password field there.

But I think your questions was whether the vespene user password matters, and it doesn't.

The systemd init script handles that and is set up by this:

(So all processes will be running as Vespene already)

However, the user you are sudo'ing TO in the sudo isolation plugin can have a password, and that is what is configured on the worker pool in the UI.

Hopefully makes sense?


#18

Yup I am on the same page now.


#19

Great, I have those two github tickets to improve the docs path around all of this at least and will re-run my sudo testing on Ubuntu soon.

I don't really want to muck with people's sudoers files in the installer and we don't really setup the Vespene build user now, so there's a little bit of last mile config we need to make easier to understand for sure.


#20

Thanks for this.
Adding vespene and my worker user to the same nopassword sudoer group got my builds running on Bionic.