Project repos and venv


#1

I've been poking around, and I managed to get a project to clone a repo from a GitLab server, and I noticed that everything in /tmp/vespene/# subdirectories get chmoded to 777.

I also see vespene_launch.sh in /tmp/vespene/#, which is just the project script filled in with template variables. Makes sense.

But does the chmoding mean that a project repo is meant to contain nothing but scripts that are executed by the worker user? I was a bit surprised by this and wondered if it was perhaps not intentional.

I was also a bit surprised to see that install scripts install requirements.txt globally. I noticed in another thread that you had a hard time keeping up with Ansible's breaking changes. Me too, so I've been using python3-venv and pip-tools to pin Ansible and other dependencies to particular playbooks and roles. I do the same with Django apps. Works pretty well.

python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip pip-tools
pip-sync requirements.txt

I was actually poking around to find out it if made sense to set up Vespene to run an Ansible playbook in a Python venv, but my playbook repo got chmoded to all-executable, and I found that the worker user wasn't able to modify the venv since the repo directory is owned by the vespene user. So I'm pretty sure this all of this means my thoughts are not what you have in mind and all, but I haven't quite figured out what the intention is yet.

Oh, there's an stray "-v" typo in the git --config core.askPass="bash " line in the git scm plugin. Oddly, I also wasn't able to get the core.askPass=bash answer_file technique to work at all on the git command line in Ubuntu 18.04, so I just put a deploy token in the git URL. Whatever environment git is running in can't find an answer file no matter where I put it. Can't figure out why not.

Anyhow, thanks for the hard work.


#2

So the 777 thing is temporary per https://github.com/vespene-io/vespene/issues/96

This should be configurable for people that prefer 770.

Nope, of course not :)

It's anything you want to build and run. It's a build server after all :)

The chmod is just there so vespene can sudo to something like vespene_build (a less priveledged user) before actually running the build script. We can't leave the ownership as vespene to do that, and I also really can't assume things like filesystem acls are available.

This is the first time I've heard anyone having a problem with this, which seems to point to something unusual about permissions on the box. (umask weirdness?)

So it's basically a fact that if you don't use virtualenv somewhere you will get jumped on by users who are virtualenv fans :) I do use them for development and they are ok for that.

I think virtualenv's always feel weird for production.

The problem is virtualenv's are a pain for most users who are not hardcore python developers to understand and would unneccessarily pollute the supervisor configurations and elsewhere. It is recommended that you not install anything else on your Vespene machines that is not related to Vespene or the build/runtime environments - and that's a good security practice anyway. I mean, that's what actual VMs are for, etc. So it's not going to be a production issue.

The build will be running by the user who is set up in the worker pool, so like "vespene_build" or what have you.

The directory should indeed be writeable. Perhaps if you can share an "ls" of the permissions or something.

This doesn't look stray to me, can you elaborate?

Hmm, this worked for me. I wonder if this is permissions related and there is something non-stock about the box? Here's the code in question, which uses mkstemp:

If you are saying manually trying core.askPass isn't working just on the command line, that's pretty interesting and might be a git version issue?


#3

Re virtual envs, I hope you didn't think I was jumping on you, just surprised since I find them useful and worth the hassle. I work in a Python shop, and we usually deploy one or more Python "micro-services" on hosts, so venvs make maintenance a bit easier since different interpreters can run in their own environment and avoid having to update every service when you update some global dependency. As you say, Vespene should be the only thing running on your box, though, so I play by your rules.

Re my Vespene host, it's just a freshly downloaded and installed Ubuntu 18.04 VM, nothing odd or non-stock about it at all. I thought I'd use what seems to be the recommended OS, and I'm pretty sure I followed the installation instructions properly.

Re chmod, I think I see why you chmod -R 777 /tmp/vespene/25 everything. I think what I'll try is to clone my playbook repo by hand into a directory owned by the vespene_build user and have the build script call that. Avoid having Vespene clone at all. Also avoid the overhead of setting up the venv on every build.

Re -v, in vespene/plugins/scm/git.py in master:

cmd = "git clone %s%s %s%s %s-v" % (branch_spec, shlex.quote(self.repo), self.build.working_dir, recursive, ask_pass)

I noticed the use of tempfile.mkstemp, but I don't see anything that puts a -v on the end, so I get something like this in the logs:

timeout 600 git clone https://<user>@<repo> /tmp/vespene/5 --recursive   --config core.askpass="bash /tmp/tmpb1fdb53y"-v
Cloning into '/tmp/vespene/5'...
fatal: cannot run bash /tmp/tmpb1fdb53y-v: No such file or directory

I removed the lines in git.py that delete the answer file so I could look at it, and it doesn't have the -v on the end, so I removed it in the code and had a moment of joy, but then ran into the askPass problem.

Re core.askPass, I noticed that if I create an answer file manually, make it executable and run build (no "bash" in the command), it works, but if I change git.py to generate the answer file as executable, the file is still owned by the Vespene interpreter (lsof) even though it's closes, so it's busy and can't be read. Sigh. Git/bash not finding the file doesn't seem to have anything to do with the file being in /tmp, it can't read the file no matter where I put it.

Re umask and git version, they are configured as by the Vespene setup scripts. As the vespene user:

umask
0002
git --version
git version 2.17.1

Thanks for getting back to me, hope this helps.


#4

By the way, yes, I'm saying that running git from the command line on my stock Ubuntu 18.04 VM with --config core.askPass="bash <answer_file>" results in:

fatal: cannot run bash <answer_file>: No such file or directory

#5

This is a total wild guess, but what shell is your vespene_build (or whatever user your worker is sudoing to) using?
Or can the same user actually see /tmp/tmpb1fdb53y-v (and create files in /tmp)
I recall some weirdness of having to create a /tmp/vespene dir when I was messing around getting stuff started (although you are clearly past that).

If the above is not relevant please accept my apologies, but hope there's some ideas there for you to investigate.
Jon


#6

In general, regarding the "bash answer_file" technique: My vespene user starts with sh, but I have the same problem with the full git command manually in bash as well, both as the vespene user, and as a normal shell user with a home dir and all. I'm not sure what's going on. Thanks!


#7

Ha no, just I've seen like nine gazillion pull requests and it's a popular topic :)

I tried to get them working nicely originally but really they made the supervisor config a mess and I didn't want to have to explain them to those that weren't familiar.

Hmm, ok so I think that's got it - there's a missing space when you don't use a password vs a SSH key. I bet this was a bug introduced when I added the --recursive flag.

Adding a space before the -v will fix it, one sec...


#8

Ok, if you update Vespene on all of your workers you should be good to go:

Sorry about that, thanks for the report and apologies it took so long to figure it out. With askpass not in there it works, which means I was only trying it without after adding --recursive.

Should be good to go now, most likely not a permissions issue anymore either, just that pathing problem with the space.


#9

Thank you. It took long? I'll try git clone with "bash <answer_file>" on other platforms and let you know what I find. Still having trouble on 18.04.


#10

Did you update the Vespene code on each worker and restart the workers?

If you just updated the checkout, note that the actual code lives in /opt/vespene - you need to run an update there.

If you are still encountering problems after doing that, please share the full output from the build page.


#11

I have no idea what your system looks like to make a command like this work, but I can't get it to work, not with git 2.11.0 (Debian 9.6), 1.8.3.1 (CentOS Linux release 7.6.1810), or 2.17.1 (Ubuntu 18.04.1 LTS).

I saw your post at https://medium.com/@michaeldehaan/automating-git-from-python-for-non-interactive-applications-7e8c68095eb6.

The repo I'm trying to clone is on a GitLab server, and I'm using a read-only repo token/password, which could be the problem, but I doubt it. I'm not really willing to pay for private github repos to test this out, though.

In all cases, making the answer file executable and calling it without bash works fine.

The command I'm using looks like this, copied more or less from Vespene output:

git clone https://<my-gitlab+deploy-token>@<my-gitlab-url> <destination> --config core.askPass="bash <full-path-to-answer-file>"

I've also tried specifying the full path to bash, but that's not the problem.

In the answer file, copied from a file as generated by the function in commands.py:

#!/bin/bash
echo <my-gitlab-deploy-token-password>

The output is this, and git pops up the password prompt:

error: cannot run bash <full-path-to-answer-file>: No such file or directory

When I add the password to the URL in the Vespene project repository definition, and don't specify a login for the repo, the repo gets cloned fine, so that's what I'll do if I need to clone a repo for a project.


#12

Ok so, once again, I want to make sure that you've actually updated Vespene on the worker machine and restarted the worker, as the problem should have been the missing space before the "-v".

Can you confirm what git SHA revision you have on that worker machine? This is the version in /opt/vespene, so if you just updated the project and haven't re-run setup, those files likely aren't copied over yet.

I will need full unredacted output from Vespene, showing the error message in context of any traceback. I don't need to see your password, but I want to see more context surrounding your error message.


#13

I ran the project after updating the worker and restarting the service, but what I've described is the result of git clone commands outside of the context of Vespene, assuming that if I can't get it to work on the command line, how can it work in Vespene?

The current commit is 1fe4609eb352eeec8ea7169a18ed01e96da3006d. The upgrade instructions (https://docs.vespene.io/upgrades.html) aren't clear to me, but running setup/1_prepare.sh again seems to update worker code. I would like to have run it on a fresh VM, but that's not possible at the moment.

This is neither here nor there for me, and I though I was raising the issue as a courtesy, but clearly it's not. I'm satisfied that if it doesn't work for me, it doesn't work for me, and if other people aren't complaining, then I won't either, and I don't want to waste any more of your time. Thank you for the work you've put in to this and your other projects.


#14

Not following, this, just to be clear.

Thanks for SHA info. Looks like you are up to date.

I am trying to make sure your problem is fixed, but I do need to see full output from any tracebacks or messages to know exactly what commands it is executing. The last command you had pasted had the space problem before the "-v" so I need to see what the current problem might be.


#15

Hi Michael, I just thought you might be getting frustrated with my lack of clarity. I've had some time for this, so hopefully this helps.

I've tried to do this on as simple and as stock a setup as is possible to remove as many unknowns as possible.

I've installed Vespene from scratch on a fresh (and up-to-date) Ubuntu 18.04 VM, making no changes at all to 0_common.sh. I did not install the tutorial records.

As the Vespene superuser, I added the default tutorial-pool worker pool (running as the vespene_builder user, which exists), the SCM login for my GitLab repo, and a project.

The project is configured with the GitLab repo (and login, and worker) and otherwise does nothing else. (Personal note, I love how simple this whole process is. Very intuitive.)

I've redacted the GitLab server URL from the following logs, and I formatted it slightly so it looks better in markdown, but that's it.

I think this is what you've requested. Let me know if there's anything else you need.

Worker log

2018-12-30 21:12:40,805 - vespene - DEBUG - building: 1, project: Gitlab clone test
2018-12-30 21:12:41,045 - vespene - DEBUG - executing: mkdir -p /tmp/vespene/1
2018-12-30 21:12:41,108 - vespene - DEBUG - executing: chmod 770 /tmp/vespene/1
2018-12-30 21:12:41,401 - vespene - DEBUG - executing: timeout 600 git clone https://gitlab+deploy-token-5@software.example.com/devops/docs.git /tmp/vespene/1  --config core.askpass="bash /tmp/tmp_nfwxl8i" -v
2018-12-30 21:12:47,321 - vespene - ERROR - an error occurred
Traceback (most recent call last):
  File "/opt/vespene/vespene/workers/builder.py", line 158, in go
    self.main()
  File "/opt/vespene/vespene/workers/builder.py", line 67, in main
    self.checkout_and_record_scm_info()
  File "/opt/vespene/vespene/workers/builder.py", line 77, in checkout_and_record_scm_info
    output = self.scm_manager.checkout()
  File "/opt/vespene/vespene/workers/scm.py", line 46, in checkout
    self.provider.checkout()
  File "/opt/vespene/vespene/plugins/scm/git.py", line 88, in checkout
    output = commands.execute_command(self.build, cmd, output_log=False, message_log=True, timeout=600, env=key_mgmt)
  File "/opt/vespene/vespene/workers/commands.py", line 132, in execute_command
    raise Exception("Failed")
Exception: Failed
2018-12-30 21:12:47,323 - vespene - DEBUG - flagging build as failure
2018-12-30 21:12:47,429 - vespene - DEBUG - flagging build as done

Build output

0.00m | mkdir -p /tmp/vespene/1
0.00m | chmod 770 /tmp/vespene/1
0.01m | ----------

Pre hooks...

0.01m | ----------

Cloning repository...

0.01m | timeout 600 git clone https://gitlab+deploy-token-5@software.example.com/devops/docs.git /tmp/vespene/1  --config core.askpass="bash /tmp/tmp_nfwxl8i" -v
0.01m | Cloning into '/tmp/vespene/1'...

0.10m | fatal: cannot run bash /tmp/tmp_nfwxl8i: No such file or directory

0.11m | fatal: could not read Password for 'https://gitlab+deploy-token-5@software.example.com': No such device or address

0.11m | build failed with exit code 128
0.11m | ----------

Post hooks...

Notes

To make sure that the answer file actually exists, I removed the os.remove statement from plugins/scm/git.py so the file stuck around, and reran the project (after restarting Vespene). The file exists and contains the correct password.

As I mentioned, I've had no luck getting the --config core.askpass="bash answer_file" technique to work on the command line in any flavor of Linux I've tried. I get the same error using the exact same answer file generated by Vespene.

Note that when I make the answer file executable and use it as --config core.askPass=answer_file on the command line, (without "bash" in the command), it works fine.

I don't have access to a Mac, so I can't test this on OS X.


#16

Yeah not frustrated just trying to sort it out. I don't want to leave issues open that are biting people, especially as it's a relatively small program at this point.

Ok, so the file does exist.

This doesn't sound like a GitLab specific issue to me then.

It might be your git version but I still kind of doubt it.

What are the permissions on that file and can the Vespene user (who should have written this file) read it?

It seems like another likelihood could be that bash might not be in path... which would be weird, but could be?


#17

Re GitLab, I doubt it too since I can use the answer file as an executable in core.askPass="/tmp/anwer_file" and the repo gets cloned no problem.

Re git versions, I've tried 2.11.0 (Debian 9.6), 1.8.3.1 (CentOS Linux release 7.6.1810), or 2.17.1 (Ubuntu 18.04.1 LTS).

Re permissions, I can cat it as the vespene user. I've tried the clone from the command line as root without success, so I don't think it's that.

vespene@vespene$ ls -la /tmp/tmpzc9954q3
-rw-------  1 vespene vespene   37 Dec 30 14:47 tmpzc9954q3

Re bash in path, I wondered too, but I've tried /bin/bash /path/to/answer_file without success.

Re OS, does bash /tmp/tmpwhatever work from the command line on OS X? If so, I wonder if it's something git or bash does on Linux, perhaps a security feature? that isn't implemented in OS X.


#18

Shouldn't be OS X specific, I did do quite a bit of testing on Linux. I can perhaps try this combo again, but if we look at the message:

cannot run bash /tmp/tmp_nfwxl8i: No such file or directory

This looks like it is bash giving the error, because we are invoking it with bash, so it all seems weird to me. So throw out my thoughts about bash being not found.

Here's a Linux reference to ask pass with gitlab:

Might be an issue with the git version, I mean what if the file (not command) git is trying to execute is "bash /tmp/tmp_nfxwl8i", treating it as a file with a space in it, and newer git versions understand how to use arguments?

This seems to explain the problem.

If so, chmod +x on the file in question and then executing it without "bash" in front of it should work, and you should be able to confirm whether or not that is any good locally, outside of Vespene?


#19

I think it's git too. According to the git-config man page,

Some commands (e.g. svn and http interfaces) that interactively ask
for a password can be told to use an external program given via the
value of this variable. Can be overridden by the GIT_ASKPASS
environment variable. If not set, fall back to the value of the
SSH_ASKPASS environment variable or, failing that, a simple
password prompt. The external program shall be given a suitable
prompt as command-line argument and write the password on its
STDOUT.

Executing the answer file with bash does write the password to STDOUT, but I guess bash expects an executable program (as is bash), but without arguments.

Making the file executable works. There's no option in os.mkstemp to make the file executable "safely", os.chmod(fname, 0o700) does the trick, if you think that's the right way to do it.

However, you get the following if you don't os.close the file descriptor, so I've added a real var (instead of _) for the fd and added a line to close it. Seems to work fine.

    0.02m | fatal: cannot exec '/tmp/tmpoj401q1_': Text file busy

I changed as little as possible. Besides removing "bash" from the command, answer_file looks like this:

def answer_file(answer):
    (fd, fname) = tempfile.mkstemp()
    fh = open(fname, "w")
    fh.write("#!/bin/bash\n")
    fh.write("echo %s" % answer);
    fh.close()
    os.close(fd)
    os.chmod(fname, 0o700)
    return fname

I'll prep a PR.


#20

Awesome! Looks good -I replied on the PR and I'll likely have some time to get to testing this Tuesday, if not before.