SCM plugins; mercurial


#1

Just to say I've made a start on an scm plugin for mercurial.
Will update when I have something to share.


#2

EXCELLENT!

Glad to see this, let me know if you have any questions!


#3

Thank you. So far I've basically hacked up git.py. I'm leaving out ssh for now as I don't have a ssh-capable hg to test against, but can circle back to that once I get the rest of it running.

Anyway, my question is, any suggestions how I can get a bit more info about why my clone command is failing?

Currently I get this in my worker logs

2018-12-11 23:01:11,099 - vespene - DEBUG - building: 33, project: hg test
2018-12-11 23:01:11,241 - vespene - DEBUG - executing: mkdir -p /tmp/vespene/33
2018-12-11 23:01:11,257 - vespene - DEBUG - executing: chmod 770 /tmp/vespene/33
2018-12-11 23:01:11,301 - vespene - DEBUG - executing: timeout 600 hg clone -b configuration-branch https://hg.redacted.org/hg_test_repo/ /tmp/vespene/33 --config auth.vespene.prefix=* --config auth.vespene.username=jhawkesworth --config auth.vespene.password=[VESPENE-CLOAK][BASIC][V1]67414141414142634542454664e577954334b725958324b564f3834415272557365514f6e6553416d4e69486c75673451394e68686c527a4f615a60624c68365a743162586336584c567273506c76366247366f595367302d6e654f32644d4b4c76673d3d
2018-12-11 23:01:27,336 - vespene - ERROR - an error occurred
Traceback (most recent call last):
File "/opt/vespene/vespene/workers/builder.py", line 158, in go
self.main()
File "/opt/vespene/vespene/workers/builder.py", line 67, in main
self.checkout_and_record_scm_info()
File "/opt/vespene/vespene/workers/builder.py", line 77, in checkout_and_record_scm_info
output = self.scm_manager.checkout()
File "/opt/vespene/vespene/workers/scm.py", line 46, in checkout
self.provider.checkout()
File "/opt/vespene/vespene/plugins/scm/hg.py", line 69, in checkout
output = commands.execute_command(self.build, cmd, output_log=False, message_log=True, timeout=600, env=key_mgmt)
File "/opt/vespene/vespene/workers/commands.py", line 132, in execute_command
raise Exception("Failed")
Exception: Failed

The code I have so far is here - https://github.com/vespene-io/vespene/compare/master...jhawkesworth:hg-scm-plugin

One thing I think is problematic is if password contains something like a '(' - I haven't yet found a way to escape that reliably. When testing the generated 'hg clone... ' command line I had to either single quote the entire password string, or prefix the ( with a \

Is there a good way to escape this that you know of?

Sorry if this is just is just a basic python question - I don't get to code much python so am permanently a bit of a noob at it. It has that perl-like quality of letting you learn just enough to get something done which is sometime a double edged sword.


#4

Ah, this is an easy one actually.

What's happening is the passwords are stored encrypted in the database, and you have to call some code in the plugin to get the unencrypted versions.

Now, that password really shouldn't be in the command line, we do various things with expect in the git plugin to send that along.

The important thing to catch is the difference between self.project.scm_login.password and self.project.scm_login.get_password() - the first is the encrypted password, the second is the actual password, decrypted by the secrets file.

The output of the command will be logged if you change the way you are calling commands.execute_command.

The key part here would be to change "output_log=False" to "output_log=True"

This raises a decent question of why I turned it off on the git plugin, probably because I thought it was too noisy. That doesn't quite make sense though, as it's maybe important if the checkout fails.

What I think we should PROBABLY do is add a new field to the project model that is something like "verbose_checkouts" and if that is True we could show the full checkout info in the log or something like that.

Does hg provide a way to get the password from an executable file? If not, we can also decide to not log the command execution itself, but I'd prefer if there was a better way. We could also add a feature to the command class to regex-scrub the log entries if we really needed to. I suspect there's a way though - it's been a very long time since I've used hg.


#5

Thanks for this. I am getting close...

Now I'm using self.project.scm_login.get_password() I am getting the password I entered in the logs. Yay.

I suspect you set output_log=False because you'd set output_message=True to make the command output go into the messages output for the build so its visible in UI (where I'd argue its more useful than just buried in the worker log).

So armed with the output from the message log I can now see I get

0.28m | abort: authorization failed

0.28m | build failed with exit code 255

So .... I'm missing something else as the generated command line works just fine when run at the cli using the same user as the worker does, interactively, but fails as above via the worker.

Only ways I have found so far to supply hg with the password on the command line are via the --config options I'm using now or via username:password@ in the source url. Sounds like I'm going to be reading up some more on mercurial so I'll look and see if there's another way.

I'm sort of fine with not logging the command line, I only did it because I wasn't sure I'd generated it correctly.


#6

Yeah, it's been too long since I wrote this so I kinda forgot, "output_log" has a companion field, "message_log=True", which you are using, because you did get the output... so that was all fine. I was barking up the wrong tree about your question about how to debug the problem some more.

"So .... I'm missing something else as the generated command line works just fine when run at the cli using 
the same user as the worker does, interactively, but fails as above via the worker."

Maybe you have some local configuration file that is making it work? Like this stack overflow about a push talks about enabling SSL or not in a config file that is probably outside the repo:

https://stackoverflow.com/questions/946745/mercurial-push-abort-authorization-failed

I'd maybe try the clone/checkout on a clean VM to see if the command works there, maybe?


#7

So it turned out I had the command syntax wrong after all, and at long last I have a version of this works for me now.

I haven't found a way to get hg to accept the password from an arbitrary file so I'm not sure best way to avoid logging out the password in the message log right now. It will read from ~/.hgrc but I wouldn't want to trash that if it already existed.

One option might be to add something to execute_command to pass in a loggable version of the command line. Unfortunately 'log_command' already exists as a parameter and is a boolean so that might be an ugly change. I'll think some more.

I agree it would be useful to have a verbose checkout option in the model. I was also wondering about a checkout extra args as hg has a million configurable options.

Assuming I can figure out a way around the logging issue, would you be happy with a PR for this without ssh: support? By the looks of things I'll need to spin up something to test ssh: mode.


#8

I'm wondering if there is a config option that allows you to tell it where to look for the ".hgrc". This config file option is found in a lot of tools, so it might be be there is either an environment variable or a "--config" for it.

I just now tried to briefly dig through the hg source code, but it is not organized in a way I could find out the answer and online docs were lacking.

Failing that, I'd probably propose we add an option to my Command class that provides a regex to scrub the command text, though this still enters into history.

I also just now (how did I not know about this) found out about

The command class also does take environment settings as parameters, so we could also keep it out of history.

That would be good enough for me.

The regex could probably be passed the actual password to delete and replace with "PASSWORD".

Sound ok? There may be better ideas.

I'm ok with it not supporting SSH, we should just make sure the documentation indicates it's not there yet.
However, I suspect the mercurcial SSH support is really easy to set up with a VM, so it would be great if you could add it also.

This is maybe easy to implement, and you can take a look at what the git version does to try to make the authorized keys checks not hang the system.

But if not, that's totally ok.

I'd like to hold off on checkout arguments until we have a specific need as I'm trying to balance "self-service ops" type use cases with giving them too much flexibility.


#9

Thanks, for this and in particular for diving into the hg sources.

So far my searching hasn't uncovered a way within mercurial to override the location of the $HOME/.hgrc I suppose we could temporarily set HOME to something else but that feels wrong to me.

Maybe it would be better to just use $HOME/.hgrc if present, otherwise attempt to fall back to entered credentials, and just document this.

That still leaves need to scrub the comand text and not store command line in history. I'll have a go at that in a separate PR. I can't think of a reason off the top of my head to sanitize but not avoid history so I'd implement the history avoidance if the scrub regex is supplied.

It turns out the ssh support is really loosely coupled to hg (nice) so I can test it and will make that an option.
Interestingly you can't specify password as part of ssh hg clone source for good reasons:

https://www.mercurial-scm.org/pipermail/mercurial/2015-December/049156.html

Agreed about checkout arguments.