Remote Features


#1

Right now OpsMop is local only.

I ran a quick survey and was surprised lots of pull mode CM is still going on - I know a lot of people use CM at image build time or first boot, but there's a lot of classical management still going on (this is good, actually!).

Do people want a push or pull mode?

How might it work?

Originally I had some cool ideas that were going to serialize everything into JSON and shoot it over the wire, but really we can just bundle up the project directory and blast a zipfile.

As such, I think the code to deploy the code and run it and get data back could be another binary in the opsmop project, but might not need to be opsmop!

If we do pull mode, there might just be a really easy way to upload a zipfile to (highly scalable location with multiple pluggable transports), check for new versions, and suck them down and run on change.

There is some idea about whether Vespene wants to have a way to visualize OpsMop results, but I think I want to keep the two going down their own paths for a while.


#2

I use both push and pull modes (with Ansible), so I'd hope to have both.

I would request/urge you to take a hard look at Mitogen for remote execution. I funded Mitogen on Kickstarter, and he has had little success in getting Ansible itself to support it.
I haven't used Mitogen "in anger" and I don't know/understand your requirements...

My primary use of pull mode is when I am installing an OS on a machine, part of the kickstart/preseed process is to run my "base machine configuration" playbook. Although I started with ansible-pull, I ended up having to move away from that, and check out a git repo with my playbook, fetch/install all the roles I need, and then run the playbook.

There is an important advantage of the Puppet model of periodically (cron) running your CM, it strongly motivates idempotency, and prevents configuration "drift". Nothing is preventing me from doing that with Ansible, but it is not a "built-in" feature.


#3

Hi @dcj,

Doing something that supports both would be great.

I was thinking about that while driving today, and here's what I am pondering at the moment:

  • In order to support pull mode, we need a way to watch for a signal to update, to update content & run it, and then to send results
  • In order to support push mode, we need a way to push content and run it, and asynchronously wait for results

What both of these things have in common is perhaps the idea that OpsMop supports multiple callbacks at once, one for CLI output and another for sending REST somewhere (anywhere, to anything), or whatever the user wants to get status.

We then need to do pull and push modes.

For push mode:

Invoke-ssh-agent and add your keys - however - outside of the program.
Assume a script in the opsmop /bin called opsmop-push:

  • rsync content everywhere
  • optionally run a bootstrapping script to install python3 and opsmop
  • run one SSH invocation of executing opmop specifying a callback

For pull mode
Assume several scripts in /bin called opsmop-bundle and opsmopd

  • opsmop bundle, more or less ultimately pushes a zipfile or a set of files (however) into some datastore
  • it also supports plugins to watch for a signal and a seperate set of plugins to load from a transport
  • an example signal combo could be SQS + s3, or Redis, or whatever
  • opsmopd downloads some content from the server and runs it

Both of these should imply some concept of inventory to determine what servers to talk to, but I think this can be a lot simpler than what I've done previously and should just be a command line tool that returns JSON.

I kind of like the idea of the servers self-identifying with their content. For instance, there could be the idea that a server consults a local inventory class (if on AWS) to determine what AWS tags it has, and then runs that content. If a server wasn't on a cloud server, it could also look for tag names in /etc/opsmop/tags.d or something.

Dealing with a plugin to access secrets from the language would also be pretty easy, and we can talk about that when we get the basics down.

If we were to make a program to recieve events from BOTH pushed and pulled tools, it might be a very basic flask app and then it itself takes plugins, one of which could be a database but the most basic version would just output events to screen or logfile. This type of tool would be like /bin/opsmop-listen or something and would not require a database unless the receptor plugins required one.

I definitely want to concentrate on this being exceptionally scalable (scale-out being more important than latency, but 100% parallel for sure) so I don't want to just do git checkouts if I can help it - it would be too easy to thrash a git server.

For those that wanted to orchestrate multi-tier deployments in push mode they could just execute opsmop-push once on one role and then do the next layer once it got back.

So... that's just ONE possible thought for a minimal strategy. I would be open to other ideas.

Transports (how to pull data - PushTransports and PullTransports) and callbacks (how to report results) as well as Inventory modules (what's my AWS tags) would all be 100% pluggable and subclassable.

How does that sound?

The only weirdness is maybe having to run two processes during a push, but the CLI could also find a way to spin up that watcher inside a fork too.

I want to try to keep things more modular if I can, so the push and pull tools kind of only halfway know it is opsmop and could also be adapted to other tools.


#4

Got some better ideas about writing opsmop-pull

usage:
opsmop-pull [daemon|once] pull_cfg.py

pull_cfg.py defines several variables TRANSPORT=Transport(), CLI_STATUS_CALLBACKS=[ CliStatusCallback() ], PULL_STATUS_CALLBACKS=[ PullStatusCallback1(), ... ]

  • bin/opsmop-full first asks transports.is_there_content()
    REPO_URL pulled from environment for GitTransport

  • bin/opsmop-pull calls transport.download_content() which returns (temp_dir, filename)

  • we then call opsmop.core.api with CLI_STATUS_CALLBACKS specified in the pull_cfg.cfg

  • this is wrapped in exception handling and after policy execution is done we also call PULL_STATUS_CALLBACKS as set. Technically if finer grained reporting is wanted the CliStatusCallbacks could send fine-grained events wherever.

  • bin/opsmop-pull calls transport.sleep(), loops if called with 'daemon' mode.

  • initial version comes with GitTransport(repo_url=<>) and recommended use is with ssh-agent to deal with checkout keys

This is such an easy thing to do I may go ahead and implement this real-soon-now :)


#5

See the new opsmop-demo repo here for how opsmop-pull might be configured: