Ability to contanerize the components of Vespene


#1

As of launch time, the setup scripts appear to have been designed to run on a VM or dedicated host, as there is a requirement at the database setup step for systemd (on Ubuntu Bionic).

It would be nice to have the ability to containerize the various services which make up Vespene to allow for singular or clustered deployments via Docker / Kubernetes.


#2

I would be totally into a couple of Docker files for worker and Vespene configurations, or maybe even having a script that generates those Docker files to make it a little easier to customize (not sure if that is needed).

Most likely this could live in something like setup/docker and we could move the other setup stuff into setup/bash or something like that

Even if the Docker files reused those setup scripts to build the image (I think they should), we could also pass them flags to make them act a little differently. like if the environment variable "DOCKER_BUILD" was set, maybe it skips the systemd parts?

This way everybody still uses the same scripts, the Docker setup doesn't drift much and it just skips the systemd parts.

How does that sound?

I'm not super familiar with Docker to be the one that would build this, but I'm all for it and happy to help make it possible. I don't mind if the setup scripts get a little bit more conditional logic, but think it would be good if the Docker scripts can still share them.

Not quite sure how we ask about the best way to bake in database setup so those work the first time on launch?


#3

I’ll help out on this over the coming weeks will get helm charts setup for k8s as well if you want. You want to startup a new repo maybe to track it under for the docker image, etc?


#4

Awesome! My gut feeling without knowing what shape it might take is keep it in the main repo.

I kind of think we'd need a base docker image but people also need to be able to build their own worker images, though, and if automation to generate those docker files requires some templating of docker files or whatever, maybe a seperate repo is better?

No strong feelings I guess.

I do want it to use the main shell scripts inside the docker builds though, just because I'm unlikely to test the use images super frequently in my personal workflow, that way they are less likely to diverge?

Kind of make sense?


#5

It makes sense to Dockerize the main repo, but maybe orchestration specific files could go to a separate repo to avoid an explosion of different formats: That is a compose or stack file can be useful for local dev but you probably want k8s or helm specs too.


#6

That sounds good to me, if someone wants to start a repo with what that might be named and looked like, I can start an empty one and take PRs to it or something later.

For what I'm paying Github I can add like 5 people to the organization before the price goes up, so I hope that isn't too annoying if we still operate on PRs with those?


#7

Is it a good idea to jump to k8s and helm charts at the get go?

Right now the only thing going for me with Vespene is the user friendly setup. Adding k8s to the mix without documentation and implementation would just mess things up. IMHO just go with docker images that can be used to spin up the master, worker, db and add on to it later for docker stack and k8s.

I think a lot of work is required on the images and making them customizable before rushing into k8s.

Just my $0.02.


#8

I'm super unfamiliar with those things, so if it's complicated - YES, I agree with @chopraaa - but I'd also like to see what that content looks like, and maybe it's not super complicated.

I had inferred maybe we are talking about one or two extra YAML files or something.

I also hope you find some extra interesting things along the way, whether there are some there in the docs or we add them :)

To be clear, it's never going to install Docker to install, and definitely would never require k8.

I'd also be open to having a docker chapter in the docs and then some links to some repos for extra orchestration content if people are interested at taking a look at them, kind of a like "you may be interested in" sort of thing?

Images are prereqs for sure.


#9

well I use kubernetes everyday and most of my applications run in there so yea adding it is necessary for me and its nothing special just a yaml file to literally do kubectl create -f blah and your entire stack is setup on kubernetes not sure how it could be any easier than that?

Here is a flask example I setup for a recent Kubernetes class I taught, I have one for django as well for my other side project I have but that code is private atm but I can port it over very easily.


#10

Yeah that seems totally reasonable.

Seeing I tend to hate git submodules TONS (I do need to add a --recursive to my git plugin, LOL, just thought about that)... keeping this in the main repo in a subdirectory is probably easiest.

If anyone wants to start up a pull request I'm game, just need to figure out how to make it externally configurable in an easy enough way in case people want to customize /etc/vespene settings or what the worker queue name for the image is (and what gets installed in the image).

I suspect that might take a little bit of thinking.


#11

I'm not saying it's not necessary for you. A lot of people live and breath k8s these days. It's just that - I don't.

From my perspective, people tend to oversimplify k8s deployment (similar to the way you just did) but deploying to a k8s cluster isn't the first thing someone (me in this case) would do to test Vespene (it's beta after all).

That said, my only ask is proper documentation for the images you will use for the Docker/k8s services + usage of these services. The documentation @mpdehaan has written is ridiculously detailed and upping the complexity on it would greatly undermine his effort I think.

I'd also be open to having a docker chapter in the docs and then some links to some repos for extra orchestration content if people are interested at taking a look at them, kind of a like "you may be interested in" sort of thing?

Ah, yes.

Cheers.


#12

I’ll push up a basic dockerfile in the morning that just follows the instructions and will map in the config etc and we will build upon it from there to eventually have Django in 1 container and Postgres in another etc. I already have a compose setup for this my snake app uses it that way but I need to understand the code here and I can help port it into micro services bascically


#13

Just to be clear though we can put kubernetes yaml files in a kubernetes folder in the repo and hurts no one be but now allows it in enterprise environments.

All it requires is a docker setup which we are already working on so it really is no more complexity but will allow more users to use it easier by doing helm install, docker-compose up, or kubectl create -f kubernetes folder


#14

Here ya go https://github.com/vespene-io/vespene/pull/39 let me know if you want to change anything but works fine for me :)


#15

Nice to see the traction this is getting, thanks for the initial structure you've put together @btotharye. I'm mindful of @mpdehaan's desire to integrate the setup shell scripts (at least at this stage) for consistency; I'll aim to iterate on what you've done to achieve that.


#16

Quick heads up I have modified the shell scripts to run as a vespene user.

I'm not sure what matters for the Docker files, but it does run sudo a bunch inside, so I think they might require a bit more logic.

Hope this doesn't overcomplicate any additional work but easier now than later.


#17

I tested you branch, the docker-compose file works fine on Windows with minor changes.


#18

Hi all, there's a lot going on here and I'm starting to get a little confused as to what to track.

See my last comments on https://github.com/vespene-io/vespene/pull/64 for what I'd be looking for in order to merge something.

It might be ok if we have some fully complete competing examples to figure out what they can do and can't, and we can look at those repos and see where to go from there maybe?


#19

For folks looking at this, here's some nice progress too - I think the key is going to be focusing on a production setup, not just a demo.

How do you accomodate those types of changes as I've commented on the PR within a docker setup, etc?

IMHO, docker makes handling state quite horrible, but I'm willing to entertain this if it is easy to maintain - hence my comments about reusing setup script code and not having multiple places for too many things.

But I don't want to endorse a docker strategy that doesn't work in prod, and isn't going to be HA or upgradeable.


#20

I like that @efim-a-efim has an "start from ubuntu18:04" approach to reuse the regular setup.

However maybe before going ahead with the Dockerfiles we should make a coordinated effort to ensure both ui and workers can be treated as 12FA. That would make them far easier to deploy and manage and we would avoid new wrapper 'run' scripts baked into the docker images.