Django channels for live updates


#1

I've got a lot of mileage out of Ansible (huge Ursula K LeGuin fan and The Dispossessed was probably my favourite), and I see that you're a minimalist who's good at using the tools at hand, and I totally understand why you don't want to get involved in the React/Angular/Vue/etc. Javascript "ecosystem".

That said, I've been using the Django channels project (https://channels.readthedocs.io/en/latest) to push notifications to browsers. It's simple and works well, and I believe there's still some argument about integrating it into Django's next major version. (I guess we'll see.)

Basically you'd swap gunicorn out for Daphne, the asgi server that channels recommends in its docs, write a consumer or two in Django that trigger on db signals, and add a websocket to project and build pages so users could see live build status. Maybe a separate per-build view to tail progress if you really wanted to go crazy.

Not sure if you had considered it, just a thought. Ignore me at your leisure. (Though it's an argument for requirements in a venv - pip-tools's pip-sync makes it easy to swap libs in and out :D)


#2

Hi @tessercat,

Ansible was mostly an Ender's game reference to the way he drove multiple ships remotely at once, but I really do need to read that yet :)

Thanks for sharing the channels reference - I came across a reference to ASGI somewhere but couldn't find a practical application.

There are currently two things that folks are looking for live updates for, probably

  • the build console output, so you don't have to press reload - which might as well include the status on the same view

  • the main red/green type lights on the project and build index pages

I'm not familiar with pip-sync but typically just leave old stuff lying around in my virtualenvs. Either way, I don't think it requires a venv-in-prod type change.

So yeah, quite open to looking at this... if this is something you'd like to maybe work on (if you think it's really minimal) a basic example that just live-updates the build console and messages log seems like an easier proof of concept than the status lights.

One thing to maybe be mindful of is it should probably not steal focus if the user is looking at another tab.

The stuff that generates the supervisord config will also need to change (search the code for gunicorn).

Something you might want to try to add?

Thanks!!


#3

So this is probably all exceptionally outdated, but one thing that is important is browser support.

What javascript libraries do we need to pull in (any?)

If we can get by with just jquery that remains ideal, I imagine for the build-light updates we can write some easy functions that send over the HTML for the individual cells rather than the raw data, to avoid duplication of UI code between the live update code and the existing templates - as unusual as that might be, it seems quite useful to not have to have two implementations that need to result in the same UI.


#4

Hi @mpdehaan, I can probably do this if someone doesn't get to it before me. I've been evaluating Vespene as a simple build server, and from what I've seen it does basically what my company needs in a way that I'm pretty sure even the NOC can get on board with. (Also started reading about OpsMops.) I'm on the road for Christmas, but I can look at this in the new year. I'll try weaving it into what you've already got and write up a ticket with what I think would change.

Re JS, I've just been using raw browser websockets and assuming users have an up-to-date browser, FF or Chrome, and jQuery works fine for replacing DOM elements if raw JS doesn't.

Re pip-sync (along with pip-compile that compiles requirements.in files to requirements.txt files), it's part of pip-tools, and it's the simplest thing I could find that does what I need - sync pip reqs in a venv. I found pipenv unusable, and I haven't tried Poetry yet, but pip and pip-tools works fine for now.

The only thing I'm concerned about is the coming REST framework. I haven't looked, but I assume you'll be using the Django REST framework. Not sure how that will fit in with asgi/wsgi/websockets. I'll keep digging through the code/docs/tickets.

Merry Christmas.


#5

Glad it looks like a good candidate for your shop, and Sounds good!

My main concern with any new additions to Vespene are keeping things easy to maintain, so minimal use of Javascript libraries that may need frequent updates to keep working with browsers is a major goal - as is keeping up code reuse in the Python code.

(skipping the pip-tools discussion...)

It's definitely going to be Django REST framework, a really basic API surfacing things like:

/api/builds
/api/projects
etc

It will be very 1:1 with the database, and I don't think it will affect what you are doing at all.

I suspect this doesn't affect the websocket development at all, as that is all stuff that happens inside the routes and (unless I'm mistaken) the websocket stuff doesn't need to use REST.

Merry Christmas to you also!