It's learn-to-write-modules month!


#1

This month (well, all the time, but especially this month) I want to be around to help people learn to work on modules.

If there is something you would like to add, or maybe even just develop an internal module for your own usage, post up here, and I'd be happy to answer questions.

In the meantime, I'm going to be working on CLI upgrades, improving some error messages, and then push and pull modes.

As I've posted on twitter, some good easy ideas of things to try, if you want to:

  • add Facts for Arch or BSD or SUSE (etc)
  • add a Package provider for Arch or BSD or SUSE (etc)
  • add some facts you think might be missing
  • add "from_url" to the File module
  • anything else you might want to think of

We're going to be a little careful about what we take on module-wise in core, so if you are ever unsure if something should be a core module, let's talk about it.

Up front discussions are also always good because if something should take a slightly different shape, that can also maybe save some work up front.

Meanwhile, OpsMop is quite happy with modules not being shared upstream too, so if you want to develop something for in-house usage, that's also fine, and I'm happy to answer questions and help with that too!

If you have any internals questions - or ideas for other things - those are also good as well!


#2

I would like to start working on some modules for network interface configuration and associated facts.

I started working on the Facts module using ioctl which is pretty straightforward and should work on all Linux + OSx, I think I have an okay handle on that part.

In regards to the module(s), I am not sure what the best structure is. I think the obvious starting place is to have a network interface module with providers for interfaces_file, sysconfig, nmcli, etc. But there is also a distinction between bonds, bridges, and regular interfaces. Each with a lot of parameters in common, but also a lot that aren't shared. I don't like the idea of just lumping all of the parameters into one module as that will wind up being especially confusing trying to figure out which parameters are applicable to a bond but not a bridge.

So I'm not sure if it would be better to maybe have three modules; interface_bridge, interface_bond, interface each with the respective providers, or find another way to organize things. I guess one of the alternatives would be to have a module for the different network managers and then have a bond/bridge/int provider for them.

I'm going to keep thinking on this, and probably come up with some mock-ups of the way I think the module interface/parameter structure should be presented, but I figure I would pose it for discussion first since I figure this has a good chance of winding up in core.


#3

This is where I should admit that in the last 3 years I've kinda been benched so I'm not clear what the state of the art in Linux OS network configuration is.

I see there's netplan configuration for various things now.

Is that a possible good backend for a provider - a good idea, or a bad one?

How much do we really need fact modules, versus how much we'd be just declaring this is the way something is? In those cases, I'd probably encourage developing the least amount of facts neccessary for your specific use case, and making sure things grow with needs.

It sounds like you have a lot of physical datacenter configuration needs that I normally don't have, so I'd also ask there what are the static configuration use cases, etc, that you are trying to work with.

It seems quite obvious that we need to minimally expose IP addresses and such in facts, but understanding the why of what people want to do would be helpful for me.

I definitely want to fight the sprawl found in past efforts where possible, so if there are clever ways to accomodate configuration that seems to be a plus.

However, at the same time, I want to avoid a trap I've seen a lot of ansible modules fall into - where they just take dicts and have very little validation.

If we want to have a full object model representing part of a configuration that should be true, and one resource that processes that model, technically it's possible. (what I mean here is feeding essentially a tree of Python objects into a resource, not just a simple dict, so that it can validate up front)

if facts want to return a whole object model (hopefully the same one?), they could do that too - we definitely don't want to have a giant number of fact modules where we don't have to. They could just as easily return a list of "Interface" objects that are also used by the configurations... this also has the benefit of making the invocations more like just regular python.

I'm not sure it's a great idea, but it's possible ?

Definitely ++TONS for the resource pseudocode first approach.


#4

Its kind of a mess right now. Netplan is good, but catches a lot of undeserved vehement hatred for some reason. As a result of this it never really made it off of Debian derivatives. One of the wonderful things about it is that it can actually do some pre-flight checking on configs before applying, which I don't think any of the other network managers can say. The vast majority of infrastructure I've interacted with is CentOS, where the old network service is still reading from /etc/sysconfig/network-scripts for the most part. Without an awful lot of wrench turning in your images you are either going to run that or Network Manager. NetworkManager does have a CLI, and I think they have feature parity with the GUI now, but it kind of falls over when you do more complicated stuff.

I think the two sensible places to start are Netplan; which covers all of Ubuntu, and can be easily bootstrapped into Debian, but isn't really available for CentOS/Fedora or anything else (and I don't think that is likely to change) and something to manage /etc/sysconfig/network-scripts files. NetworkManager is still pretty ubiquitous, and can be used on pretty much any "mainstream" Linux distro, but it doesn't work for some of the more complex configurations I work with.

The wildcard is systemd-networkd (which netplan leverages anyways). Its pretty much everywhere, but I've seldom if ever seen it used by itself. Admittedly, I need to do more research on this one in particular.

I'm mostly interested in Facts for the test/rollback system that we were discussing in another thread. I definitely see your point here.

One of my day-job duties is running a multi-tenant virtualization farm, so we have a few different variations of physical servers that each have 6 or more bridges (all in different segregated subnets/vlans) built on 2 or more bonds on 4 or more physical interfaces. This is a pretty common set up for anyone running Openstack or Cloudstack especially if VXLAN or another SDN protocol is used for tenant isolation. Automating this set up with existing automation tools can get really frustrating really fast.

I'm a huge fan of the concept that facts would return the same object as what gets fed into Resources, this also plays super well into the rollback concept. I'm thinking that a simple inheritance model where you had a generic "interface" with all of the basic attributes (it has addresses and stuff) and then had a few other object types that inherit it (bonds, bridges, and physical interfaces). Where is the best place to declare those, inside the provider module itself or inside of opsmop.types?


#5

Excellent, glad to have your experience here!

I'm a huge fan of the concept that facts would return the same object as what gets fed into Resources, this 
also plays super well into the rollback concept. I'm thinking that a simple inheritance model where you had a 
generic "interface" with all of the basic attributes (it has addresses and stuff) and then had a few other object 
types that inherit it (bonds, bridges, and physical interfaces). Where is the best place to declare those, 
inside the provider module itself or inside of opsmop.types?

It seems like they are types, but maybe if they have resources below themselves (nested, as it were, like if Interface has a Bond) those sub-types don't return providers, just maybe some of them? Not saying that level of modelling needs to exist, but if it did, that would be ok.

I'm trying very hard to avoid the "everything is a dict" model we got into before, though, so if more classes are needed that is ok.

If there starts getting to be too many, we could make a subpackage types.network

This seems like it would work really well with the rollback syntax...

class Foo(Role):

old_network_state = None

def pre():
     old_network_state = Blah()

def set_resources():
      return Resources(
      )

def test():
       return False

def set_rollback_resources():
      return Resources(old_network_state)

I haven't thought what those network state objects would be like at all, but it might be better if that's an array of interface objects or something similar?

How might bonds want to be expressed?

Maybe it's an array of interface objects and bond objects, and that might be ok.

We could just have the rollback be:

return Resources(**old_state)

The catch would be that if we added any new resources, that weren't in the old state, it might get weird.

So that might be a reason to return the state of everything instead, and then the objects could remove things that weren't referenced.

But in other cases users are likely going to want to just describe parts of things...

Just thinking out loud here.


#6

How about something like this:

class BondExample(Role):

def pre(self):
    old_state = NetworkFacts.get_interface_collection()

def set_resources(self):

    Interfaces = InterfaceCollection() # A container for interfaces, the same thing returned by the above step
    # It probably also has built in methods for sorting and returning a subset of interfaces, such as .get_bonds()

    bond = Bond(name="bond0",
                boot_proto="static",
                ipv4_address="10.1.1.20/24",
                ipv4_gateway="10.1.1.1",
                mtu=9000,
                mode="802.3ad",
                xmit_hash_policy="layer3+4",
                miimon=100)
    Interfaces.Add(bond) # Add the bond interface to our collection
    for ifnum in range(1, 5): # Creates interfaces em1, em2, em3, em4
        Interfaces.Add(Interface(name="em{}".format(ifnum),
                                             slave=True,
                                             master=bond))
    Interfaces.Add(SubInterface(parent=bond,
                                 vlan=1000,
                                 ipv4_address="192.168.0.2/24",
                                 ipv4_gateway="192.168.0.1"))
    return Resources(Network(provider='sysconfig',
                             auto_reload=True, # Could also be a handler and let the user do this on their own
                             interface=Interfaces # Accepts a single interface or a collection of interfaces
                             ))

def test(self):
    return Resources(Shell('ping -c 1 10.1.1.1')) # Or however you establish test conditions

def set_rollback_resources(self):
    # Since old_state contains a collection, we are basically doing this the same way as above
    return Resources(Network(provider='sysconfig',
                             absent=True,
                             interface=Interfaces.get_bonds()),
                     Network(provider='sysconfig',
                             absent=True,
                             interface=Interface.get_subinterfaces()),
                     Network(provider='sysconfig',
                             auto_reload=True,
                             interface=old_state)
                     )
# This ensures all the bonds and subinterfaces that we tried to add are removed, and then restores the old configuration

In theory I suppose the last part could just be written like this:

    def set_rollback_resources(self):
    return Resources(Network(provider='sysconfig',
                             absent=True,
                             interface=Interfaces),
                     Network(provider='sysconfig',
                             auto_reload=True,
                             interface=old_state))

Thoughts?

Will we have access to the same variable namespace in all of these methods, or will we have to set self attributes of the class in order to re-use objects from other methods?


#7

Random thoughts:

old_state = NetworkFacts.get_interface_collection()

Maybe:

get_interfaces()

Object could just be named "Interfaces"

And in your set_resources:

interfaces = Interfaces()
interfaces.add(Bond(...))

return Resources(
     Echo("the interfaces collection is itself a resource and not a true Collection object in OpsMop, so we can just do this..."),
     interfaces,
     Echo("and that's it")
)

Also if there's some way that a Bond is constructed more like this:

 intf1 = Interface(...)
 intf2 = Interface(...)
 bond = Bond(intf1, intf2)
 Interfaces.add(intf, intf2, bond)  # just a **args method, can add anything together

I'm not sure if that makes sense, but does that help eliminate the "SubInterface" class and the concept of needing parents?


#8

I like this, I'll incorporate it into my next few examples.

In Linux networking, interfaces have parents so we have to define that somehow, I figure associating the objects is the most sensible option. That way when you are building the config for each of the slaves, you just need to check the value of self.master.name to get the value you need to stick in the config file for the master parameter. I don't even think it needs to be a two way relationship necessarily. You could reference the interface via a string, but that seems very un-pythonic.

Sub-interfaces are a valid type of linux interface. The most common usage is accessing a tagged VLAN on a switchport, but you also see a lot of them when you get into SDN protocols.

In the above example, the network dependency map looks something like this:

 +------------+   +------------+  +------------+   +------------+
 |    em1     |   |    em2     |  |    em3     |   |     em4    |    <- Physical Interfaces
 +------------+   +------------+  +------------+   +------------+
     |                   |         |              |
     |                   |         |              |
     |                 +-v---------v+             |
     +----------------->   bond0    <-------------+       <- Bond Interface
                       +-------^----+
                               |
                               |
                               |
                      +----------------+
                      |   bond0.1000   |       <- VLAN Subinterface
                      +----------------+

Each layer has it's own possible attributes, a lot of which is shared amongst all the interface types. Any interface can have an IP address, or an MTU; but only Physical Interfaces have a Speed or Duplex, and only bonds have a hash policy or mode as examples. That is why I think they all deserve their own type classes; although they could inherit from a type that provides all the attributes they have in common (which is a lot, probably the majority.)

Its also worth mentioning, that if you are using NMcli or just using the ifconfig or ip utility to configure all of this, there is a dependency structure as well. In this case, both the bond slaves and vlan subinterface can't be assigned to the bond until it exists. This doesn't matter for Netplan or sysconfig methods since you are generating config files and then bouncing a service, but it would when command line providers are added.

Here are some more examples that integrate your suggestions:

# Single interface

def set_resources(self):
    interfaces = Interfaces(provider='sysconfig',
                            auto_reload=True)
    interfaces.add(Interface(name='eth0',
                             bootproto='dhcp'))
    return Resources(interfaces)

# Two Interfaces

def set_resources(self):
    interfaces = Interfaces(provider='sysconfig',
                            auto_reload=True)
    eth0 = Interface(name='eth0',
                     bootproto='dhcp')
    eth1 = Interface(name='eth0',
                     bootproto='static',
                     ipv4_address='10.1.1.1/24')
    interfaces.add(eth0, eth1)
    return Resource(interfaces)

# Simple Bond

def set_resources(self):
    interfaces = Interfaces(provider='netplan',
                            auto_reload=True)
    # Create Bond Interface Object
    bond = Bond(name="bond0",
                boot_proto="dhcp",
                mtu=9000,
                mode="802.3ad",
                xmit_hash_policy="layer3+4")
    Interfaces.Add(bond)
    # Create Slave Interface Objects
    for ifnum in range(1, 3):
        Interfaces.Add(Interface(name="eth{}".format(ifnum),
                                             slave=True,
                                             master=bond))
    return Resources(interfaces)

# A tagged vlan interface

def set_resources(self):
    interfaces = Interfaces(provider='sysconfig',
                            auto_reload=True)
    physical_interface = Interface(name="eth0",
                             boot_proto="none",)
    vlan_subinterface = SubInterface(vlan=1000,
                                     parent=physical_interface,
                                     boot_proto='dhcp')
    interfaces.add(physical_interface, vlan_subinterface)
    return Resources(interfaces)

I'm still not sure whether I like the idea of separate types or creating a single type with a "type" parameter. I guess you could do it that way, but it seems like validation could wind up being more complex.

class Interface(Type):

def __init__(self, name=None, **kwargs):
    self.setup(name=name, **kwargs)

def fields(self):
    return Fields(
        type = Field(kind=str, default='interface'),
        # Common Fields
        ipv4_address = Field(kind=str),
        ipv4_broadcast = FIeld(kind=str),
        # Bond specific options
        mode = Field(kind=str, default='active+passive'),
        miimon = Field(kind=str),
        # Slave specific options
        master = Field(kind=Interface) # Will this work?
    )

def validate(self):
    ERROR_MESSAGE = "A {} interface can't accept the '{}' parameter"
    bond_options = ['mode', 'miimon']
    slave_options = ['master']
    if self.type == 'bond':
        for i in slave_options:
            if getattr(self, i):
                raise SomeException(ERROR_MESSAGE.format(self.type, i))
    elif self.type == 'interface':
        for i in bond_options:
            if getattr(self, i):
                raise SomeException(ERROR_MESSAGE.format(self.type, i))

I could just imagine that getting pretty bloated with all of the Fields that would need to be added to cover my use cases.


#9

This is very helpful, thanks!!

I last did something with bonding for Cobbler like 10 years ago, so I'm really not super up to speed on all the latest. The VLAN tagging example ASCII helps me understand quite a bit, I understand what VLANs are but had never configured them for Linux networking.

Quick question: what would auto_reload on "interfaces" do?

Also I see you have both sysconfig and netplan, is it possible to always use sysconfig, even if there is netplan? Again, I haven't dug into netplan so I don't know, but I'm trying to see if there is a way where we don't end up with two providers with very different capabilities, because that's extra work - even if one of them is easier.

If your work has a need for one, but not the other, I'd be leaning on doing just the one.

I think most of the examples would read shorter if they are assigned to variables first, like:

Interfaces.Add(Interface(name="eth{}".format(ifnum),
                                             slave=True,
                                             master=bond))

... could be...

for n in range(1,3):
    intf = Interface(f'eth{n}', slave=True, master=bond)
    interfaces.add(intf)

also just not breaking the lines up makes it a little less intimidating IMHO?

Currently in the opsmop-demo repo there is a module_docs/ directory, and this code is used to generate all of the examples, so it would be pretty easy to translate all of these to executable examples. Because that execution would likely ruin the system you ran all three of them on, we could solve this by just not commenting out all the roles in "set_roles". But I think this makes excellent documentation for what is possible.

As for the type parameter vs subclassing proposal, we get the best mileage out of the Fields structure if they are subclassed instead and it will automatically not accept parameters that it can't use. To keep the whole provider tree organized we probably need a 'network/' subpackage under types. I understand what you are saying with the Fields structure being long, but I also see that if the Field structure WAS long, it's going to be longer if the types do three or four types of objects - which seems like a mess to me. It's also a bit less readable in the language rather than explicitly having something called "Bond", I think?

Also, I think I've said it, but I think interfaces is the only true type that can have providers. We may need to introduce the concept of an "abstract" type, so that if you throw it into a Resources() collection that produces a validation error. This is a minor nit though and could be added later.

Looking good!


#10

Reload the networking service after generating the config files. So more or less just eliminates the need for a handler since we already know which network service we are interacting with.

At the end of the day, its going to be up to the user to figure out which system their system is configured to use. As for sane defaults though, Ubuntu == netplan and RHEL derivitives == sysconfig.

That is my plan, I just want to make sure I don't shoot the person that is eventually going to write the other one in the foot :).

Agreed, I was a little bit on auto-pilot when I wrote those, I have a habit of hitting things with the format hammer that don't need to be.

Okay, we are on the same page here.

Yeah, I was going to ask about that. Makes sense.

I'll get started on it. I'll probably work on getting normal physical interfaces working first and then submit a PR for that piece.