Recently, I've been in several conversations about how to reconcile two links in the management tool chain, configuration management and ad hoc control. The conversation usually revolves around conventional viewpoints about the nature and roles of these tools:

Configuration management wants to achieve and enforce a system state. The normal mode of operation is for the configuration management tool to run autonomously on each node, applying the rules of that node's specification. In the large, the set of hosts eventually converge on the declarations made in the specification.

Ad hoc control tools are generally used to for special problems or on-demand management activities. Their use is often arbitrary and driven by management need and not on a regular periodic basis. Problem diagnosis, information gathering, deployment changes, emergency fixes across many hosts are example cases where ad hoc control tools are often used.

If these tools are used independently, one or more scenarios like the following can arise:

• Changes made by ad hoc tools are reverted or partly undone by the configuration management tool
• Configuration management tool is turned off and only run directly when the rules are needed to fire
• Administration code is reinvented in each tool but their implementations focus on different concerns
• One tool gets thrown out or its role and scope are severely diminished in favor of the other

These tools can interfere with each other and to some they can appear mutually exclusive.

Clearly, both management modes are important. On the one hand, it's beneficial to have an ongoing compliance and enforcement loop driven by specification. On the other hand, issues of the day require near instantaneous action. Those actions may not become routine and deemed part of the desired permanent state maintained by configuration management.

Planning out how these two kinds of tools can interoperate begins with a discussion of boundaries and lines of responsibilities. It also helps interoperation when both tools share a common model and points of control.

Here are a few areas where I have seen interoperation facilitated:

• Centralize the control loop: Initiate the control loop from a central point of administration. Configuration management loses some autonomy but the pull model is preserved which is important for scalable execution. 
• Split Install from Startup: Separate the startup action from the installation procedures in the provisioning process. This defers startup to a centrally orchestrated authority. This is vital for multi-tier applications.
• System control interface: Define a set of system commands that control service state on each host. Configuration management and ad hoc tools can avoid reinventing the wheel. This also gives a point of collaboration between teams.
• Dependency driven packaging: Use packages to install files on node. Use package dependencies to install files in an order. System package utilities provide other benefits like verification and repositories.

Alex,

This is one of the reasons I'm creating Noah. While Chef and Puppet both can handle application level configuration perfectly well (for instance see Chef's data-driven configuration stuff), the requirement to perform a new Chef or Puppet run is pretty much a pain in the butt just to change a logging level or update a list of memcache nodes.

I would wager that 95% of users of either tools run them in an ad-hoc fashion because of concerns about changes applied to the CM repo going out "before they're tested".

Look at Wealthfront (formerly KaChing!) and how they use ZooKeeper (my inspiration for Noah) as a coordination tool for the entire infrastructure.

Noah/Zookeeper/whatever can happily coexist with existing CM tools as long as the system of record is determined to be the CM tool. Bootstrap a new memcached node via Chef? The base config file for your application gets updated but memcached startup (or some CM task) also registers the new node with Noah so that existing and running nodes can be instantly aware and reconfigure themselves.

Excellent post and something that I've been meaning to write about myself!

John,

I agree there is a missing link the chain. Having something like Noah/Zookeeper that can be used to store and access both configuration and operational state is extremely useful.

Alex