Configuring Tomcat properties files with Augeas and Puppet.

Introduction

This post covers quite a few different things, it is taken from a real-world example of something I was asked to do recently which not only involved some cool Puppetmastery using exported resources, storeconfigs and custom definitions, but also forced me to start learning Augeas, which I’ve been meaning to get around to. So, heres the story.

Some background.

To put this into context, I recently had a requirement to add some Puppet configuration to manage some Tomcat properties files. On further investigation this turned out to be a little more complicated as the requirements weren’t as simple as chucking some templated configuration files out with a few variables replaced.

The requirement was for a group of Tomcat servers to contain one properties file with a chunk of configuration in for each server in the group. So for example, each node in the group needed to have something like

There could be many, many servers in a given group and I don’t want to be maintaining a massive list of variables as that will just get messy, so the answer here is to use exported virtual resources and Puppets’ storeconfig feature. My thinking here is that now I can configure each node in the group with an exported resource that looks something like

… and then simply collect them all with something like …

All good so far. Then I started thinking what application::instance() would look like. The requirement was for one properties file with all the nodes configuration in, so I can’t spit out several files from a template, that would be too easy. I looked around at various solutions for building files from fragments but to be honest nothing really appealed to me as elegant or clean, so I started investigating solutions for line-by-line editing. Traditionally this has been done by wrapping a series of piped echo commands and greps in a couple of exec’s, various examples of this exist, often called “line()”, but why do that when we have Augeas, a ready made tool for managing elements within a configuration file in a structured and controlled fashion.

So, I thought this would be worth experimenting with!

Creating an Augeas lens

Augeas uses the term lenses to describe a class that defines how it interacts with different types of files. There are lenses for all sorts of configuration files, such as hosts, httpd.conf, yum.repos.d…etc. You name it, there is probably a lens for it. At the time of writing however, Augeas is not bundled with a lens that can process Tomcat properties files, although I’ve been told this is coming out soon. Thinking that a tomcat properties file is pretty uncomplicated, I decided that instead of searching for someone elses pre-written version of a Tomcat lens I would write my own to gain a better understanding of how Augeas works.

My first thoughts when reading throught he Augeas documentation was, “Oh my god what have I got myself into”. I soon discovered that this was no simple little tool, and the configuration for lenses seemed immensely complicated. However, then I found this tutorial and ran through it. Slowly it started to make a bit more sense, and I realise that actually this is one powerful application.

Creating a test file

My first job was to create a test file, it’s useful to do this first as you’ll want to run augparse periodically to test your lens. The main function of the test file is to parse your example configuration into an augeas tree, and then vica versa and compare the outcomes.

My Tomcat test file looks like this

Here I’m testing a variety of scenarios, including indentations, spaces around “=” and comments. The first part tests that when I parse my configuration file using my lens that I get the expected tree, this is the get part. The second is the put part, this tests that setting a couple of variables in the augeas tree and parsing it back out as raw configuration will output in an expected manor, the augparse tool will use the lens I create to compare both of these outcomes and ensure my lens is doing what it should.

Creating the lens

At a very basic level, a lens describes a file. So before I started writing the lens for Tomcat properties file I thought about describing my file in plain English, and I came up with
Any one line is either a comment, a property or a blank lineA comment is a series of spaces/tabs followed by a hash, followed by textA property consists of alphanumerical values seperated by periodsA value is a string of textAn equals sign separates the property from the valueAny line can be indented with tabs and spacesWhite spaces or tabs can surround the separator

That doesnt seem so complicated, so then I thought of how I need to represent these in Augeas. Firstly, I thought about my primitive types here that I can use to build up a comment, a key/value pair and a blank line, the 3 functions of any one line. These break down toBlank lineEnd of lineSeparatorProperty name partValue partIndentation

Using these building blocks, I can define a comment, a key/value pair and a blank line, for example, a standard key/value pair line would be…

(spaces, tabs or null)(alphanumeric characters and periods)(spaces, tabs or null)(=)(spaces, tabs or null)(characters that are not end-of-line)(end-of-line)

So, when I write regular expressions to define the above, the Augeas configuration looks something like this.

Now I’ve defined my building blocks I can tell Augeas how these apply to the basic 3 elements of my configuration file; comments, blank lines and properties.

Finally I set up my lens and filter by telling Augeas that my lens consists of my 3 basic elements, and define which files I wish to be parsed using my lens

So my final lens file, which I install into /usr/share/augeas/lenses looks like this

Testing my lens

I use the augparse command to run my test file I created earlier against my new lens to make sure there are no parsing errors.

As I final test, I create a test.properties file with the following example configuration

Now I can use augtool to view and change one of my configurtation variables.

Pulling this into Puppet

Now I have a working lens, I can manipulate my configuration file using the augeas resource type provided by Puppet. First off, I want to build my tomcat property type using Augeas.

Now I have a custom definition of tomcat::property that I can implement in my application::instance type. My instance definition needs to be able to set several tomcat variables in the application.properties file, so now I can do the following:

Here I’m defining my application::instance type, and after that I include a resource collector to apply all exported definitions of my instance type. Finally, I just need to actually define the instances that I want to configure. Remember, each host in the group needs to know about every other host, so for each node I can now create something like the following and have it export it’s resource for all other nodes to collect.

Now, with a combination of exported virtual resources, custom definitions and augeas I have the solution.

If you want to use my Tomcat Augeas module for yourself, you can download it here