Puppet data from CouchDB using hiera-http
Introducing hiera-http
I started looking at various places people store data, and ways to fetch it and realized that a lot of data storage applications are RESTful, yet there doesn’t seem to be any support in Hiera to query these things, so I whipped up hiera-http, a Hiera back end to connect to any HTTP RESTful API and return data based on a lookup. It’s very new and support for other stuff like SSL and Auth is coming, but what it does support is a variety of handlers to parse the returned output of the HTTP doc, at the moment these are limited to YAML and JSON (or just ‘plain’ to simply return the whole response of the request). The following is a quick demonstration of how to plug CouchDB into Puppet using Hiera and the hiera-http backend.
Hiera-http is available as a rubygem, or from GitHub: http://github.com/crayfishx/hiera-http
CouchDB
Apache CouchDB is a scalable database that uses no set schema and is ideal for storing configuration data as everything is stored and retrieved as JSON documents. For the purposes of this demo I’m just going to do a very simple database installation with three documents and a few configuration parameters to demonstrate how to integrate this in with Puppet.
After installing couchdb and starting the service I’m able to access Futon, the web GUI front-end for my couchdb service – using this I create three documents, “dev”, “common” and “puppet.puppetlabs.lan”
CouchDB documents
CouchDB documents
Next I populate my common and dev documents with some variables.
Common document populated with data
Now CouchDB is configured I should be able to query the data over HTTP
[root@puppet ~]# telnet 192.168.93.138 5984
Trying 192.168.93.138…
Connected to 192.168.93.138 (192.168.93.138).
Escape character is ‘^]’.
GET /configuration/common HTTP/1.0
HTTP/1.0 200 OK
Server: CouchDB/1.0.2 (Erlang OTP/R12B)
Etag: “2-2ffb42b336d8d920ef46ac1865c491aa”
Date: Mon, 29 Oct 2012 18:45:48 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 137
Cache-Control: must-revalidate
{“_id”:”common”,”_rev”:”2-2ffb42b336d8d920ef46ac1865c491aa”,”port”:80,”ntpserver”:”10.0.0.1″,”dnsservers”:[“192.168.0.1″,”192.168.0.2”]}
Connection closed by foreign host.
[root@puppet ~]#
Query with Hiera
After installing hiera-http I can query this data directly from Hiera…
# gem install hiera-http
First I need to configure Hiera with the HTTP back end. The search hierarchy is determined by the :paths: configuration parameter and since CouchDB returns JSON I set that as the output handler.
—
:backends: [‘http’]
:http:
:host: 192.168.93.138
:port: 5984
:output: json
:failure: graceful
:paths:
– /configuration/%{fqdn}
– /configuration/%{environment}
– /configuration/common
I can now query this directly from Hiera on the command line
[root@puppet /]# hiera dnsservers
[“192.168.0.1”, “192.168.0.2”]
[root@puppet /]# hiera ntpserver
10.0.0.1
And of course, that means that this data is now available from Puppet and if I add some overriding configuration variables to my dev document in CouchDB, my lookup will resolve based on my environment setting in Puppet
$ntpserver=hiera(‘ntpserver’)
notify { $ntpserver: }
[root@puppet /]# puppet apply test.pp
notice: 10.0.0.1
notice: /Stage[main]//Notify[10.0.0.1]/message: defined ‘message’ as ‘10.0.0.1’
notice: Finished catalog run in 0.04 seconds
[root@puppet /]# puppet apply test.pp –environment=dev
notice: 192.168.100.101
notice: /Stage[main]//Notify[192.168.100.101]/message: defined ‘message’ as ‘192.168.100.101’
notice: Finished catalog run in 0.04 seconds
Hiera-http is fully featured and supports all standard Hiera back end functions such as hiera_hash, hiera_array order overrides.
Future stuff
I’m going to carry on working on new features for hiera-http – including basic auth, HTTPS/SSL, proxys and a wider variety of output handlers – I would like for this back end to be flexible enough to allow users to configure Hiera with any network service that uses a RESTful API to perform data lookups. Keep watching.