Puppet data from CouchDB using hiera-http

Puppet Training Courses

Introducing hiera-http

I started looking at various places people store data, and ways to fetch it and realized that a lot of data storage applications are RESTful, yet there doesn’t seem to be any support in Hiera to query these things, so I whipped up hiera-http, a Hiera back end to connect to any HTTP RESTful API and return data based on a lookup. It’s very new and support for other stuff like SSL and Auth is coming, but what it does support is a variety of handlers to parse the returned output of the HTTP doc, at the moment these are limited to YAML and JSON (or just ‘plain’ to simply return the whole response of the request). The following is a quick demonstration of how to plug CouchDB into Puppet using Hiera and the hiera-http backend.

Hiera-http is available as a rubygem, or from GitHub: http://github.com/crayfishx/hiera-http

CouchDB

Apache CouchDB is a scalable database that uses no set schema and is ideal for storing configuration data as everything is stored and retrieved as JSON documents. For the purposes of this demo I’m just going to do a very simple database installation with three documents and a few configuration parameters to demonstrate how to integrate this in with Puppet.

After installing couchdb and starting the service I’m able to access Futon, the web GUI front-end for my couchdb service – using this I create three documents, “dev”, “common” and “puppet.puppetlabs.lan”

CouchDB documents

CouchDB documents

Next I populate my common and dev documents with some variables.

Common document populated with data

Now CouchDB is configured I should be able to query the data over HTTP

[root@puppet ~]# telnet 192.168.93.138 5984
Trying 192.168.93.138...
Connected to 192.168.93.138 (192.168.93.138).
Escape character is '^]'.
GET /configuration/common HTTP/1.0
 
HTTP/1.0 200 OK
Server: CouchDB/1.0.2 (Erlang OTP/R12B)
Etag: "2-2ffb42b336d8d920ef46ac1865c491aa"
Date: Mon, 29 Oct 2012 18:45:48 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 137
Cache-Control: must-revalidate
 
{"_id":"common","_rev":"2-2ffb42b336d8d920ef46ac1865c491aa","port":80,"ntpserver":"10.0.0.1","dnsservers":["192.168.0.1","192.168.0.2"]}
Connection closed by foreign host.
[root@puppet ~]#

Query with Hiera

After installing hiera-http I can query this data directly from Hiera…

# gem install hiera-http

First I need to configure Hiera with the HTTP back end. The search hierarchy is determined by the :paths: configuration parameter and since CouchDB returns JSON I set that as the output handler.

---

:backends: ['http']

:http:
  :host: 192.168.93.138
  :port: 5984
  :output: json
  :failure: graceful
  :paths:
    - /configuration/%{fqdn}
    - /configuration/%{environment}
    - /configuration/common

I can now query this directly from Hiera on the command line

[root@puppet /]# hiera dnsservers
["192.168.0.1", "192.168.0.2"]
 
[root@puppet /]# hiera ntpserver
10.0.0.1

And of course, that means that this data is now available from Puppet and if I add some overriding configuration variables to my dev document in CouchDB, my lookup will resolve based on my environment setting in Puppet

$ntpserver=hiera('ntpserver')
notify { $ntpserver: }

[root@puppet /]# puppet apply test.pp
notice: 10.0.0.1
notice: /Stage[main]//Notify[10.0.0.1]/message: defined 'message' as '10.0.0.1'
notice: Finished catalog run in 0.04 seconds
 
[root@puppet /]# puppet apply test.pp --environment=dev
notice: 192.168.100.101
notice: /Stage[main]//Notify[192.168.100.101]/message: defined 'message' as '192.168.100.101'
notice: Finished catalog run in 0.04 seconds

Hiera-http is fully featured and supports all standard Hiera back end functions such as hiera_hash, hiera_array order overrides.

Future stuff

I’m going to carry on working on new features for hiera-http – including basic auth, HTTPS/SSL, proxys and a wider variety of output handlers – I would like for this back end to be flexible enough to allow users to configure Hiera with any network service that uses a RESTful API to perform data lookups. Keep watching.

Share

3 comments - What do you think?  Posted by craig - November 21, 2012 at 4:03 pm

Categories: hiera, Puppet   Tags: , , , , ,

I’m joining Puppet Labs

Puppet Training Courses

Back in September last year I flew over to the US for PuppetConf 2011, and more recently took a trip to Edinburgh for PuppetCamp. Both times when I returned my wife asked me how it went, and both times my answer was simple; “I’ve gotta work for these guys!” I said. So after recently returning from a short excursion to Portland, Oregon I am very thrilled and honoured to announce that I’ve accepted a full time position with Puppet Labs.

I’ve been an IT contractor for many years now, and started working with Puppet in 2008. Since then I’ve worked more and more with Puppet at numerous companies including most recently at the BBC. I’ve also become increasingly more involved in the Puppet user community at large over the past 4 years. I have a real passion for working with the product, I love the Puppet community and now I’m really looking forward to joining the company that made it all happen and being a part of all the awesome things to come.

I’ll be joining Puppet Labs later this month as a Professional Services Engineer and look forward to maintaining and building upon the many relationships I have with various Puppet users as well as engaging with a wider section of the user community in my new professional capacity too. I would like to thank the various people at Puppet Labs involved in my interviews for the opportunity, with special thanks extended to Aimee Fahey (@PuppetRecruiter) for all her assistance during the hiring process.

Share

4 comments - What do you think?  Posted by craig - June 6, 2012 at 10:07 am

Categories: Uncategorized   Tags:

Designing Puppet – Roles and Profiles.

Puppet Training Courses

Update, Feb 15th.

Since writing this post some of the concepts have become quite popular and have generated quite a lot of comments and questions in the community. I recently did I talk at Puppet Camp Stockholm on this subject and hopefully I might have explained it a bit better there than I did below :-). The slides are available here and a YouTube video will be uploaded shortly.

Introduction

So you’ve installed Puppet, downloaded some forge modules, probably written a few yourself too. So, now what? You start applying those module to your nodes and you’re well on your way to super-awesomeness of automated deployments. Fast forward a year or so and your infrastructure has grown considerably in size, your organisations business requirements have become diverse and complex and your architects have designed technical solutions to solve business problems with little regard for how they might actually be implemented. They look great in the diagrams but you’ve got to fit in to Puppet. From personal experience, this often leads to a spell of fighting with square pegs and round holes, and the if statement starts becoming your go-to guy because you just can’t do it any other way. You’re probably now thinking its time to tear down what you’ve got and re-factor. Time to think about higher level design models to ease the pain.

There is a lot of very useful guidance in the community surrounding Puppet design patterns for modules, managing configurable data and class structure but I still see people struggling with tying all the components of their Puppet manifests together. This seems to me to be an issue with a lack of higher level code base design. This post tries to explain one such design model that I refer to as “Roles/Profiles” that has worked quite well for me in solving some off the more common issues encountered when your infrastructure grows in size and complexity, and as such, the requirements of good code base design become paramount.

The design model laid out here is by no means my suggestion on how people should design Puppet, it’s an example of a model that I’ve used with success before. I’ve seen many varied designs, some good and some bad, this is just one of them – I’m very interested in hearing other design models too. The point of this post is to demonstrate the benefits of adding an abstraction layer before your modules

What are we trying to solve

I’ve spent a lot of time trying to come up with what I see as the most common design flaws in Puppet code bases. One source of problems is that users spend a lot of time designing great modules, then include those modules directly to the node. This may work but when dealing with large and complex infrastructures this becomes cumbersome and you end up with a lot of node level logic in your manifests.

Consider a network consisting of multiple different server types. They will all share some common configuration, some subsets of servers will also share configuration while other configuration will be applicable only to that server type. In this very simple example we have three server types. A development webserver (www1) that requires a local mysql instance and PHP logging set to debug, a live webserver (www2) that doesn’t use a local mysql, requires memcache and has standard PHP logging, and a mail server (smtp1). If you have a flat node/module relationship with no level of abstraction then your nodes file starts to look like this:

node www1 { 
 include networking
 include users
 include tomcat
 include jdk
 include mysql
 include memcache
 include apache
 class { "php": 
    loglevel  => "debug"
 }
}
 
node www2 { 
  include networking
  include users
  include tomcat
  include jdk
  include memcache
  include apache
  include php
}
 
node smtp1 { 
  include networking
  include users
  include exim
}

Note: if you’re already thinking about ENC’s this will be covered later

As you can see, the networking and users modules are universal across all our boxes, Apache, Tomcat and JDK is used for all webservers, some webservers have mysql and PHP logging options vary depending on what type of webserver it is.

At this point most people try and simplify their manifests by using node inheritance. In this very simple example that might be sufficient, but it’s only workable up to a point. If you’re environment grows to hundreds or even thousands of servers, made up over 20 or 30 different types of server, some with shared attributes and subtle differences, spread out over multiple environments, you will likely end up with an unmanagable tangled web of node inheritance. Nodes also can inherit only one other node, which will be restrictive in some edge cases.

Adding higher level abstraction

One way I have found to minimise the complexity of node definitions and make handling nuances between different server types and edge case scenarios a lot easier is to add a layer (or in this case, two layers) of seperation between my nodes and the modules they end up calling. I refer to these as roles and profiles.

Consider for a moment how you would represent these servers if you weren’t writing a Puppet manifest. You wouldn’t say “www1 is a server that has mysql, tomcat, apache, PHP with debug logging, networking and users” on a high level network diagram. You would more likely say “www1 is a dev web server” so really this is all the information I want to be applying directly to my node.

So after analysing all our nodes we’ve come up with three distinct definitions of what a server can be. A development webserver, a live webserver and a mailserver. These are your server roles, they describe what the server represents in the real world. In this design model a node can only ever have one role, it cant be two things simultaneously. If your business now has an edge case for QA webservers to be the same as live servers, but incorporate some extra software for performance testing, then you’ve just defined another role, a QA Webserver.

Now we look at what a role should contain. If you were describing the role “Development webserver” you would likely say “A development webserver has a Tomcat application stack, a webserver and a local database server”. At this level we start defining profiles.

Unlike roles, which are named in a more human representation of the server function, a profile incorporates individual components to represent a logical technology stack. In the above example, the profile “Tomcat application stack” is made up of the Tomcat and JDK components, whereas the webserver profile is made up of the httpd, memcache and php components. In Puppet, these lower level components are represented by your modules.

Classes

 

Now our nodes definitions look a lot simpler and are representitive of their real world roles…

node www1 { 
  include role::www::dev
}
 
node www2 { 
  include role::www::live
}
 
node smtp1 { 
  include role::mailserver
}

Roles are simply collections of profiles that provide a sensible mapping between human logic and technology logic. In this scenario our roles may look something like:

class role { 
  include profile::base
}
 
class role::www inherits role { 
  # All WWW servers get tomcat
  include profile::tomcat
}
 
class role::www::dev inherits role::www { 
  include profile::webserver::dev
  include profile::database
}
 
class role::www::live inherits role::www { 
  include profile::webserver::live
}
 
class role::mailserver inherits role { 
  include profile::mailserver
}

Whether or not you choose to use inherited classes in the way I have done is up to you of course, some people stay clear of inheritence completely, others over use it. Personally I think it works for the purposes of laying out roles and profiles to minimise duplication.

The profiles included above would look something like the following

class profile::base { 
  include networking
  include users 
}
 
class profile::tomcat { 
  class { "jdk": } 
  class { "tomcat": } 
}
 
class profile::webserver { 
  # Configuration for all webservers
  class { "httpd": } 
  class { "php": } 
  class { "memcache": } 
}
 
class profile::webserver::dev inherits profile::webserver { 
  Class["php"] { 
    loglevel   => "debug"
  }
}
 
class profile::webserver::live inherits profile::webserver { 
  # Any live webserver specific stuff here 
}
 
class profile::database {
  class { "mysql": } 
}
 
class profile::mailserver { 
  class { "exim": } 
}

In summary the “rules” surrounding my design can be simplified as;

  • A node includes one role, and one only.
  • A role includes one or more profiles to define the type of server
  • A profile includes and manages modules to define a logical technical stack
  • Modules manage resources
  • Modules should only be responsible for managing aspects of the component they are written for
  • Let’s just clarify what we mean by “modules”

    I’ve talked about profiles and roles like they are some special case and modules being something else. In reality, all of these classes can be, and should be modularised. I make a logical distinction between the profile and role modules, and everything else (e.g.: modules that provide resources).

    Other useful stuff to do with profiles.

    So far I’ve demonstrated using profiles as collections of modules, but it has other uses too. As a rule of thumb, I don’t define any resources directly from roles or profiles, that is the job for my modules. However, I do realise virtualised resources and occasionally do resource chaining in profiles which can solve problems that otherwise would have meant editing modules and other functionality that doesn’t quite fit in the scope of an individual module. Adding some of this functionality at the modular level will reduce the re-usability and portability of your module.

    Hypothetically lets say I have a module, let’s call it foo for originalities sake. The foo module provides a service type called foo, in my implementation I have another module called mounts that declares some mount resource types. If I want all mount resource types to be initiated before the foo service is started as without the filesystems mounted the foo service will fail. I’ll go even further and say that foo is a Forge module that I really don’t want to (and shouldn’t have to) edit, so where do I put this configuration? This is where having the profiles level of abstraction is handy. The foo module is coded perfectly, it’s the use case determined from my own technology stack that is requiring that my mount points exists before the foo service, so since my stack is defined in the profile, this is where I should specify it. e.g.:

    class profile::foo { 
      include mounts
      include foo
      Mount <| |>  ->  Service['foo']
    }

    It’s widely known that good modules are modules that you don’t need to edit. Quite often I see people reluctant to use Forge modules because their set up requires some peripheral set up or dependancies not included in the module. Modules exist to manage resources directly related to what they were written for. For example, someone may choose to edit a forge mysql module because their set up has dependancies on MMM being installed after MySQL (purely hypothetical). The mysql module is not the place to do this, mysql and mmm are separate entities and should be configured and contained within their own modules, tying the two together is something you’ve defined in your stack, so again, this is where you’re profiles come in…

    class profile::database { 
      class { "mysql": }
      class { "mmm": } 
      Package["mysql"] -> Package["mmm"]
    }

    This approach is also potentially helpful for those using Hiera. Although Hiera and Puppet are to become much more fused in Puppet 3.0, at the moment people writing forge modules have to make them work with Hiera or not, and people running Hiera have to edit the modules that aren’t enabled. Take a hypothetical module from the Forge called fooserver. This module exposes a paramaterized class that has an option for port, I want to source this variable from Hiera but the module doesn’t support it. I can add this functionality into the profile without the need for editing the module.

    class profile::fooserver { 
      $fooport = hiera("fooserver_port")
      class { "fooserver": 
        port  => $fooport 
      }
    }

    What about using an ENC?

    So you’re probaby wondering why I haven’t mentioned using an ENC (External Node Classifier). The examples above don’t use any kind of ENC, but the logic behind adding a layer of separation between your nodes and your modules is still the same. You could decide to use an ENC to determine which role to include to a node, or you could build/configure an ENC to perform all the logic and return the list of components (modules) to include. I prefer using an ENC in place of nodes definitions to determine what role to include and keep the actual roles and profiles logic within Puppet. My main reason for this is that I get far greater control of things such as resource chaining, class overrides and integration with things like Hiera at the profile level and this helps overcome some tricky edge cases and complex requirements.

    Summary

    None of the above is set in stone, what I hope I’ve demonstrated though is that adding a layer of abstraction in your Puppet code base design can have some significant benefits that will avoid pitfalls when you start dealing with extremely complex, diverse and large scale set ups. These include

  • Reducing complexity of configuration at a node level
  • Real-world terminology of roles improves “at-a-glance” visibility of what a server does
  • Definition of logical technology stacks (profiles) gives greater flexibility for edge cases
  • Profiles provide an area to add cross-module functionality such as resource chaining
  • Modules can be granular and secular and tied together in profiles, thus reducing the need to edit modules directly
  • Reduced code duplication
  • I use Hiera to handle all of my environment configuration data, which I won’t go into detail about in this post. So, at a high level my Puppet design can be represented as;

    Puppet Layout

    As I said previously, this is not a the way to design Puppet, but an example of one such way. The purpose of this post is to explore higher level code base design for larger and more complex implementations of Puppet, I would love to hear other design models that people have used either successfully or not and what problems it solved for you (or introduced :)) so please get in touch with your own examples.

    Share

    6 comments - What do you think?  Posted by craig - May 23, 2012 at 8:08 pm

    Categories: Puppet, Uncategorized   Tags:

    Introducing hiera-mysql MySQL Backend for Hiera

    Puppet Training Courses

    Introduction

    Some time ago I started looking at Hiera, a configuration datastore with pluggable back ends that also plugs seamlessly into Puppet for managing variables. When I wrote hiera-gpg a few months ago I realised how easy extending Hiera was and the potential for really useful backends that can consolidate all your configuration options from a variety of systems and locations into one streamlined process that systems like Puppet and other tools can hook into. This, fuelled by a desire to learn more Ruby, lead to hiera-mysql, a MySQL Backend for Hiera.

    Installing

    hiera-mysql is available as a ruby gem and can be installed with:

    # gem install hiera-mysql

    Note: this depends on the Ruby mysql gem, so you’ll need gcc, ruby-devel and mysql-devel packages installed. Alternativley the source can be Downloaded here

    MySQL database

    To demonstrate hiera-mysql, here’s a simple MySQL database some sample data;

    mysql> use mytest;
    Database changed
    mysql> SHOW CREATE TABLE configuration \G
    *************************** 1. row ***************************
           Table: configuration
    Create Table: CREATE TABLE `configuration` (
      `id` int(11) NOT NULL AUTO_INCREMENT,
      `var` varchar(255) DEFAULT NULL,
      `val` varchar(255) DEFAULT NULL,
      `env` varchar(255) DEFAULT NULL,
      PRIMARY KEY (`id`)
    ) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=latin1
    1 row in set (0.00 sec)
     
    mysql> SELECT * FROM configuration;
    +----+-------+--------+------+
    | id | var   | val    | env  |
    +----+-------+--------+------+
    |  1 | color | green  | live |
    +----+-------+--------+------+
    1 rows in set (0.00 sec)

    Configuring Hiera

    In this example we’re going to pass the variable “env” in the scope. hiera-mysql will interpret any scope variables defined in the query option, and also has a special case for %{key}. Example:

    # /etc/hiera.yaml
    ---
    :backends: - mysql
    
    :logger: console
    
    :mysql:
        :host: localhost
        :user: root
        :pass: examplepassword
        :database: mytest
        :query: SELECT val FROM configuration WHERE var='%{key}' AND env='%{env}'

    Running Hiera

    With the above example, I want to find the value of the variable colour in the scope of live

    [root@dev1 ~]# hiera color env=live
    green

    If I add more rows to the database that match the criteria, and use Hiera’s array search function by passing -a I can make Hiera return all the rows

     
    mysql> INSERT INTO configuration VALUES (2,'color','purple','live');
    Query OK, 1 row affected (0.00 sec)
     
     
    [root@dev1 ~]# hiera color env=live -a 
    ["green", "purple"]

    Hiera’s pluggable nature means that you can use this back end alongside other back ends such as YAML or JSON and configure your search order accordingly.

    Limitations

    Currently hiera-mysql will only return the first element of a row, or an array of first elements, so you can’t do things like SELECT foo,bar FROM table. I intend to introduce this feature by implementing Hiera’s hash search in a future release. Also, the module could do with slightly better exception handling around the mysql stuff. Please let me know if theres anything else that would improve it.

    Puppet

    And of course, because Hiera is completely transparent, accessing these variables from Puppet couldn’t be easier!

    class foo { 
        $env="live"
        $color=hiera("color")
    }

    References

  • Github homepage for hiera-mysql
  • Official Hiera Project Homepage
  • Hiera – A pluggable hierarchical data store
    Share
  • 5 comments - What do you think?  Posted by craig - March 3, 2012 at 12:53 pm

    Categories: hiera, Linux, Puppet, Uncategorized   Tags: ,

    Secret variables in Puppet with Hiera and GPG

    Puppet Training Courses

    Last week I wrote an article on Puppet configuration variables and Hiera. This almost sorted out all my configuration variable requirements, bar one; what do I do with sensitive data like database passwords, hashed user passwords…etc that I don’t want to store in my VCS repo as plaintext.

    Hiera allows you to quite easily add new backends, so I came up with hiera-gpg, a backend plugin for Hiera that will GPG decrypt a YAML file on the fly. It’s quite minimal and there is some stuff I’d like to do better – for instance it currently shells out to the GPG command, hopefully someone has some code they can contribute that’ll use the GPGME gem instead to do the encryption bit.

    Once you’re up and running with Hiera, you can get the hiera-gpg backend from Rubygems…

    # gem install hiera-gpg

    We run several Puppetmasters, so for each one I create a GPG key and add the public key to a public keyring that’s kept in my VCS repo. For security reasons I maintain a dev and a live keyring so only live Puppetmasters can see live data.

    # gpg --gen-key

    Currently hiera-gpg doesn’t support key passwords, I’ll probably add this feature in soon but it would mean having the password stored in /etc/puppet/hiera.yaml as plaintext anyway, so I don’t see that as adding much in the way of security.

    So I have my GPG secret key set up in roots homedir:

    # gpg --list-keys
    /root/.gnupg/pubring.gpg
    ------------------------
    pub   1024D/1F77BE20 2011-10-06
    uid                  puppet.live.mycorp.com (Live Puppet) <root@puppet.live.mycorp.com>
    sub   2048g/CB8EE07E 2011-10-06

    Next I add my GPG public key to the keyring for live puppetmasters (in my set up, /etc/puppet/keyrings/live is a subversion checkout)

    # gpg --homedir=/etc/puppet/keyrings/live --import ~/.gnupg/pubkey.txt 
    gpg: key 1F77BE20: public key "puppet.live.mycorp.com (Live Puppet) <root@puppet.live.mycorp.com>" imported
    gpg: Total number processed: 1
    gpg:               imported: 1

    Now I can create a YAML file in my hieradata folder and encrypt it for the servers in my live keyring.

    # cd /etc/puppet/hieradata/live
    # cat mysql.yaml
    ---
    rootpwd: jhns732ns
    # gpg --trust-model=always --homedir=/etc/puppet/keyrings/live --encrypt -o mysql.gpg -r puppet.live.mycorp.com mysql.yaml
    # rm mysql.yaml
    # ls
    mysql.yaml.gpg

    If like me you have more than one puppetmaster in your live keyrings, multiple -r entries can be specified on the command line for gpg, you should encrypt your file for all the puppet master keys that are allowed to decrypt it.

    Now you just need to tell Hiera about the GPG backend, My previous Hiera configuration now becomes:

    # cat /etc/puppet/hiera.yaml 
    ---
    :backends: - yaml
               - gpg
    
    :logger: console
    
    :hierarchy: - %{env}/%{location}/%{calling_module}
                - %{env}/%{calling_module}
                - common
     
    
    :yaml:
       :datadir: /etc/puppet/hieradata
    
    :gpg:
       :datadir: /etc/puppet/hieradata

    Here we’re telling Hiera to behave exactly as it used to when we just had the YAML back end, and if it doesn’t find the value you are requesting from YAML it will query the GPG back end which will pick up on your %{calling_module}.gpg.

    Now I can query Hiera on the command line to find my live MySQL root password with:

    # hiera -c /etc/puppet/hiera.yaml rootpwd calling_module=mysql env=live
    jhns732ns

    In Puppet, I reference my variables in exactly the same way as any other variable

    class mysql ($rootpwd=hiera("rootpwd")) {

    Theres probably lots of stuff I can improve here, but I have the basics of what I need, a transparent method of data storage using GPG encryption and no sensitive data stored in my VCS repo as plain text.

    Share

    8 comments - What do you think?  Posted by craig - October 10, 2011 at 4:43 pm

    Categories: Puppet, Uncategorized   Tags: , , ,

    Puppet configuration variables and Hiera.

    Puppet Training Courses

    Managing configuration variables within Puppet has always given me a bit of a headache, and I’ve never really found a way to do it that I’m all together happy with, particularly when dealing with the deployment of complex applications that require a lot, sometimes hundreds, of different configuration variables and multiple environments. I started thinking a while ago that Puppet wasn’t the best place to be keeping these variables in the first place. For starters, this is really valuable data we’re talking about, there may be lots of other applications that may benefit from having access to the way your software is configured, so why should Puppet retain all of this information exclusively for itself? The original extlookup() function in Puppet provides some decoupling of configuration data from Puppet manifests, but I found it a bit limiting and not very elegant having to maintain a bunch of CSV files. I’ve been interested in R.I.Pienaar’s Hiera for a while and thought I’d give it a proper spin and see if it meets my needs.

    Hiera itself is a standalone configuration data store that supports multiple back ends including YAML, JSON and Puppet itself, and adding more back ends to it is a fairly non-challenging task for anyone competent with Ruby. Thanks to hiera-puppet it plugs nicely into Puppet.

    Configuring a basic Hiera setup

    After installing hiera (gem install hiera), I want to test it by setting up a pretty basic configuration store that will override my configuration variables based on environment settings of dev, stage or live. Let’s take a variable called $webname. I want to set it correctly in each of my three environments, or default it to localhost.

    Firstly, I create four YAML files in /etc/puppet/hieradata

    [root@localhost hieradata]# cat dev.yaml
    ---
    webname: dev.app.local
     
     
    [root@localhost hieradata]# cat stage.yaml
    ---
    webname: stage.app.mydomain.com
     
     
    [root@localhost hieradata]# cat live.yaml
    ---
    webname: www.app.mydomain.com
     
     
    [root@localhost hieradata]# cat common.yaml
    ---
    webname: localhost

    Now I have a YAML file representitive of each environment, I create a simple config in /etc/puppet/hiera.yaml that tells Hiera to search for my environment YAML file followed by common.yaml.

    ---
    :backends: - yaml
    
    :logger: console
    
    :hierarchy: - %{env}
                - common
    :yaml:
       :datadir: /etc/puppet/hieradata

    Now using hiera from the command line, I can look up the default value of $webname with the following command

    [root@localhost puppet]# hiera -c /etc/puppet/hiera.yaml webname
    localhost

    But now if I want to know the value for the live and dev environments I can pass an env flag to Hiera

    [root@localhost puppet]# hiera -c /etc/puppet/hiera.yaml webname env=dev
    dev.app.local
    [root@localhost puppet]# hiera -c /etc/puppet/hiera.yaml webname env=live
    www.app.mydomain.com

    Accessing this from Puppet

    I can now access these variables directly from my Puppet modules using the hiera() function provided by hiera-puppet. In this example, I already have a fact called ${::env} that is set to dev, stage or live (in my particular set up we use the puppet environment variable for other things)

    class myapplication ( $webname = hiera("webname") ) { 
    ...
    }

    Adding more scoping

    OK, thats a fairly simple set up but demonstrates how easy it is to get up and running with Hiera. The requirements I had were a little more complex. Firstly, our hierarchy is broken down into both environment (live, stage, dev..etc) and location. I have multiple environments in multiple locations, a particular location will either be a live, stage or dev environment. So some variables I want to override on the environment level, and some at the more granular location level.

    Secondly, I don’t like the idea of asking Hiera for $webname. That doesn’t tell me anything; what is $webname, what uses it? Consider a more generic variable called $port – that’s going to be confusing. So I started thinking about ways of grouping and scoping my variables. The way I solved this was to introduce a module parameter as well as environment and location in Hiera and place variables for a particular module in it’s own YAML file, using a filesystem layout to determine the hierarchy.

    My new hierdata file system looks a little like this

    |-- dev
    |   |-- glasgow
    |   `-- london
    |-- live
    |   |-- birmingham
    |   |-- london
    |   `-- dublin
    `-- stage
        |-- birmingham
        |-- london
        `-- dublin
    

    Now for each of my modules, I create a YAML file in the folder level that I want to override with the values for my module. Taking the previous example, lets say that I want $webname to be www.myapp.mycorp.com for all live environments, except for Dublin, which I want to be a special case of www.myapp.mycorp.ie. To accomplish this I create the following two files:

    [root@localhost hieradata]# cat live/myapplication.yaml 
    ---
    webname: www.myapp.mycorp.com
     
     
    [root@localhost hieradata]# cat live/dublin/myapplication.yaml 
    ---
    webname: www.myapp.mycorp.ie

    Hiera-puppet will pass the value of $calling_module from Puppet to Hiera, and we can use this in our hierarchy in hiera.yaml. NOTE: Currently you will need this patch to hiera-puppet in order for this to work!

    So our new /etc/puppet/hiera.yaml file looks like:

    ---
    :backends: - yaml
    
    :logger: console
    
    :hierarchy: - %{env}/%{location}/%{calling_module}
                - %{env}/%{calling_module}
                - common
    :yaml:
       :datadir: /etc/puppet/hieradata

    On the command line, we can now see that environment, location and calling module are now used when looking up a configuration variable

    [root@localhost hieradata]# hiera -c /etc/puppet/hiera.yaml webname env=live calling_module=myapplication 
    www.myapp.mycorp.com
     
    [root@localhost hieradata]# hiera -c /etc/puppet/hiera.yaml webname env=live calling_module=myapplication location=london
    www.myapp.mycorp.com
     
    [root@localhost hieradata]# hiera -c /etc/puppet/hiera.yaml webname env=live calling_module=myapplication location=dublin
    www.myapp.mycorp.ie

    In Puppet, I have ${::env} and ${::location} already set as facts, and since $calling_module will get automatically passed to Hiera from Puppet, my myapplication class looks no different…

    class myapplication ( $webname = hiera("webname") ) { 
    ...
    }

    but knowing the module name means I can easily find where this value is set, and I can easily see what configuration a module requires by examining its YAML files under /etc/puppet/hierdata

    Conclusion

    In conclusion, I’m now convinced that moving configuration variable data out of Puppet is a very good thing. Now other systems can easily query this valuable information either on the command line or directly with Ruby. By forcing the use of $calling_module I’ve introduced a sort of pseudo scoping for my variables, so, for example… “port” now becomes “port calling_module=apache” and gives me a lot more meaning.

    Many thanks to R.I.Pienaar for help in setting this up, as well as providing the patch to scope.rb that enabled me to get this working.

    Share

    5 comments - What do you think?  Posted by craig - October 5, 2011 at 1:31 pm

    Categories: Linux, Puppet   Tags: , ,

    Puppet, Parameterized classes .vs. Definitions

    Puppet Training Courses

    Firstly, a little background on this topic. During PuppetConf this year I attended a very interesting talk by Digant C. Kasundra about the Puppet implementation at Stanford University. At one point he asked, “Who uses definitions?”, and I raised my hand. The next question was, “Who uses parameterized classes?”, and I also raised my hand, this was followed by “Who uses both?”, and I was one of a small minority of people who raised a hand again. The next question was, “Who knows what the difference between a parameterized class and a definition is?”. I took a second or two to think about this and the talk moved on after no-one in the audience raised their hand, and I didn’t get a chance to answer this in the Q&A due to running out of time, but I’ve been thinking about it since. Digant’s view was that the two are very similar and he advocates the use of definitions, which is certainly not a bad point of view, but I don’t think you should be using one or the other, but rather, use either one appropriately, and given the fact that no-one could answer Digant’s question in his talk, I felt it worth expanding on the issue in a blog post and would really welcome any feedback.

    So, what is the core differences in how they are used? Well, the answer is, as Digant rightly pointed out, not a lot, but the small difference between them that does exist is very important, and should easily dictate which one you use in a given situation. Firstly, let’s look at what they actually are;

    A class, parameterized or not, is a grouping of resources (including resources provided by definitions) that can be included into a manifest with one statement. If you make your classs parameterized then you can include that class with dynamic parameters that you can override depending on how your catalog is compiled. A definition is a template that defines what effectively is no different from any other resource type and gives you a boiler plate solution for applying a series of resources in a certain way.

    So now it’s fairly clear that these two things are actually quite different, but you may be thinking, “sure, they’re different but there is nothing that a parameterized class does that you can’t do with a definition, right?” – well, yes, that’s right, but there is plenty that a definition does that you may not want it to do, mainly, puppet allowing it to be instantiated multiple times.

    As an example of this, at the BBC we have a core application, let’s call it acmeapp for semantics sake. The core application takes almost 50 different parameters for configuration and can get deployed differently depending on what type of server profile we are deploying to. We only ever want acmeapp defined once as it is responsible for some core resources such as creating the base install directories, therefore we use a parameterized class for this and sub classes can inherit of this and override any number of variables. Either way, I’m only ever going to apply the acmeapp class once in my catalog. The end result looks something like;

     
    class profiles::acmeapp  { 
        class { "::acmeapp" : } 
    }
     
    .....
     
    class profiles::acmeapp::sometype inherits profiles::acmeapp { 
        Class["::acmeapp"] { 
            somevar    => "customvalue",
        }
    }

    Whilst the above is possible with a definition, it would allow it to be defined multiple times, which means the resources within the definition would end up duplicated and my catalog failing. I would rather Puppet fails because I’ve tried to use my class twice, rather than fail because I’ve duplicated a resource, such as the file type that creates the installation directory. It makes it very clear to anyone working with my Puppet code that this class should only be applied once.

    Now, lets look at another example. At multiple points in our manifests we need to set up MySQL grants – to initiate an Exec for every grant would be bad practice, as we’ll end up in a lot of code re-use. This is where definitions come in. At the BBC we have a mysql class that not only provides the MySQL packages and services, but also exposes some useful functions to manage your databases and grants through a series of definitions, this is the code to the definition that controls MySQL grants….

    class mysql { 
     
    ...
     
            # Definition: mysql::grant
            # 
            # Description: Add a grant/user to MySQL
            # 
            # Usage: 
            #    mysql::grant { "username":
            #               db              => "databasename",
            #               password        => "password"
            #    }
            # 
            # Options:
            #       db              - Name of the database
            #       password        - Password for the grant
            #       tables          - Tables to grant privileges (default: *)
            #       privs           - List of privileges to grant (default: ALL PRIVILEGES)
            #       host            - Host to grant privs to (default: localhost)
            #       user            - Specify username here to override the resource title default
            #       dbhost          - Specify the mysql server to connect to, different to "host"
            # 
            # 
            # 
            define grant (
                            $db,
                            $password,
                            $privs          = "ALL PRIVILEGES",
                            $tables         = "*",
                            $host           = "localhost",
                            $user           = false,
                            $dbhost         = false,
                            $grantopt       = false
     
     
            ) {
     
                    $sqlgrantopt = $grantopt ? {
                            false   => "",
                            true    => " WITH GRANT OPTION",
                            default => ""
                    }
     
                    $myuser = $user ? {
                            false   => $title,
                            default => $user
                    }
     
                    # Determine if we need to pass a -h argument to mysql client
                    # to tell it where to set the database up - grant root permissions
                    # on the mysql server must be set up before this is run
                    # If no host is given MySQL will assume localhost
                    #
                    $hostflag = $dbhost ? {
                            false   => "",
                            default => "-h $dbhost"
                    }
     
     
                    $sql = "GRANT $privs ON $db.$tables TO $myuser@'$host' IDENTIFIED BY '$password' $sqlgrantopt"
                    $grantsql = "SHOW GRANTS FOR '$myuser'@'$host'"
                    $grantgrep = "grep \"GRANT $privs ON \\`$db\\`.$tables TO '$myuser'@'$host'$sqlgrantopt\""
     
                    $mysql = "mysql -u root -p$mysql::mysql_root_pw $hostflag"
                    exec { "mysql::grant::$db::$title":
                            path    => "/usr/bin:/sbin:/usr/sbin:/bin",
                            command => "$mysql -e \"$sql\"",
                            unless  => "$mysql -e \"$grantsql\" | $grantgrep",
                            require => Package["mysql"],
                    }
            }
    }

    As you can see, this is very different to our usage or a parameterized class, here we’ve templated a significant amount of functionality into one easy-to-use defined resource type. We can re-use this functionality as many times as we need, in the above example we set up two grants for the acmeapp application, one for a VIP IP address and one for an application IP range by specifying something like:

            ## Setup grants.
            mysql::grant {
                            "acmeapp_vip":
                                    user            => "acmeapp",
                                    host            => $vip_ip,
                                    db              => "acmeapp",
                                    dbhost          => $dbhost,
                                    password        => "acmepass",
                                    grantopt        => true,
                                    require         => Mysql::Database["acmeapp"];
     
                            "acmeapp_ip_range":
                                    user            => "acmeapp",
                                    host            => $app_ip_range,
                                    db              => "acmeapp",
                                    dbhost          => "$dbhost",
                                    password        => "acmepass",
                                    grantopt        => true,
                                    require         => Mysql::Database["acmeapp"];
            }

    Hopefully this gives you a good idea of the subtle differences between parameterized classes and definitions, but also that they are both very independent features of Puppet that have their own uses.

    Share

    2 comments - What do you think?  Posted by craig - September 24, 2011 at 3:35 am

    Categories: Puppet, Uncategorized   Tags:

    Configuring Tomcat properties files with Augeas and Puppet.

    Puppet Training Courses

    Introduction

    This post covers quite a few different things, it is taken from a real-world example of something I was asked to do recently which not only involved some cool Puppetmastery using exported resources, storeconfigs and custom definitions, but also forced me to start learning Augeas, which I’ve been meaning to get around to. So, heres the story.

    Some background.

    To put this into context, I recently had a requirement to add some Puppet configuration to manage some Tomcat properties files. On further investigation this turned out to be a little more complicated as the requirements weren’t as simple as chucking some templated configuration files out with a few variables replaced.

    The requirement was for a group of Tomcat servers to contain one properties file with a chunk of configuration in for each server in the group. So for example, each node in the group needed to have something like

    application.server1.host=server1
    application.server1.uuid=98726252
    application.server1.name=Server One
     
    application.server2.host=server2
    application.server2.uuid=98272727
    application.server2.name=Server Two
    ... 
     
    application.server199.host=server199
    application.server199.uuid=897234983
    application.server199.name=Server One Hundred and Ninety Nine
     
    ... and so on, you get the picture.

    There could be many, many servers in a given group and I don’t want to be maintaining a massive list of variables as that will just get messy, so the answer here is to use exported virtual resources and Puppets’ storeconfig feature. My thinking here is that now I can configure each node in the group with an exported resource that looks something like

    @@application::instance { "server1": 
           uuid     => "909282982",
           descr   =>  "Server 1",
    }

    … and then simply collect them all with something like …

    Application::Instance <<| |>>

    All good so far. Then I started thinking what application::instance() would look like. The requirement was for one properties file with all the nodes configuration in, so I can’t spit out several files from a template, that would be too easy. I looked around at various solutions for building files from fragments but to be honest nothing really appealed to me as elegant or clean, so I started investigating solutions for line-by-line editing. Traditionally this has been done by wrapping a series of piped echo commands and greps in a couple of exec’s, various examples of this exist, often called “line()”, but why do that when we have Augeas, a ready made tool for managing elements within a configuration file in a structured and controlled fashion.

    So, I thought this would be worth experimenting with!

    Creating an Augeas lens

    Augeas uses the term lenses to describe a class that defines how it interacts with different types of files. There are lenses for all sorts of configuration files, such as hosts, httpd.conf, yum.repos.d…etc. You name it, there is probably a lens for it. At the time of writing however, Augeas is not bundled with a lens that can process Tomcat properties files, although I’ve been told this is coming out soon. Thinking that a tomcat properties file is pretty uncomplicated, I decided that instead of searching for someone elses pre-written version of a Tomcat lens I would write my own to gain a better understanding of how Augeas works.

    My first thoughts when reading throught he Augeas documentation was, “Oh my god what have I got myself into”. I soon discovered that this was no simple little tool, and the configuration for lenses seemed immensely complicated. However, then I found this tutorial and ran through it. Slowly it started to make a bit more sense, and I realise that actually this is one powerful application.

    Creating a test file

    My first job was to create a test file, it’s useful to do this first as you’ll want to run augparse periodically to test your lens. The main function of the test file is to parse your example configuration into an augeas tree, and then vica versa and compare the outcomes.

    My Tomcat test file looks like this

    module Test_tomcat =
            let conf = "
    # Test tomcat properties file
    #tomcat.commented.value=1
            # config
    tomcat.port = 8080
    tomcat.application.name=testapp
            tomcat.application.description=my test application
    "
     
    test Tomcat.lns get conf =
            { }
            { "#comment" = "Test tomcat properties file" }
            { "#comment" = "tomcat.commented.value=1" }
            { "#comment" = "config" }
            { "tomcat.port" = "8080" }
            { "tomcat.application.name" = "testapp" }
            { "tomcat.application.description" = "my test application" }
     
    test Tomcat.lns put conf after
            set "tomcat.port" "99";
            set "tomcat.application.host" "foo.network.com"
            = "
    # Test tomcat properties file
    #tomcat.commented.value=1
            # config
    tomcat.port = 99
    tomcat.application.name=testapp
            tomcat.application.description=my test application
    tomcat.application.host=foo.network.com
    "

    Here I’m testing a variety of scenarios, including indentations, spaces around “=” and comments. The first part tests that when I parse my configuration file using my lens that I get the expected tree, this is the get part. The second is the put part, this tests that setting a couple of variables in the augeas tree and parsing it back out as raw configuration will output in an expected manor, the augparse tool will use the lens I create to compare both of these outcomes and ensure my lens is doing what it should.

    Creating the lens

    At a very basic level, a lens describes a file. So before I started writing the lens for Tomcat properties file I thought about describing my file in plain English, and I came up with

  • Any one line is either a comment, a property or a blank line
  • A comment is a series of spaces/tabs followed by a hash, followed by text
  • A property consists of alphanumerical values seperated by periods
  • A value is a string of text
  • An equals sign separates the property from the value
  • Any line can be indented with tabs and spaces
  • White spaces or tabs can surround the separator

    That doesnt seem so complicated, so then I thought of how I need to represent these in Augeas. Firstly, I thought about my primitive types here that I can use to build up a comment, a key/value pair and a blank line, the 3 functions of any one line. These break down to

  • Blank line
  • End of line
  • Separator
  • Property name part
  • Value part
  • Indentation

    Using these building blocks, I can define a comment, a key/value pair and a blank line, for example, a standard key/value pair line would be…

    (spaces, tabs or null)(alphanumeric characters and periods)(spaces, tabs or null)(=)(spaces, tabs or null)(characters that are not end-of-line)(end-of-line)

    So, when I write regular expressions to define the above, the Augeas configuration looks something like this.

            (* Define some basic primitives *)
            let empty               = Util.empty
            let eol                 = del /[ \t]*\n/ "\n"
            let sepch               = del /[ \t]*=[ \t]*/ "="
            let value_to_eol        = /[^ \t\n](.*[^ \t\n])?/
            let indent              = del /^[ \t]*/ ""
            let entry               = /[A-Za-z][A-Za-z0-9\.]+/

    Now I’ve defined my building blocks I can tell Augeas how these apply to the basic 3 elements of my configuration file; comments, blank lines and properties.

            (* define comments and properties*)
            let comment             = [ label "#comment" . indent . del /#[ \t]*/ "#" . store value_to_eol . eol ]
            let property            = [ indent . key entry . sepch . store value_to_eol . eol ]

    Finally I set up my lens and filter by telling Augeas that my lens consists of my 3 basic elements, and define which files I wish to be parsed using my lens

            (* setup our lens and filter*)
            let lns                 = ( property | empty | comment  ) *
            let filter = incl "/opt/tomcat/webapps/conf/*.properties"
                            . Util.stdexcl

    So my final lens file, which I install into /usr/share/augeas/lenses looks like this

    (* Augeas module for editing tomcat properties files
     Author: Craig Dunn <craig@craigdunn.org>
    *)
     
     
    module Tomcat =
            autoload xfm
     
     
            (* Define some basic primitives *)
            let empty               = Util.empty
            let eol                 = del /[ \t]*\n/ "\n"
            let sepch               = del /[ \t]*=[ \t]*/ "="
            let value_to_eol        = /[^ \t\n](.*[^ \t\n])?/
            let indent              = del /^[ \t]*/ ""
            let entry               = /[A-Za-z][A-Za-z0-9\.]+/
     
     
            (* define comments and properties*)
            let comment             = [ label "#comment" . indent . del /#[ \t]*/ "#" . store value_to_eol . eol ]
            let property            = [ indent . key entry . sepch . store value_to_eol . eol ]
     
     
     
            (* setup our lens and filter*)
            let lns                 = ( property | empty | comment  ) *
            let filter = incl "/opt/tomcat/webapps/conf/*.properties"
                            . Util.stdexcl
     
            let xfm = transform lns filter

    Testing my lens

    I use the augparse command to run my test file I created earlier against my new lens to make sure there are no parsing errors.

     # cd /usr/share/augeas/lenses
     # augparse tests/test_tomcat.aug 
     #

    As I final test, I create a test.properties file with the following example configuration

    application.port=80
     application.user=tomcat

    Now I can use augtool to view and change one of my configurtation variables.

    # augtool
     
    augtool> ls /files/opt/tomcat/webapps/conf/test.properties/
    application.port = 80
    application.user = tomcat
     
    augtool> set /files/opt/tomcat/webapps/conf/test.properties/application.port 443
    augtool> save
    Saved 1 file(s)
    augtool> quit
     
     # cat /opt/tomcat/webapps/conf/test.properties 
    application.port=443
     application.user=tomcat






    Pulling this into Puppet

    Now I have a working lens, I can manipulate my configuration file using the augeas resource type provided by Puppet. First off, I want to build my tomcat property type using Augeas.

     
    class tomcat { 
    ....
     
            define property (
                    $variable       = false,
                    $value          = "",
                    $target
            ) {
     
                    $prop = $variable ? {
                            false   => $title,
                            default => $variable
                    }
     
                    augeas { "$target/$prop/$value":
                            context => "/files${target}",
                            changes => "set $prop $value",
                    }
     
            }
    }

    Now I have a custom definition of tomcat::property that I can implement in my application::instance type. My instance definition needs to be able to set several tomcat variables in the application.properties file, so now I can do the following:

    class application { 
    ...
            define instance (
                            $uuid,
                            $descr
            ) {
                    Tomcat::Property { 
                        target       => "/opt/tomcat/webapps/conf/app.properties" 
                    }
     
                    tomcat::property { "application.$title.uuid": value     => "$uuid" }
                    tomcat::property { "application.$title.name": value     => "$descr"}
                    tomcat::property { "application.$title.host": value     => "$title"}
            }
     
            Application::Instance <<| |>>
     
    }

    Here I’m defining my application::instance type, and after that I include a resource collector to apply all exported definitions of my instance type. Finally, I just need to actually define the instances that I want to configure. Remember, each host in the group needs to know about every other host, so for each node I can now create something like the following and have it export it’s resource for all other nodes to collect.

            @@application::instance { "server1": 
                    uuid    => "987234987239487293487234987",
                    descr    => "Server 1",
            }

    Now, with a combination of exported virtual resources, custom definitions and augeas I have the solution.

    If you want to use my Tomcat Augeas module for yourself, you can download it here

    Share
  • 3 comments - What do you think?  Posted by craig - June 22, 2011 at 1:01 pm

    Categories: Linux, Puppet   Tags:

    Preventing users from changing their password with PAM

    Puppet Training Courses

    Blocking AD users from using passwd

    I had to design a system recently for a client which has a mixture of local users and remote users that are authenticated using LDAP against Active Directory (actually, with Quest Authentication Services running in between). One of the requirements was that AD users should not be able to change their password using the passwd command as they had an external management system for users that fed into AD (and other things). I needed to allow normal users to operate normally but fail AD users with some polite message to tell them what was going on, rather than just a random error that would cause them to call support every time. Trolling the web didn’t seem to reveal much apart from doing nasty things to /bin/passwd like chattr’ing it, or moving it to /sbin… since we’re not in the 90′s anymore I was sure there was a way to do this with PAM.

    My PAM knowledge is limited to say the least, and maybe my google-fu isn’t up to much because I struggled to find anything that did exactly what I wanted.

    Eventually, after some tweaking, I came up with the following which seems to work on CentOS…

    Edit /etc/pam.d/passwd and change it to read :-

     #%PAM-1.0
     
    password        requisite   pam_cracklib.so retry=3
    password        sufficient  pam_unix.so use_authtok
    password        required       pam_echo.so \
                           You CANNOT change your password using the Linux passwd command
    password        required       pam_echo.so \
                           You must change your windows password in Active Directory
    password        required     pam_deny.so

    This should work normally for root and local users but give a warning and fail to AD users.

    Share

    Be the first to comment - What do you think?  Posted by craig - March 31, 2011 at 3:38 pm

    Categories: Howtos, Linux   Tags: ,

    Puppet – working with define based virtuals

    Puppet Training Courses

    Define Based Virtuals

    Define-based virtuals are quite a powerful feature of Puppet, but many people either don’t understand them or don’t know how to apply them effectivly to their manifests. Due to Puppet’s normalized configuration structure, you can’t configure the same resource in two different places. For example, if the user mysql is configured in your database class, and then you realise that your webserver class also needs the mysql user, then, short of including the whhole database class, you cannot just configure the user here. The answer to this lies in resource virtualization.

    Resources that are virtualized are effectivly non-real, they won’t be included in any configuration until they are explicitly realized. The good news is you can realize your resource multiple times in your manifests, so by defining the resource as virtual in a seperate class, both your webserver class and databasse class can realize it.

    Version 0.23 of Puppet took this one step further, with the ability to virtualize your definitions rather than just native resource types. Now we can define a collection of resources, virtualize them, and realize them at whim in our manifests. I’ve seen a few Puppet installations and this method is far under used, and under appreciated.

    To demonstrate, this is an example of how to manage your users with virtual define-based resources.

    User management with defines and virtuals.

    I’ve seen a few ways of managing users within a Puppet estate. The following is a quick guide to get you started managing users, passwords and ssh keys for your users. My favourite approach is to use define-based virtuals and split it out into several sections

    • Back end module to specify the definition for a user
    • A class with a list of virtualized users
    • Realizing your users ad-hoc in your classes

    The advantage of this approach is that you can keep a central place for users whilst still being able to easily pick and choose which users get deployed with which classes. Lets break it down into our components;

    Users module

    As we want to manage users as well as ssh keys it makes sense to wrap this up into a definition, which we’ll call localuser. We create a module called users::virtual where we define the following class.

    class users::virtual {
            define localuser ($uid,$gid,$pass,$sshkey="") {
     
                    user { $title:
                            ensure  =>      "present",
                            uid     =>      $uid,
                            gid     =>      $gid,
                            shell   =>      "/bin/bash",
                            home    =>      "/home/$title",
                            comment =>      $realname,
                            password =>     $pass,
                            managehome =>   true,
                    }
     
                    if ( $sshkey != "" ) {
                     ssh_authorized_key { $title:
                             ensure  =>      "present",
                             type    =>      "ssh-rsa",
                             key     =>      "$sshkey",
                             user    =>      "$title",
                             require =>      User["$title"],
                             name    =>      "$title",
                     }
                    }
            }
    }

    This class sets up our localuser definition in users::virtual, other resources can also be added here that will affect every user.

    Creating our users class.

    Now you can create a users.pp class that imports your module and defines the specific users you want to configure. We configure these as virtuals to give us the option of realizing them whenever we want.

    class mysite::users {
    include users::virtual
     
            @users::virtual::localuser { "bobsmith":
                    uid     =>      "3202",
                    gid     =>      "users",
                    pass    =>      '$1$fU8c0mlIjDFYCRu0U+r1',
                    sshkey  =>      "AAAAB3NzaC1yc2EAAAABIwAAAQEAtbsafzNX08oT63vnKh6LNYVpFM9U42knt+tUMvhTQaOEGVnsRH6zVQj86PLYo9HD7MCVqYAloKRN6hVvoqU++CSLO0zUYsQ4bX/+DQthtKcOwU76QLFTcXVRIIGMH++GLHGjphEhjPAJc/rPM0YswCetOm3JVGVB9x/WJFOmoT+a7r4IXaULaNTYZOPZ6fr/CvUB/w3NBvPnmLMxwPFOgBLxcQ9Tbpa5sjwi1thlXl1ZfQ8Sh++gg60odTHbAhwZOU70mA8WGOmkuETDQzunQvTK14fGDvFSHJNE5nYse8IPChbfrSMJl1PsWB+SiiGrPVQtly9BEOYi/aOokj3vfQ==",
            }
     
    }

    The passord should be the fully encrypted string as would appear in the shadow file, don’t forget to use single quotes if your password string contains $ to prevent it from being interpereted as a variable.

    sshey is optional (note the conditional in our users module), and, if defined will set up the key in ~/.ssh/authorized_keys. Now you have a central list of users as virtuals that call our custom define, they can be easily included, or realized, at whim in your other classes.

    Realize the user

    Within your manifests for your server profile, whenever you require the user bobsmith to be part of the configuration, you simply realize it like this

    class server::web {
            include mysite::users
            realize (
                    Users::Virtual::Localuser["bobsmith"],
            )
    }

    The above example realizes the user by the name of “bobsmith”. Another very useful way to realise users is by using Puppet’s collection syntax. For instance, to realize all the users in the users group, you would simply replace the realize statement with:

            User <| group == users |>

    … and all your virtual users that match the gid of users will be included.

    Conclusion

    What we’ve acheived by using virtual define-based resources here is primarily,

    • An abstracted module for defining what a user is, rather than who
    • A central place to manage all of our virtual users
    • A very flexible and easy way to pull users into our classes based on name or attribute

    For more information on working with Puppet virtuals, check out the documentation here

    Share

    1 comment - What do you think?  Posted by craig - March 21, 2011 at 9:19 pm

    Categories: Howtos, Linux, Puppet   Tags: ,

    Next Page »