Puppet data from CouchDB using hiera-http

Introducing hiera-http

I started looking at various places people store data, and ways to fetch it and realized that a lot of data storage applications are RESTful, yet there doesn’t seem to be any support in Hiera to query these things, so I whipped up hiera-http, a Hiera back end to connect to any HTTP RESTful API and return data based on a lookup. It’s very new and support for other stuff like SSL and Auth is coming, but what it does support is a variety of handlers to parse the returned output of the HTTP doc, at the moment these are limited to YAML and JSON (or just ‘plain’ to simply return the whole response of the request). The following is a quick demonstration of how to plug CouchDB into Puppet using Hiera and the hiera-http backend.

Hiera-http is available as a rubygem, or from GitHub: http://github.com/crayfishx/hiera-http

CouchDB

Apache CouchDB is a scalable database that uses no set schema and is ideal for storing configuration data as everything is stored and retrieved as JSON documents. For the purposes of this demo I’m just going to do a very simple database installation with three documents and a few configuration parameters to demonstrate how to integrate this in with Puppet.

After installing couchdb and starting the service I’m able to access Futon, the web GUI front-end for my couchdb service – using this I create three documents, “dev”, “common” and “puppet.puppetlabs.lan”

CouchDB documents
CouchDB documents

Next I populate my common and dev documents with some variables.

Common document populated with data

Now CouchDB is configured I should be able to query the data over HTTP

Query with Hiera

After installing hiera-http I can query this data directly from Hiera…

First I need to configure Hiera with the HTTP back end. The search hierarchy is determined by the :paths: configuration parameter and since CouchDB returns JSON I set that as the output handler.

I can now query this directly from Hiera on the command line

And of course, that means that this data is now available from Puppet and if I add some overriding configuration variables to my dev document in CouchDB, my lookup will resolve based on my environment setting in Puppet

Hiera-http is fully featured and supports all standard Hiera back end functions such as hiera_hash, hiera_array order overrides.

Future stuff

I’m going to carry on working on new features for hiera-http – including basic auth, HTTPS/SSL, proxys and a wider variety of output handlers – I would like for this back end to be flexible enough to allow users to configure Hiera with any network service that uses a RESTful API to perform data lookups. Keep watching.

Follow and share if you liked this

Designing Puppet – Roles and Profiles.

Update, Feb 15th.

Since writing this post some of the concepts have become quite popular and have generated quite a lot of comments and questions in the community. I recently did I talk at Puppet Camp Stockholm on this subject and hopefully I might have explained it a bit better there than I did below :-). The slides are available here and a YouTube video will be uploaded shortly.

Introduction

So you’ve installed Puppet, downloaded some forge modules, probably written a few yourself too. So, now what? You start applying those module to your nodes and you’re well on your way to super-awesomeness of automated deployments. Fast forward a year or so and your infrastructure has grown considerably in size, your organisations business requirements have become diverse and complex and your architects have designed technical solutions to solve business problems with little regard for how they might actually be implemented. They look great in the diagrams but you’ve got to fit in to Puppet. From personal experience, this often leads to a spell of fighting with square pegs and round holes, and the if statement starts becoming your go-to guy because you just can’t do it any other way. You’re probably now thinking its time to tear down what you’ve got and re-factor. Time to think about higher level design models to ease the pain.

There is a lot of very useful guidance in the community surrounding Puppet design patterns for modules, managing configurable data and class structure but I still see people struggling with tying all the components of their Puppet manifests together. This seems to me to be an issue with a lack of higher level code base design. This post tries to explain one such design model that I refer to as “Roles/Profiles” that has worked quite well for me in solving some off the more common issues encountered when your infrastructure grows in size and complexity, and as such, the requirements of good code base design become paramount.

The design model laid out here is by no means my suggestion on how people should design Puppet, it’s an example of a model that I’ve used with success before. I’ve seen many varied designs, some good and some bad, this is just one of them – I’m very interested in hearing other design models too. The point of this post is to demonstrate the benefits of adding an abstraction layer before your modules

What are we trying to solve

I’ve spent a lot of time trying to come up with what I see as the most common design flaws in Puppet code bases. One source of problems is that users spend a lot of time designing great modules, then include those modules directly to the node. This may work but when dealing with large and complex infrastructures this becomes cumbersome and you end up with a lot of node level logic in your manifests.

Consider a network consisting of multiple different server types. They will all share some common configuration, some subsets of servers will also share configuration while other configuration will be applicable only to that server type. In this very simple example we have three server types. A development webserver (www1) that requires a local mysql instance and PHP logging set to debug, a live webserver (www2) that doesn’t use a local mysql, requires memcache and has standard PHP logging, and a mail server (smtp1). If you have a flat node/module relationship with no level of abstraction then your nodes file starts to look like this:

Note: if you’re already thinking about ENC’s this will be covered later

As you can see, the networking and users modules are universal across all our boxes, Apache, Tomcat and JDK is used for all webservers, some webservers have mysql and PHP logging options vary depending on what type of webserver it is.

At this point most people try and simplify their manifests by using node inheritance. In this very simple example that might be sufficient, but it’s only workable up to a point. If you’re environment grows to hundreds or even thousands of servers, made up over 20 or 30 different types of server, some with shared attributes and subtle differences, spread out over multiple environments, you will likely end up with an unmanagable tangled web of node inheritance. Nodes also can inherit only one other node, which will be restrictive in some edge cases.

Adding higher level abstraction

One way I have found to minimise the complexity of node definitions and make handling nuances between different server types and edge case scenarios a lot easier is to add a layer (or in this case, two layers) of seperation between my nodes and the modules they end up calling. I refer to these as roles and profiles.

Consider for a moment how you would represent these servers if you weren’t writing a Puppet manifest. You wouldn’t say “www1 is a server that has mysql, tomcat, apache, PHP with debug logging, networking and users” on a high level network diagram. You would more likely say “www1 is a dev web server” so really this is all the information I want to be applying directly to my node.

So after analysing all our nodes we’ve come up with three distinct definitions of what a server can be. A development webserver, a live webserver and a mailserver. These are your server roles, they describe what the server represents in the real world. In this design model a node can only ever have one role, it cant be two things simultaneously. If your business now has an edge case for QA webservers to be the same as live servers, but incorporate some extra software for performance testing, then you’ve just defined another role, a QA Webserver.

Now we look at what a role should contain. If you were describing the role “Development webserver” you would likely say “A development webserver has a Tomcat application stack, a webserver and a local database server”. At this level we start defining profiles.

Unlike roles, which are named in a more human representation of the server function, a profile incorporates individual components to represent a logical technology stack. In the above example, the profile “Tomcat application stack” is made up of the Tomcat and JDK components, whereas the webserver profile is made up of the httpd, memcache and php components. In Puppet, these lower level components are represented by your modules.

 

puppet

Now our nodes definitions look a lot simpler and are representitive of their real world roles…

Roles are simply collections of profiles that provide a sensible mapping between human logic and technology logic. In this scenario our roles may look something like:

Whether or not you choose to use inherited classes in the way I have done is up to you of course, some people stay clear of inheritence completely, others over use it. Personally I think it works for the purposes of laying out roles and profiles to minimise duplication.

The profiles included above would look something like the following

In summary the “rules” surrounding my design can be simplified as;

  • A node includes one role, and one only.
  • A role includes one or more profiles to define the type of server
  • A profile includes and manages modules to define a logical technical stack
  • Modules manage resources
  • Modules should only be responsible for managing aspects of the component they are written for

Let’s just clarify what we mean by “modules”

I’ve talked about profiles and roles like they are some special case and modules being something else. In reality, all of these classes can be, and should be modularised. I make a logical distinction between the profile and role modules, and everything else (e.g.: modules that provide resources).

Other useful stuff to do with profiles.

So far I’ve demonstrated using profiles as collections of modules, but it has other uses too. As a rule of thumb, I don’t define any resources directly from roles or profiles, that is the job for my modules. However, I do realise virtualised resources and occasionally do resource chaining in profiles which can solve problems that otherwise would have meant editing modules and other functionality that doesn’t quite fit in the scope of an individual module. Adding some of this functionality at the modular level will reduce the re-usability and portability of your module.

Hypothetically lets say I have a module, let’s call it foo for originalities sake. The foo module provides a service type called foo, in my implementation I have another module called mounts that declares some mount resource types. If I want all mount resource types to be initiated before the foo service is started as without the filesystems mounted the foo service will fail. I’ll go even further and say that foo is a Forge module that I really don’t want to (and shouldn’t have to) edit, so where do I put this configuration? This is where having the profiles level of abstraction is handy. The foo module is coded perfectly, it’s the use case determined from my own technology stack that is requiring that my mount points exists before the foo service, so since my stack is defined in the profile, this is where I should specify it. e.g.:

It’s widely known that good modules are modules that you don’t need to edit. Quite often I see people reluctant to use Forge modules because their set up requires some peripheral set up or dependancies not included in the module. Modules exist to manage resources directly related to what they were written for. For example, someone may choose to edit a forge mysql module because their set up has dependancies on MMM being installed after MySQL (purely hypothetical). The mysql module is not the place to do this, mysql and mmm are separate entities and should be configured and contained within their own modules, tying the two together is something you’ve defined in your stack, so again, this is where you’re profiles come in…

This approach is also potentially helpful for those using Hiera. Although Hiera and Puppet are to become much more fused in Puppet 3.0, at the moment people writing forge modules have to make them work with Hiera or not, and people running Hiera have to edit the modules that aren’t enabled. Take a hypothetical module from the Forge called fooserver. This module exposes a paramaterized class that has an option for port, I want to source this variable from Hiera but the module doesn’t support it. I can add this functionality into the profile without the need for editing the module.

What about using an ENC?

So you’re probaby wondering why I haven’t mentioned using an ENC (External Node Classifier). The examples above don’t use any kind of ENC, but the logic behind adding a layer of separation between your nodes and your modules is still the same. You could decide to use an ENC to determine which role to include to a node, or you could build/configure an ENC to perform all the logic and return the list of components (modules) to include. I prefer using an ENC in place of nodes definitions to determine what role to include and keep the actual roles and profiles logic within Puppet. My main reason for this is that I get far greater control of things such as resource chaining, class overrides and integration with things like Hiera at the profile level and this helps overcome some tricky edge cases and complex requirements.

Summary

None of the above is set in stone, what I hope I’ve demonstrated though is that adding a layer of abstraction in your Puppet code base design can have some significant benefits that will avoid pitfalls when you start dealing with extremely complex, diverse and large scale set ups. These include

  • Reducing complexity of configuration at a node level
  • Real-world terminology of roles improves “at-a-glance” visibility of what a server does
  • Definition of logical technology stacks (profiles) gives greater flexibility for edge cases
  • Profiles provide an area to add cross-module functionality such as resource chaining
  • Modules can be granular and secular and tied together in profiles, thus reducing the need to edit modules directly
  • Reduced code duplication

I use Hiera to handle all of my environment configuration data, which I won’t go into detail about in this post. So, at a high level my Puppet design can be represented as;

puppet_big

 

As I said previously, this is not a the way to design Puppet, but an example of one such way. The purpose of this post is to explore higher level code base design for larger and more complex implementations of Puppet, I would love to hear other design models that people have used either successfully or not and what problems it solved for you (or introduced :)) so please get in touch with your own examples.

Follow and share if you liked this

Introducing hiera-mysql MySQL Backend for Hiera

Introduction

Some time ago I started looking at Hiera, a configuration datastore with pluggable back ends that also plugs seamlessly into Puppet for managing variables. When I wrote hiera-gpg a few months ago I realised how easy extending Hiera was and the potential for really useful backends that can consolidate all your configuration options from a variety of systems and locations into one streamlined process that systems like Puppet and other tools can hook into. This, fuelled by a desire to learn more Ruby, lead to hiera-mysql, a MySQL Backend for Hiera.

Installing

hiera-mysql is available as a ruby gem and can be installed with:

Note: this depends on the Ruby mysql gem, so you’ll need gcc, ruby-devel and mysql-devel packages installed. Alternativley the source can be Downloaded here

MySQL database

To demonstrate hiera-mysql, here’s a simple MySQL database some sample data;

Configuring Hiera

In this example we’re going to pass the variable “env” in the scope. hiera-mysql will interpret any scope variables defined in the query option, and also has a special case for %{key}. Example:

Running Hiera

With the above example, I want to find the value of the variable colour in the scope of live

If I add more rows to the database that match the criteria, and use Hiera’s array search function by passing -a I can make Hiera return all the rows

Hiera’s pluggable nature means that you can use this back end alongside other back ends such as YAML or JSON and configure your search order accordingly.

Limitations

Currently hiera-mysql will only return the first element of a row, or an array of first elements, so you can’t do things like SELECT foo,bar FROM table. I intend to introduce this feature by implementing Hiera’s hash search in a future release. Also, the module could do with slightly better exception handling around the mysql stuff. Please let me know if theres anything else that would improve it.

Puppet

And of course, because Hiera is completely transparent, accessing these variables from Puppet couldn’t be easier!

References

  • Github homepage for hiera-mysql
  • Official Hiera Project Homepage
  • Hiera – A pluggable hierarchical data store
    Follow and share if you liked this
  • Secret variables in Puppet with Hiera and GPG

    Last week I wrote an article on Puppet configuration variables and Hiera. This almost sorted out all my configuration variable requirements, bar one; what do I do with sensitive data like database passwords, hashed user passwords…etc that I don’t want to store in my VCS repo as plaintext.

    Hiera allows you to quite easily add new backends, so I came up with hiera-gpg, a backend plugin for Hiera that will GPG decrypt a YAML file on the fly. It’s quite minimal and there is some stuff I’d like to do better – for instance it currently shells out to the GPG command, hopefully someone has some code they can contribute that’ll use the GPGME gem instead to do the encryption bit.

    Once you’re up and running with Hiera, you can get the hiera-gpg backend from Rubygems…

    We run several Puppetmasters, so for each one I create a GPG key and add the public key to a public keyring that’s kept in my VCS repo. For security reasons I maintain a dev and a live keyring so only live Puppetmasters can see live data.

    Currently hiera-gpg doesn’t support key passwords, I’ll probably add this feature in soon but it would mean having the password stored in /etc/puppet/hiera.yaml as plaintext anyway, so I don’t see that as adding much in the way of security.

    So I have my GPG secret key set up in roots homedir:

    Next I add my GPG public key to the keyring for live puppetmasters (in my set up, /etc/puppet/keyrings/live is a subversion checkout)

    Now I can create a YAML file in my hieradata folder and encrypt it for the servers in my live keyring.

    If like me you have more than one puppetmaster in your live keyrings, multiple -r entries can be specified on the command line for gpg, you should encrypt your file for all the puppet master keys that are allowed to decrypt it.

    Now you just need to tell Hiera about the GPG backend, My previous Hiera configuration now becomes:

    Here we’re telling Hiera to behave exactly as it used to when we just had the YAML back end, and if it doesn’t find the value you are requesting from YAML it will query the GPG back end which will pick up on your %{calling_module}.gpg.

    Now I can query Hiera on the command line to find my live MySQL root password with:

    In Puppet, I reference my variables in exactly the same way as any other variable

    Theres probably lots of stuff I can improve here, but I have the basics of what I need, a transparent method of data storage using GPG encryption and no sensitive data stored in my VCS repo as plain text.

    Follow and share if you liked this

    Puppet configuration variables and Hiera.

    Managing configuration variables within Puppet has always given me a bit of a headache, and I’ve never really found a way to do it that I’m all together happy with, particularly when dealing with the deployment of complex applications that require a lot, sometimes hundreds, of different configuration variables and multiple environments. I started thinking a while ago that Puppet wasn’t the best place to be keeping these variables in the first place. For starters, this is really valuable data we’re talking about, there may be lots of other applications that may benefit from having access to the way your software is configured, so why should Puppet retain all of this information exclusively for itself? The original extlookup() function in Puppet provides some decoupling of configuration data from Puppet manifests, but I found it a bit limiting and not very elegant having to maintain a bunch of CSV files. I’ve been interested in R.I.Pienaar’s Hiera for a while and thought I’d give it a proper spin and see if it meets my needs.

    Hiera itself is a standalone configuration data store that supports multiple back ends including YAML, JSON and Puppet itself, and adding more back ends to it is a fairly non-challenging task for anyone competent with Ruby. Thanks to hiera-puppet it plugs nicely into Puppet.

    Configuring a basic Hiera setup

    After installing hiera (gem install hiera), I want to test it by setting up a pretty basic configuration store that will override my configuration variables based on environment settings of dev, stage or live. Let’s take a variable called $webname. I want to set it correctly in each of my three environments, or default it to localhost.

    Firstly, I create four YAML files in /etc/puppet/hieradata

    Now I have a YAML file representitive of each environment, I create a simple config in /etc/puppet/hiera.yaml that tells Hiera to search for my environment YAML file followed by common.yaml.

    Now using hiera from the command line, I can look up the default value of $webname with the following command

    But now if I want to know the value for the live and dev environments I can pass an env flag to Hiera

    Accessing this from Puppet

    I can now access these variables directly from my Puppet modules using the hiera() function provided by hiera-puppet. In this example, I already have a fact called ${::env} that is set to dev, stage or live (in my particular set up we use the puppet environment variable for other things)

    Adding more scoping

    OK, thats a fairly simple set up but demonstrates how easy it is to get up and running with Hiera. The requirements I had were a little more complex. Firstly, our hierarchy is broken down into both environment (live, stage, dev..etc) and location. I have multiple environments in multiple locations, a particular location will either be a live, stage or dev environment. So some variables I want to override on the environment level, and some at the more granular location level.

    Secondly, I don’t like the idea of asking Hiera for $webname. That doesn’t tell me anything; what is $webname, what uses it? Consider a more generic variable called $port – that’s going to be confusing. So I started thinking about ways of grouping and scoping my variables. The way I solved this was to introduce a module parameter as well as environment and location in Hiera and place variables for a particular module in it’s own YAML file, using a filesystem layout to determine the hierarchy.

    My new hierdata file system looks a little like this

    Now for each of my modules, I create a YAML file in the folder level that I want to override with the values for my module. Taking the previous example, lets say that I want $webname to be www.myapp.mycorp.com for all live environments, except for Dublin, which I want to be a special case of www.myapp.mycorp.ie. To accomplish this I create the following two files:

    Hiera-puppet will pass the value of $calling_module from Puppet to Hiera, and we can use this in our hierarchy in hiera.yaml. NOTE: Currently you will need this patch to hiera-puppet in order for this to work!

    So our new /etc/puppet/hiera.yaml file looks like:

    On the command line, we can now see that environment, location and calling module are now used when looking up a configuration variable

    In Puppet, I have ${::env} and ${::location} already set as facts, and since $calling_module will get automatically passed to Hiera from Puppet, my myapplication class looks no different…

    but knowing the module name means I can easily find where this value is set, and I can easily see what configuration a module requires by examining its YAML files under /etc/puppet/hierdata

    Conclusion

    In conclusion, I’m now convinced that moving configuration variable data out of Puppet is a very good thing. Now other systems can easily query this valuable information either on the command line or directly with Ruby. By forcing the use of $calling_module I’ve introduced a sort of pseudo scoping for my variables, so, for example… “port” now becomes “port calling_module=apache” and gives me a lot more meaning.

    Many thanks to R.I.Pienaar for help in setting this up, as well as providing the patch to scope.rb that enabled me to get this working.

    Follow and share if you liked this

    Puppet – working with define based virtuals

    Define Based Virtuals

    Define-based virtuals are quite a powerful feature of Puppet, but many people either don’t understand them or don’t know how to apply them effectivly to their manifests. Due to Puppet’s normalized configuration structure, you can’t configure the same resource in two different places. For example, if the user mysql is configured in your database class, and then you realise that your webserver class also needs the mysql user, then, short of including the whhole database class, you cannot just configure the user here. The answer to this lies in resource virtualization.

    Resources that are virtualized are effectivly non-real, they won’t be included in any configuration until they are explicitly realized. The good news is you can realize your resource multiple times in your manifests, so by defining the resource as virtual in a seperate class, both your webserver class and databasse class can realize it.

    Version 0.23 of Puppet took this one step further, with the ability to virtualize your definitions rather than just native resource types. Now we can define a collection of resources, virtualize them, and realize them at whim in our manifests. I’ve seen a few Puppet installations and this method is far under used, and under appreciated.

    To demonstrate, this is an example of how to manage your users with virtual define-based resources.

    User management with defines and virtuals.

    I’ve seen a few ways of managing users within a Puppet estate. The following is a quick guide to get you started managing users, passwords and ssh keys for your users. My favourite approach is to use define-based virtuals and split it out into several sections

    • Back end module to specify the definition for a user
    • A class with a list of virtualized users
    • Realizing your users ad-hoc in your classes

    The advantage of this approach is that you can keep a central place for users whilst still being able to easily pick and choose which users get deployed with which classes. Lets break it down into our components;

    Users module

    As we want to manage users as well as ssh keys it makes sense to wrap this up into a definition, which we’ll call localuser. We create a module called users::virtual where we define the following class.

    This class sets up our localuser definition in users::virtual, other resources can also be added here that will affect every user.

    Creating our users class.

    Now you can create a users.pp class that imports your module and defines the specific users you want to configure. We configure these as virtuals to give us the option of realizing them whenever we want.

    The passord should be the fully encrypted string as would appear in the shadow file, don’t forget to use single quotes if your password string contains $ to prevent it from being interpereted as a variable.

    sshey is optional (note the conditional in our users module), and, if defined will set up the key in ~/.ssh/authorized_keys. Now you have a central list of users as virtuals that call our custom define, they can be easily included, or realized, at whim in your other classes.

    Realize the user

    Within your manifests for your server profile, whenever you require the user bobsmith to be part of the configuration, you simply realize it like this

    The above example realizes the user by the name of “bobsmith”. Another very useful way to realise users is by using Puppet’s collection syntax. For instance, to realize all the users in the users group, you would simply replace the realize statement with:

    … and all your virtual users that match the gid of users will be included.

    Conclusion

    What we’ve acheived by using virtual define-based resources here is primarily,

    • An abstracted module for defining what a user is, rather than who
    • A central place to manage all of our virtual users
    • A very flexible and easy way to pull users into our classes based on name or attribute

    For more information on working with Puppet virtuals, check out the documentation here

    Follow and share if you liked this

    Software review: Rundeck

    I’ve been looking around lately for something that can handle command-and-control automation across an estate of Linux servers. My requirements are to be able to run jobs and tasks remotely, either ad-hoc or at specified times and capture the results in a meaningful and useful way. I’ve come across some of the “Enterprise” solutions touted by a few proprietary vendors (cough, Control M, cough) and not found anything that really impresses me, or should I say, doesn’t scare me… until now, that is.

    My own requirements are fairly specific, I want a GUI and a CLI, I want to be able to run it without client agents, remote execution with SSH is a big bonus and a flat-file configuration structure will make integrating it into other systems fairly painless.

    An open source effort to address these requirements has recently been unveiled by DTO Solutions named RunDeck. At first glance it looked like a good fit for my purposes. I decided to give it a test-run after looking at the overall architecture design. Heres a breakdown of my experiences so far with it.

    Installation

    Installation couldn’t be easier. For test purposes I opted to simply download the self contained .jar file and launch it with java -jar launcher.jar and had a working default system in no time. DTO also provide an RPM for use with yum and the source is available in github.

    Configuration

    RunDeck uses the concept of a Resource Model to define a collection of hosts, or nodes, on your network. Minimally, you can add a resource model by defining a list of hosts in a simple XML document. Each node within your resource model can be assigned different custom tags and basic OS information (system type, OS, kernel version…etc). Your nodes should be configured with passwordless ssh keys to allow the RunDeck user to execute your commands. RunDeck also supports external sources for your resource model meaning you can import nodes automatically from other systems such as Chef or Puppet. In fact, James Turnbull recently wrote puppet-rundeck, for integrating RunDeck with Puppet, read more in his blog post

    The GUI

    RunDeck has an HTML GUI that is clean, uncomplicated and easy to navigate. The three main sections are broken down in to History, Jobs and Run and it doesn’t take long to get comfortable moving around. A huge plus for me is that it seems everything I can do in the GUI can also be done from the command line, extra brownie points for that!

    Node overview and Ad-hoc job run with RunDeck

    RunDeck in action

    Configuring jobs in RunDeck is clear and self explanatory, however the options and abilities are surprisingly versatile. Your job can be made up of any of the following;

    • Commands
    • References to other saved jobs
    • Server-side scripts to be run on the remote node
    • Inline scripts saved within your job

    The latter two are particularly useful for me, it means I can import a variety of legacy scripts that I have no hope of getting rid of in the near future, but at least I can wrap some decent execution framework around them.

    Adding a job with RunDeck
    Adding a job with RunDeck

    Once you’ve defined your job you can assign it to a group of hosts based on a very flexible filter that supports regex’s against most properties defined in your resource model (host name, OS, custom tags…etc).

    Your job can either be saved to run manually or scheduled to run automatically, and even supports crontab formatted scheduling data.

    Ad-hoc commands

    It’s worth mentioning that if the above sounds a bit much to go through when all you want to do is loop through a list of servers and execute a one off command, then you can use RunDeck’s Ad-Hoc feature. Under the Run section you are presenting with a list of hosts that can be filtered with multiple criteria, and at the top a text box to issue commands. So simple, but so very useful.

    Conclusion

    I won’t list every feature in depth, as this has been very adequately documented already. The above is my first impressions of RunDeck from spending less than a day test running it. I’m left feeling that this software is slick, well documented, reliable (a few minor bugs, but overall impressive for a 1.0 release) and does what it says on the tin. It’s defiantly placed high in my sysadmin toolstack arsenal.

    Follow and share if you liked this

    Part 3: Installing puppet-dashboard on CentOS / Puppet 2.6.1

    Puppet Dashboard

    Puppet dashboard is a fairly new app with loads of future potential and is great for monitoring your puppet estate. This is a quick guide to getting it running on puppet 2.6.1. Be sure you have the correct yum repos and ruby versions installed, see Part 1 and Part 2 for more details.

    Install the puppet-dashboard package.

    Create a MySQL database for puppet-dashboard

    Create a database for puppet-dashboard to use and set up a user with all privileges to use it. This can be done on a seperate host.

    Configure database.yaml

    Add your database parameters to the development section, note that host: can be ommitted if you are using local sockets to connect to MySQL.

    Migrate the database

    Copy reports module to site_ruby



    I hate doing this but puppetmasterd explicitly looks for reports in puppet/reports and so far I haven’t found a clean workaround to tell it to look in /usr/share/puppet-dashboard for it. If anyone knows of a way, please email me.

    Edit your puppet.conf

    Include the following in the [master] section, changing punch.craigdunn.org to your puppet server

    Restart puppetmaster and start puppet-dashboard

    Test web GUI

    Go to the following link in your browser (replacing the hostname with your fqdn)

    Configure the client

    Edit puppet.conf

    Make sure the following things are set in the [agent] section of puppet.conf on your client node.

    Run puppet in noop mode on the client

    Refresh browser

    If all has gone well, you should now see your reports in puppet dashboard for your client node.

    Follow and share if you liked this

    Part 2: Puppet 2.6.1, configure puppetmaster and puppetd

    Configure Puppetmaster

    For installing puppetmaster 2.4.1 on CentOS please click here for Part 1

    In Part 1 we covered installing the Puppetmaster and Puppetd packages on Centos 5.5. We will now configure a very basic client/server model to serve the /etc/resolv.conf file to our client. Simple enough!

    Create your first module

    Our first module will be called networking::resolver, it’s job will be to push out a resolve.conf file to clients.

    Create the directory structure under /etc/puppet

    Create your resolv.conf file

    Create your module manifest

    Configure your site and nodes

    Create a minimal site.pp


    Create a tempates file

    Create your node file

    Don’t forget to replace judy.craigdunn.org with the fqdn of your client server

    Set up puppetmaster parameters

    Create default configuration

    This is a minimal puppet.conf file, a more detailed file can be produced with puppetmasterd –genconfig

    The autosign will automatically sign certs for new clients, this is discouraged in a production environment but useful for testing. For information on running puppetmaster without autosign see the puppetca documentation.

    Set permissions for your fileserver.
    Note that this allows everything, you should restrict this in a production environment.

    Start puppetmaster

    The puppet client

    Configure puppetd
    On your client, edit puppet.conf and add the following in the [agent] section, remembering to change punch.craigdunn.org to the fqdn of your Puppetmaster.

    Allow puppetrunner

    Create a file called namespaceauth.conf and add the following, note in a production environment this should be restricted to the fqdn of your puppet master

    Start puppetd

    View pending changes

    Use –test along with –noop to do a dry run to view the changes that puppetd will make

    Now you can run puppetd without –noop to pull in your new resolv.conf file

    This is a very basic demonstration of creating a server/client pair with puppet. There is much more documentation on configuring and managing puppet here

    Next: Installing Puppet Dashboard

    Follow and share if you liked this

    Part 1: Installing puppet 2.6.1 on CentOS with YUM/RPM

    Installing Puppetmaster 2.6.1

    Assuming, like me, the thought of letting rubygems vommit all over your filesystem is not a pleasant one, then how to get the latest puppet 2.6.1 installed on CentOS 5.5 with yum isn’t very clear. Things may differ on other peoples systems, but the below worked for me.

    Set up yum repositories.

    Do this on both the client and the server
    Add the following files and save them to /etc/yum.repos.d/

    puppet.repo


    epel.repo


    ruby.repo


    Note that we include ruby and puppetlabs as the next steps in this tutorial will be to configure puppet and install puppet-dashboard. We want to upgrade to ruby 1.8.6 in order to run puppet-dashboard, doing this now will save you some pain down the line.

    Upgrade Ruby to 1.8.6

    Do this on both the client and the server
    As mentioned above, use the ruby repo to upgrade.

    Install Puppet Server

    On your puppetmaster server:


    On your puppet client

    That’s it, in part 2 and 3 we will install our client and server and install dashboard.

    Part 2 Configuring Puppetmaster and Puppetd

    Follow and share if you liked this