Upgrade your Servers with Rex - fix ShellShock

If you need to upgrade your servers to protect them against ShellShock or other vulnerabilities Rex can help you to do this fast and reliable.

Rex is a tool that runs on your workstation or a central management server. Rex will connect via ssh to your hosts and executes the given commands. There is a simple perl based DSL to ease the execution, so you don't care about running apt-get, yum, zypper, ... to install a package. For example there are functions to manage services, packages, users, files and many more.

The installation is easy and described on the webpage.

If you already have a Rexfile which defines your server groups and authentication you can just use it to upgrade bash.

$ cd path/to/your/Rexfile

$ rex -G $server_group -e "pkg 'bash', ensure => 'latest';"

If you don't have a Rexfile yet, you can also use something like this:

$ rex -H 'frontend[01..10] middleware[01..05]' \

    -u $user -p $password \

    -e "pkg 'bash', ensure => 'latest';"

If you need Key-Authentication

$ rex -H 'frontend[01..10] middleware[01..05]' \

    -u $user \

    -P /path/to/private.key -K /path/to/public.key \

   [-p keypass ] \

    -e "pkg 'bash', ensure => 'latest';"

If you want to execute the installation in parallel you can use the -t $thread_count cli option to control the threading.

$ rex -H 'frontend[01..10] middleware[01..05]' \

    -u $user -p $password \

    -t 50 \

    -e "pkg 'bash', ensure => 'latest';"

Impressions from the Linuxtag 2013

This year we had a project booth from Rex on the Linuxtag. I have to say that i liked it very much and i will try to organize a project booth next year, too. Perhaps in conjunction with a Rex workshop.

Because the project hadn't done a lot of publicity yet, this was the projects first booth on such a exhibition/conference. And i want to thank inovex to sponsor the banner, the flyers, t-shirts and the hotel for the project members.

We had a lot of interesting talks about configuration- and deploymanagement the pros and cons of a lot of other tools in this category and of course about Rex.

There where also some nice ideas how to extend Rex to make it easier for administrators (who don't know perl) to work with or to better scale to a large server cluster. I'm going to add some of these ideas in one of the next releases.

Most people didn't know Rex at all and asked to explain what Rex does in a few sentences. These conversations always ended up in a longer conversation (10-20 minutes, or even longer)

One interessing (or funny) thing i noticed during these conversation was, that american people (or people working for american companies) always asked about scalability (that was mostly the first question by them) and european people wanted to know how it works or how they can extend it (only a few asked about scalability).

This year i hadn't had the time to visit some talks because i was all the time on the booth. But from the things i've heard there was some nice talks.

So for the next time i have to plan a bit more better so that i can go to some talks, too :-)

Perl IO with IO::All

Working with Files and Directories in Perl Programming is not really a big deal. But there is a  module that ease this in a cool way.

If you want to do IO operations in Perl i advice you to take a look at IO::All.

File Operations

Opening a file in the traditional way is done with the open() function and than you can iterate over the lines.

my @lines;
open my $fh, "<", "the-file";
@lines = <$fh>; 
close $fh;

With IO::All you can simplify this to one line

my @lines = io("the-file")->getlines;

If you want to load a complete file into a string you have to do the following without IO::All:

my $content;
open my $fh, "<", "the-file";
{
    local $/;
    $content = <$fh>; 
} 
close $fh;

Compared to IO::All it looks a bit too much:

my $content = io("the-file")->all;

You can also use the "<" operator (you might already know from bash) to read a file.

my $content < io("the-file");

Writing to a file is also really easy and straight forward compared to the standard way.

open my $fh, ">", "the-file";
print $fh "Hello World\n"; 
close $fh;

With IO::All it is - as you may expect - only one line.

"Hello World\n" > io("the-file");

If you want to append to a file you can just use the ">>" operator.

"Hello World\n" >> io("the-file");

Special File Operations

With IO::All it is also easy to manipulate files without much code.

For example if you want to edit a special line inside a file you can do this with the following code:

my $file = io("the-file");
$file->[10] = "New content in Line eleven";
print $file->[20];

Directory Operations

 If you want to read the contents of a directory in the past you had code like this:

my @contents; 
opendir my $dh, "/dir";
while(my $entry = readdir($dh)) {    
    next if($entry eq "." || $entry eq "..");
    push @contents, $entry;
} 
closedir $dh;

With IO::All you can reduce this to a one liner:

my @contents = io("/dir")->all;

With this function it is also easy to traverse subdirectories.

my @contents = io("/dir")->all(0);

Or, if you don't want to go through every subdirectory you can specify the depth.

my @contents = io("/dir")->all(2);

Setting hostname and domainname under Solaris 10

Just a short post howto set the hostname and the domainname under Solaris 10. It took a while until i found the solution, hence this post.

echo your-shiny-hostname >/etc/nodename
echo your-shiny-domain >/etc/defaultdomain

now reboot and it is done.

Puppet, iterating over a hash

Today i fiddled about how to iterate over a hash in a puppet function (define). Okay, it is a bit .. scary but it works ;-)

define mymodule::myfunction($options = {}) {
     $cli_parameter = inline_template(
          "<% options.each_pair { |key, val| %> --<%= key %>=<%= val %><% } %>");
     // ...
}

Managing Servers with (R)?ex

Rex is a tool to automate the management of servers. In this blog post i will just post an example of managing some webservers with mod_jk and tomcat.

Every webserver has the same file system structure and the same os (but this is negligible).


To prepare a new webserver i've written a task to install all the needed packages, copy the configuration files and start all the required services.


This is my Rexfile.

use lib 'lib';
user "root";      # set your username
password "pass";  # set your password
pass_auth;        # enable password authentication

# put your server in this group
group "www" => "www[01..05]";

# now load every module via ,,require''
require Rex::mysite;

I've splittet the different tasks in some smaller modules.

This is the module for all the base tasks. (Saved under "lib/Rex/mysite.pm")

package Rex::mysite;

use Rex::Commands;
use Rex::Commands::Fs;
use Rex::Commands::Run;
use Rex::Commands::Pkg;
use Rex::Commands::Upload;
use Rex::Hardware;

use Data::Dumper;

use Rex::mysite::plain;

desc "Prepare system";
task "prepare", group => 'www', sub {

   my %host_info = Rex::Hardware->get(qw/ Host /);

   if($host_info{'Host'}->{'operatingsystemrelease'} =~ m/^5/) {
      install file => "/etc/apt/sources.list", {
         source => "files/sources.list.lenny",
      };
   }
   else {
      install file => "/etc/apt/sources.list", {
         source => "files/sources.list.squeeze",
      };
   }

   if(! is_dir("/root/.ssh")) {
      mkdir "/root/.ssh";
   }

   upload "files/authorized_keys", "/root/.ssh/authorized_keys";

   run "apt-get update";

   install package => "vim";
   install package => "less";

   remove package => "avahi-daemon";
};


1;

And now i have a task to install the tomcat server and configure the apache servers with mod_jk.  (Saved under "lib/Rex/mysite/plain.pm"). With the needs function i define a dependency to the prepare Task in the module Rex::mysite. So this task will be executed always in the same context (same ssh connection) of the tomcat Task in this module.

package Rex::mysite::plain;

use Rex::Commands;
use Rex::Commands::Fs;
use Rex::Commands::Run;
use Rex::Commands::Pkg;
use Rex::Commands::Upload;
use Rex::Commands::Service;

use Data::Dumper;

desc "Prepare Tomcat";
task "tomcat", sub {
   needs Rex::mysite "prepare";

   run "echo sun-java6-jdk shared/accepted-sun-dlj-v1-1 select true | /usr/bin/debconf-set-selections";
   run "echo sun-java6-jre shared/accepted-sun-dlj-v1-1 select true | /usr/bin/debconf-set-selections";

   install package => [
                        "sun-java6-jdk",
                        "libapache2-mod-jk",
                        "sc-tomcat-6.0.28",
                        "sc-tomcat-juli-6.0.28",
                        "sc-tomcat-log4j-1.2.16"
                     ];

   unlink "/etc/apache2/sites-enabled/000-default";
   upload "files/000-default", "/etc/apache2/sites-enabled/000-default";
   upload "files/jk.conf", "/etc/apache2/mods-enabled/jk.conf";

   install file => "/etc/apache2/workers.properties", {
      source    => "files/workers.properties.tpl",
      on_change => service(apache2 => "restart"),
   };

};

1;

The files i uploaded in these tasks are not realy interesting. They are just simple configuration files. Except the file workers.properties.tpl. This i a template file and looks like this.

worker.list=lb,jkstatus

worker.w1.port=8009
worker.w1.host=<%+ $::Network->{"networkconfiguration"}->{"eth0"}->{"ip"} %>
worker.w1.type=ajp13
worker.w1.lbfactor=1
worker.w1.reference=worker.template

worker.lb.type=lb
worker.lb.balance_workers=w1
worker.lb.max_reply_timeouts=10

worker.lb.sticky_session=false
worker.jkstatus.type=status

worker.template.type=ajp13
worker.template.connection_pool_timeout=600
worker.template.connection_pool_minsize=0

This is a template file. During the upload process Rex will parse this template and replace <%+ $::Network->{"networkconfiguration"}->{"eth0"}->{"ip"} %> with the ip of the networkdevice eth0.

With these tasks i can update the configuration of all my servers realy simple. If a new server needs to be installed i can just run the task on this server.

$ rex -H mynewserver mysite:plain:tomcat