Every three or four years

Every three or four years, depending on how a company writes off its hardware, you have machines to replace. Now, replacing a box with a few cables on it isn't hard: you rip the old cords out of the wall and throw the lot on the dump. After that, you place the new boxes in your data centre and plug in all the cords in their respective sockets.

But that isn't quite all there is to it, is it?

You then install a base operating system. If you are lucky, nothing much has changed and you load backup tapes (or whatever media you've used) and restore from that. If you aren't so lucky (as what happened to me), the machine you are replacing wasn't quite, shall we call it up to date?

In that case, it is more or less a start from scratch kind of operation. Software has changed, you decide to use a different IMAP server, the MTA configuration needs tweaking, Apache's authentication modules have changed (for the, it must be, trillionth and a half time), etc., etc., etc.

Oh, well, I'm all done.

Well: not quite. There is still half a load of utilities and stuff that need recompiling (new version of GCC, you know), but I should be getting there soon.

I hope. :-)

Writing

I've
kept
this
quiet
until now, but since I'm approaching a point at which I can smell a "product", I might as well spill the beans: I'm writing a book.

It started off like this:

but I'm quickly approaching the 500-page mark, much to my astonishment.

I've always wanted to write a book but never thought I'd be able to create more than about fifty pages on a single topic. After chatting to a publisher last Summer, I worked up an outline that I started to like, and it has been uphill since then. It is a hell of a lot of work, but I am greatly enjoying it. At least until the moment when my copy editor takes it all to pieces.

I'm writing about alternative DNS servers, very much concentrating on services provided by programs other than vanilla BIND, although I do discuss two special instances of it.

As soon as I'm prepared to do so, I'll post a Table of Contents. It would be a bit premature now, as I'm still fiddling with the order of Chapters, etc.

Yep: I'm writing a book.

MySQL UDF and LDAP

User-defined functions are compiled as shared object files and then added to and removed from the MySQL server dynamically.

I hacked up a small test to demonstrate their implementation, although I'll only show you the results here.


CREATE TABLE u ( username varchar(20) );
INSERT INTO u VALUES ('jpm');
SELECT * FROM u;
+----------+
| username |
+----------+
| jpm      |
+----------+

So far, nothing special, but now for something completely different:


CREATE FUNCTION ldapcn RETURNS STRING SONAME 'libudf_jp2.so';

The CREATE FUNCTION loads the shared object file into the server's address space, where it remains available until the function is dropped from the data dictionary.


SELECT username, LDAPCN(username) AS cn FROM u;
+----------+---------------+
| username | cn            |
+----------+---------------+
| jpm      | Jan-Piet Mens |
+----------+---------------+

During the SELECT, MySQL invokes my UDF, passing it the string argument, and I go off and search for the user in an LDAP directory tree, returning the user's Common Name as the function's value. That in turn is the result of the function which MySQL uses.

Powerful.

More on MySQL UDF.

Bacula in a SOHO

Inspired by reading about it in Backup & Recovery I've been implementing Bacula in my SOHO.

Bacula is a multi-platform backup solution that supports Unix/Linux, Windows and Mac OS X and which, with a bit of help from the documentation, is very easy to implement: within a few hours I had a Bacula server running on my home server and had started backups of a Windows notebook as well as a Mac OS X client.

Bacula is great in that it

  • is multi platform

  • can encrypt data in transport (TLS)

  • can store files on backup media encrypted (PKI)

  • can be managed from any Bacula client

  • can use almost anything as a device (disk, file, tape, DVD, etc.)

Before going anywhere with Bacula, do read the documentation, notably the User's Manual (also as a 700+ page PDF), and familiarize yourself with Bacula's terminology.

After installing the client and server for my Linux distribution from here (choose the RPM based on which database back end you want to use), I first fiddled around a bit, getting to know the files in /etc/bacula and then followed the very good tutorial to get a bit warmer with the setup.

I then destroyed the database and re-initialized it in order to start afresh, and added my first live client, a Windows XP machine. Installation of Bacula for Windows is a matter of a few clicks, and the installer asks for passwords, addresses, and names of the Bacula director.

Installing Bacula

As soon as I'd completed the configuration of the client (file daemon in Bacula–terminology), I used
bconsole
to start a backup job on the client.

On my Apple Mac I downloaded and unpacked the source and ran

./configure --enable-client-only --with-openssl --prefix=/usr/local --with-working-dir=/var/bacula/working
make
make install

after which I adjusted name, address and password of the director server in /usr/local/etc/bconsole.conf to allow bconsole to connect to my director. The configuration of the file daemon (client) also needs to be adjusted, so I did that in bacula-fd.conf as well, and I then launched the file daemon with

sudo /usr/local/sbin/bacula-fd

If you want bacula-fd to launch upon system startup (I do), follow the simple instructions in Bacula for OSX. After adding my Mac to bacula-dir.conf I started backing it up.

The first productional backup job was a success:

*messages
06-Apr 20:47 mens-sd: Job write elapsed time = 03:16:30, Transfer rate = 4.721 M bytes/second
06-Apr 20:47 mens-dir: Bacula 2.0.3 (06Mar07): 06-Apr-2007 20:47:22
  JobId:                  1
  Job:                    jmac.2007-04-06_17.30.40
  Backup Level:           Full (upgraded from Incremental)
  Client:                 "jmac-fd" 2.0.3 (06Mar07) powerpc-apple-darwin8.8.0,darwin,8.8.0
  FileSet:                "JmacOnly" 2007-04-06 17:30:40
  Pool:                   "Default" (From Job resource)
  Storage:                "File" (From Job resource)
  Scheduled time:         06-Apr-2007 17:30:37
  Start time:             06-Apr-2007 17:30:43
  End time:               06-Apr-2007 20:47:22
  Elapsed time:           3 hours 16 mins 39 secs
  Priority:               10
  FD Files Written:       130,894
  SD Files Written:       130,894
  FD Bytes Written:       55,649,021,181 (55.64 GB)
  SD Bytes Written:       55,669,122,267 (55.66 GB)
  Rate:                   4716.4 KB/s
  Software Compression:   4.6 %
  VSS:                    no
  Encryption:             no
  Volume name(s):         t001
  Volume Session Id:      1
  Volume Session Time:    1175873421
  Last Volume Bytes:      55,714,958,991 (55.71 GB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Termination:            Backup OK

I've tweaked the schedules a bit so that my laptop only gets backed up when I'm actually at home, but otherwise Bacula works as advertised. I'll be monitoring the jobs a bit (Bacula sends mail after a job has completed) to ensure that all is working as I want it to, but otherwise the program certainly takes care of all my demands of a backup system, and it is free to use. Bacula has a good reputation so far, and it is also used by large organizations.

For a bit more information on Bacula, I recommend an Onlamp article titled Bacula: Cross-Platform Client-Server Backups.

MailScanner Custom Functions: A Small Tutorial

MailScanner MailScanner is incredibly powerful in that it can determine at run-time whether certain processing should be invoked on a per-message basis; it does this with rulesets, which define regular expressions that are applied to sender or recipient's email addresses, domains or IP addresses, in order for the program to decide whether a certain part of MailScanner should be invoked or not. To further increase flexibility, MailScanner supports so-called custom functions, Perl routines supplied by the administrator, which replace the rulesets and further augment MailScanner's flexibility.

Having created two different custom functions for MailScanner which have had substantial productional use, I'm writing this mini-tutorial in the hope it will be of use to MailScanner administrators. I use MailScanner with the excellent Exim MTA, but the description below ought to be generic to whichever MTA you use MailScanner with, be it Exim, sendmail, Postfix or any other supported MTA.

Implementing a custom function for MailScanner is not for the faint of heart. Before attempting to do so, I strongly recomend you consider whether a ruleset provides enough flexibility for you, as they are easier to implement, test and maintain than custom code is. A number of good sample rulesets can be found in the MailScanner manual, and they demonstrate the flexibility of rulesets very well. In any case, check the documentation before continuing, and the excellent MailScanner Configuration Index is also worth keeping at hand.

A custom function for MailScanner is one or more Perl subs kept in a Perl module in the directory /usr/lib/MailScanner/CustomFunctions or wherever the MailScanner directive Custom Functions Dir points to (check the manual). Custom functions may be used in MailScanner wherever a ruleset is allowed. So, for example, instead of checking whether messages must be signed with a ruleset

Sign Clean Messages = /etc/MailScanner/rules/sign.rules

I can use a custom function to do that:

Sign Clean Messages = &SigFunc

where SigFunc is the custom function I must supply (note the leading ampersand [&] which denotes a function). My custom function is invoked with a copy of the message and MailScanner's environment, so there is little left to desire with regards to what the custom function can do. Its return value must match whatever MailScanner expects for the ruleset: yes/no are returned as integers 1 and 0 respectively, and if MailScanner expects a string, the function must supply that string.

Apart from the existing sample functions in the MailScanner distribution, there is little documentation on custom functions. The first thing I was interested in was what exactly does my custom function receive? Let us write a custom function to determine that.

We have to create a Perl module and install it into the CustomFunctions/ directory, and we have to tell MailScanner where to invoke that function. The file I'll be creating is called /usr/lib/MailScanner/CustomFunctions/myfunc.pm and I'm telling MailScanner that I want it invoked to check whether outgoing messages need to be signed. This setting guarantees that my function will be invoked for each and every message it receives.

Sign Clean Messages = &Myfunc

This module is loaded when MailScanner starts and a supplied Init function is executed, in which initialization of the custom function can take place. Just before unloading the module, MailScanner executes the supplied End function which can be used by the programmer to de-initialize the module.

The basic structure of my module will therefore be the InitMyfunc and EndMyfunc functions (both of which are unused at this time, but they must be present) as well as the actual Myfunc code.

Let me show you the code:

package MailScanner::CustomConfig;
use Data::Dumper;

use strict 'vars';
use strict 'refs';
no  strict 'subs'; # Allow bare words for parameter %'s

use vars qw($VERSION);

$VERSION = substr q$Revision: 1.1 $, 10;

sub MyfuncLOG {
    my ($text) = @_;

    MailScanner::Log::InfoLog("Myfunc: $text");
}

sub InitMyfunc {
    MyfuncLOG("Starting Myfunc");
}

sub EndMyfunc {
    MyfuncLOG("Ending Myfunc");
}

sub Myfunc {
    my($message) = @_;

    return 0 unless $message;        # Sanity

    my $msgid = $message->{id};

   
    if (open(MESSAGE, "> /tmp/msg-$msgid")) {
        print MESSAGE Dumper($message);
        close(MESSAGE);
    }
}

1;
 

Download this code: mailscanner/myfunc/myfunc.pm

The helper routine MyfuncLog utilizes MailScanner's logging module to send a message to the same log to which MailScanner usually logs. On my system those messages are passed to syslog which stores them in /var/log/maillog.

Note the solitary true value at the end of the module file (1;). This ensures that Perl can load the module. Without this, my module won't be loaded.

When testing the custom function keep a very close look at the MailScanner logs (probably /var/log/maillog) for errors. Runtime errors are hard to detect and usually cause MailScanner to croak; you'll recognize that by MailScanner restarting itself constantly. These errors are very hard to find, and, apart from copious logging, I haven't found a good way to debug them. For me, a

$ tail -f /var/log/maillog

is my friend. I'll show you what I mean:

Mar 28 22:59:18 dad MailScanner[2642]: MailScanner E-Mail Virus Scanner version 4.58.9 starting...
Mar 28 22:59:18 dad MailScanner[2642]: Read 764 hostnames from the phishing whitelist
Mar 28 22:59:18 dad MailScanner[2642]: Config: calling custom init function Myfunc
Mar 28 22:59:18 dad MailScanner[2642]: Myfunc: Starting Myfunc
Mar 28 22:59:30 dad MailScanner[2642]: New Batch: Scanning 1 messages, 672 bytes
Mar 28 22:59:30 dad MailScanner[2642]: Virus and Content Scanning: Starting
Mar 28 22:59:31 dad MailScanner[2642]: Myfunc: Invoking Myfunc for 1HWfEv-0000gh-C4
Mar 28 22:59:31 dad MailScanner[2642]: Uninfected: Delivered 1 messages

When MailScanner starts, I see my init function being invoked. The fourth line is the log message in the init function. I then submitted a message, and MailScanner starts the new batch, during which it calls Myfunc to handle our message. Although not shown here, when stopping MailScanner, I see the end function being called. Once again, these two can be used to create and later drop a connection to a database or to bind and later unbind from an LDAP directory server. Generally, the Init function sets up the environment, and the End function cleans that up.

Of course I'm specially curious as to whether our file /tmp/msg-msgid was created. ;-) Here is an excerpt of it, as it contains the whole message structure as supplied by MailScanner. The use of Data::Dumper is invaluable in decoding that. So, without further ado, heeeere is the message!

$VAR1 = bless( {
       'datestring' => 'Wed Mar 28 22:59:30 2007',
       'sascore' => 0,
       ...
       'virusinfected' => 0,
       'metadata' => {
             'headers' => [
              ...
                  {
               'body' => ' postmaster@localhost',
               'name' => 'To:',
               'flag' => 'T'
                  },
                  {
               'body' => ' jpm@fupps.com',
               'name' => 'From:',
               'flag' => 'F'
                  },
               ...
                ],
             'dv_interface_address' => '127.0.0.1.25',
             'numrcpts' => 1,
             'dv_body_linecount' => '2',
               ...
             'user' => 'root',
             'id' => '1HWfEv-0000gh-C4-H',
             'dv_received_protocol' => 'esmtp',
               ...
             'rcpts' => [
                'postmaster@localhost'
              ],
             'sender' => 'jpm@fupps.com',
               ...
           },
       'id' => '1HWfEv-0000gh-C4',
       'otherreports' => {},
       'safesubject' => ' test Wed, 28 Mar 2007 22:59:29 +0200',
       ...
       'fromuser' => 'jpm',
       'spamreport' => 'not spam',
       'subject' => ' test Wed, 28 Mar 2007 22:59:29 +0200',
       'fromdomain' => 'fupps.com',
       'spamwhitelisted' => 0,
       'scanmail' => 1,
       'todomain' => [
             'localhost'
           ],
       ...
       'size' => 672,
       'touser' => [
           'postmaster'
         ],
       ...
       'from' => 'jpm@fupps.com',
       'numberparts' => 1,
       ...
       'clientip' => '127.0.0.1',
       ...
       'headerspath' => '/var/spool/MailScanner/incoming/2642/1HWfEv-0000gh-C4.header'
     }, 'MailScanner::Message' );
 

Download this code: mailscanner/myfunc/msg1.pl

Believe it or not, I've shortened the output a bit (have a look at one of your own messages), but even so, there is a wealth of interesting information that was passed to our function: I'm sure you'll agree.

To begin with, sender and recipient details of course, as well as information on the message content: its parts, lines, size, date, sending client IP address, etc. Some of these values are MTA dependent, and most of them are undocumented at the best of times.

Getting at the values you want to check ought to be quite easy: look at how I retrieved the Message-ID above.

As already mentioned, Custom MailScanner functions can be used wherever a ruleset is expected. I myself have used such functions successfully for whitelisting incoming messages as well as for determining if an outgoing message needs a footer (signature) attached to it. Some more ideas might be:

  • Enable individual users to receive password-protected Zip files (Allow Password-Protected Archives)
  • Use a directory service to enable larger messages on a per-user basis; do remember though, that Maximum Message Size must match whatever your MTA uses
  • Decide on a per-user or per-domain basis whether Virus Scanning should be enabled (probably not a good idea)
  • Determine from message content whether Notifiy Senders should be applied or not

There are undoubtedly many other uses for custom functions, but once again: do first check if a normal ruleset will work for you. If not, I'd be pleased to hear where you are using custom functions for; just leave a comment.

Footers on Email with MailScanner

Legislation in this country is forcing us to add footers to outgoing messages which specify the company details of the senders. Corporate details could of course be added with signatures, but that leaves the phrasing to end-users, and there'd be no method to enforce using them.

Instead, I've decided that the only possible point of distribution for these footers will be our MailScanner gateways, as only they guarantee that we catch all sources of outgoing messages, irrespective of which system (MTA) and which client (MUA) the message originates from.

The technology I'm applying to add the footers is similar to that which I used for custom whitelisting, although it is a bit more complex. This additional level of complexity arises from the fact that there are a number companies we provide electronic mail service to, and that each user might require a distinct footer.

To cater for this, I'm adding a custom attribute type to our OpenLDAP directory servers. This attribute type (footerID) will store an numeric identifier which can be assigned to a user's LDAP entry.

As soon as the custom MailScanner function receives an outgoing message, it will search the directory for the sender's address and will determine from the footerID which mail footer should be added to the outgoing message.

Voila!

There is just one tiny detail missing. The code. ;-)

MySQL Grants are Self-Revoking

I have a very old MySQL server (release 3.23.58) that is loosing its privilege information (i.e. the GRANTs).

I've performed a full dump of the tables in the mysql database and have restored them to a newly created database.

$ mysqladmin create p
$ mysqldump mysql | mysql p

After shutting down the server and switching the mysql and p database directories, I brought the server back up, but there is no change. The privileges on two database tables are simply disappearing.

Any ideas, anybody?

Four-Way SyncML (Almost)

After solving the issues with the duplicates in my SyncML setup, I added another synchronization node to the picture ∗.

iSync

I've actually had that in there all the time, but I didn't want to complicate the issue by mentioning it. ;-)

The original setup hasn't really changed, but instead of altering contacts on Lotus Notes, I do so in AddressBook on the Mac.

Threeway Sync with Mac OS

It works, and I am now really happy. I've been hammering deletes, updates and inserts to all sides of this party, and I've yet to find a problem.

iSync log

∗ Until the next fubar happens with my setup. ;-)