Poison’s Blog

Although many blogposts are available on this topic, I struggled a couple of hours on this thing.

At Netlog, my employer, we’re interfacing with Gatcha, our new game-distribution platform, which runs entirely on Amazon EC2. Amazon has a great Elastic Loadbalancer which you can easily CNAME your DNS records to for public use. Our loadbalancer is currently gatchalb-154894459.eu-west-1.elb.amazonaws.com (which is no secret), and www.gatcha.com is CNAME’d to that record.

It’s a pitty that you don’t get your own IP’s to use with the loadbalancer, I’m not quite happy with the situation for two reasons:

1. You can’t CNAME the root of a domain without CNAME’ing it completely.  We were unable to get gatcha.com (without the www.) CNAME’d correctly, so we had to point it to our serverpark in Brussels where we forward it to www.gatcha.com. (We wanted to use our own MX records, so that’s why the whole domain is not CNAME’d to Amazon)
2. Amazon LB uses a TTL of 60 seconds, and they DO switch IP’s regularly. If the LB hits more traffic, it gets upscaled with more instances (and thus more IP’s). When the traffic drops it gets downscaled, and this is where the trick part begins.

Since we use nscd on all our servers to cache DNS, and we use Debian Lenny (which ships with glibc version 2.7-18), there are some flaws in nscd that ignores the TTL of DNS records (see http://sourceware.org/bugzilla/show_bug.cgi?id=4428). What happend is that we were sending requests to IP’s that weren’t active in our loadbalancer, since this was downscaled in low traffic periods, and we saw suddenly other content than what we expected. (The IP’s were assigned to a new loadbalancer instance). It took us a while to figure that out and the only thing we could do is get rid of nscd and install dnscache (which is more configurable).

A messy bug which annoyed us a few hours

Recently I needed to reconfigure the company’s mail server in order to comply with the different requirements of Yahoo, Hotmail, AOL, etc.
The only solution for us was to set up multiple postfix instances with different configuration files.

I was surprised how easy that is:
cp –rp /etc/postfix /etc/postfix2
cp –rp /var/spool/postfix /var/spool/postfix2

Add the following line to /etc/postfix/main.cf:
alternate_config_directories = /etc/postfix2

Make sure you change the following in /etc/postfix2/main.cf:
syslog_name = postfix2
queue_directory = /var/spool/postfix2
alternate_config_directories = /etc/postfix

The only thing to do now is to bind the instances to separate IP addresses (or even different ports but I didn’t try that).
In my case I’ve changed the inet_interfaces value into mail1 for the first instance and mail2 for the second one. mail1 and mail2 are entries in my /etc/hosts for 192.168.0.4 and 192.168.0.5. They are both in use by my mail server.

The only difficult part was to hack the /etc/init.d/postfix script to start and stop both instances. I’ve posted my init script and it takes care of the multiple instances.

When you run tools like ‘qshape‘ or ‘mailq‘ they all use the spool directory of the first instance. With some tools you can specify the -c parameter to change to an alternative config directory. For example ‘qshape -c /etc/postfix2‘ will analyze the queue of the second instance. But the tool ‘mailq‘ doesn’t take the parameter -c. There you can set the environment variable ‘MAIL_CONFIG’. Just ‘export MAIL_CONFIG=/etc/postfix2 && mailq‘ et voila. When you use the php mail() function just ‘putenv (”MAIL_CONFIG=/etc/postfix2″);‘ before using the mail() function and your mails will be sent thru the second instance.

Happy mailing,
Nicolas

The hacked postfix init script

While installing more tools to monitor the servers of our company I came across a really nice tool of 3ware: 3DM.

It monitors the status of your 3ware escalade RAID arrays and is able to send you an email upon an error.

The only annoying part is the installation on a Debian system: it only supports RedHat / Fedora and SUSE.

Because I need to install it on quite a few servers I don’t felt like doing it manually all the time. So I hacked the installer a bit in order to make it debian-proof.

You can download the package below. If you have any questions or comments, please let me know them.

Enjoy it,
Nicolas

3DM2 9.3.0.3 debian