Poison’s Blog

Although many blogposts are available on this topic, I struggled a couple of hours on this thing.

At Netlog, my employer, we’re interfacing with Gatcha, our new game-distribution platform, which runs entirely on Amazon EC2. Amazon has a great Elastic Loadbalancer which you can easily CNAME your DNS records to for public use. Our loadbalancer is currently gatchalb-154894459.eu-west-1.elb.amazonaws.com (which is no secret), and www.gatcha.com is CNAME’d to that record.

It’s a pitty that you don’t get your own IP’s to use with the loadbalancer, I’m not quite happy with the situation for two reasons:

1. You can’t CNAME the root of a domain without CNAME’ing it completely.  We were unable to get gatcha.com (without the www.) CNAME’d correctly, so we had to point it to our serverpark in Brussels where we forward it to www.gatcha.com. (We wanted to use our own MX records, so that’s why the whole domain is not CNAME’d to Amazon)
2. Amazon LB uses a TTL of 60 seconds, and they DO switch IP’s regularly. If the LB hits more traffic, it gets upscaled with more instances (and thus more IP’s). When the traffic drops it gets downscaled, and this is where the trick part begins.

Since we use nscd on all our servers to cache DNS, and we use Debian Lenny (which ships with glibc version 2.7-18), there are some flaws in nscd that ignores the TTL of DNS records (see http://sourceware.org/bugzilla/show_bug.cgi?id=4428). What happend is that we were sending requests to IP’s that weren’t active in our loadbalancer, since this was downscaled in low traffic periods, and we saw suddenly other content than what we expected. (The IP’s were assigned to a new loadbalancer instance). It took us a while to figure that out and the only thing we could do is get rid of nscd and install dnscache (which is more configurable).

A messy bug which annoyed us a few hours

This night I was struggling with cURL (directy from the CLI and mod_curl from php). I was trying to get the Content-Length of a static file to determine it’s filesize but I never saw the result on some objects.

poisonbook:data poison$ curl -I http://iscsi06/v/tt/001/123/1123081.jpg -H "Host: de.video.netlogstatic.com"
HTTP/1.1 200 OK
Expires: Sat, 14 Mar 2009 23:56:18 GMT
Cache-Control: max-age=2592000
Content-Type: image/jpeg
Accept-Ranges: bytes
Last-Modified: Wed, 06 Feb 2008 11:16:08 GMT
Date: Thu, 12 Feb 2009 23:56:18 GMT
Server: lighttpd/1.4.19

As you see, no Content-Length header in it. So I was thinking it’s a bug in lightty and I started to browse the forums. After a half an hour reading, testing, cursing and drinking beers I finally tried to do a manual HTTP request using telnet.

poisonbook:data poison$ telnet iscsi06 80
Trying 192.168.44.6...
Connected to iscsi06.
Escape character is '^]'.
GET /v/tt/001/123/1123081.jpg HTTP/1.1
Host: de.video.netlogstatic.com

HTTP/1.1 200 OK
Expires: Sat, 14 Mar 2009 23:57:00 GMT
Cache-Control: max-age=2592000
Content-Type: image/jpeg
Accept-Ranges: bytes
Last-Modified: Wed, 06 Feb 2008 11:16:08 GMT
Content-Length: 0
Date: Thu, 12 Feb 2009 23:57:00 GMT
Server: lighttpd/1.4.19

You can see, there IS a Content-Length header, but when it’s 0 cURL just ignores it!

Remember: Content-Length: 0 is as good as non-existant in cURL

The versions of cURL I used:

CLI: curl 7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 AND curl 7.15.5 (x86_64-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5

PHP:     [version] => 7.15.5, [host] => x86_64-pc-linux-gnu, [ssl_version] =>  OpenSSL/0.9.8c, [libz_version] => 1.2.3

Cheers,
Nicolas

At Netlog (the company I work for) we have thousands of servers. Fortunately, every server has a logical name (eg.: web001, web002, web003, …).

From time to time, some monkeywork has to be done on a bunch of those servers. Although we have very nice tools for this, sometimes you just have to logon the box and type some commands manually. iTerm, my favourite terminal program on OSX, has a nice feature for doing this: “Send input to all tabs”. The hardest part is to open a new window with all the tabs and ssh’ing to the servers themselves. Once you have an ssh session in every tab you enable to send the input to all tabs and you’re done

Suppose:

You have 10 hosts: host01, host02, …, host10 and you want to execute the command ‘df -h’ (view the disk usage).
What you can do is open iTerm, create 10 tabs (press ‘command-t’ 10 times) and type in each tab: ’ssh host0x’ where x is the box you want to connect to.
Once you’re connected to each box, simply press ‘command-shift-i’ or choose “Send input to all tabs” and pick your favourite host.
Execute your command and when you browse through all the tabs, you’ll see the info of each server. Nice isn’t it?

“Hey, this is stupid, why are you blogging about this?”

Well, the annoying part is the establishing of the ssh connection itself. I really got sick of it and I was thinking if I could get the tab ID, I could do something like ’ssh host${TABID} ‘, send this to all my tabs and got logged in automatically. I wrote a very small applescript to do this.

tell application "iTerm"
	activate
	set format to "" & (choose from list {"0", "00", "000"} with prompt "Choose the format")
	set num to text returned of (display dialog "Enter the offset (default: 1)" default answer "1")
	-- Get the current terminal
	set myTerminal to current terminal
	-- Make a list of all the sessions (tabs) in this terminal (window) to iterate over
	set SessionList to sessions of myTerminal
	repeat with mySession in SessionList
		tell mySession
			set formatText to text -(count format) thru -1 of (format & num)
			write text "" & formatText
		end tell
		set num to num + 1
	end repeat
end tell

I was really amazed by the ease of applescript. It took me 20 minutes to write and debug this (being completely new to applescript).
The biggest help for me was the MacScripter site and in particular the posts of Craig Smith.

Although the chance that this snippet is useful for anybody except me is small, it’s a nice way to demonstrate how easy it is to automate something with applescript.

Cheers,
Nicolas

Hi all,

I know this blog has been inactive for a long time, but one of my intentions for 2009 is to maintain it again. It takes some time but I’m willing to devote one hour in the week for this. I hope I can persist

Cheers,

Nicolas

When I was deploying the new corporate mailserver I needed to re-install the webmail client. We used to work with SquirrelMail (http://www.squirrelmail.org/), a very decent and extendable webmail client. But we all know that the UI of SquirrelMail is ugly, really ugly.

So after some googling I came across this new great webmail client named RoundCube (http://www.roundcube.net/). This is a new fully featured AJAX-based webmail client with a very slick UI. From the first moment I saw that RoundCube was the way to go.

After I did the installation and played around with the webclient I wanted to extend it and give the ability to my users to change their password. The basic idea was to add a tab ‘Password’ in the settings window. Since I’ve configured my mailserver (postfix) with mysql (user fields are created dynamically) it’s just a matter of updating the correct row in the database and reconnect to the IMAP server (courier-imap).
The only problem is that the framework of RoundCube is creap, and I mean really crap. It took me 6 hours to create the tab and the underlaying forms / code. Not because I work that slow but because it’s not intuitive and you have to mess in 5 files in order to add a new action (Ok, I only worked at night on it so my mind wasn’t that clear but anyway).
Below I’ve attached the patch, so if some of you want to make your own tab you can read through the patch and see how it works.

It uses an external script (ext/change_password_function.inc) with one function in it: change_password_function ($user, $currentpass, $newpass).
In my case, it just updates the postfix mailbox database but you can modify this code to fit your needs (eg. if you use the courierpassd you can connect to the password-daemon and change the user’s password here).
Enjoy it,
Nicolas

RoundCube Webmail 0.1-beta changepassword patch

Recently I needed to reconfigure the company’s mail server in order to comply with the different requirements of Yahoo, Hotmail, AOL, etc.
The only solution for us was to set up multiple postfix instances with different configuration files.

I was surprised how easy that is:
cp –rp /etc/postfix /etc/postfix2
cp –rp /var/spool/postfix /var/spool/postfix2

Add the following line to /etc/postfix/main.cf:
alternate_config_directories = /etc/postfix2

Make sure you change the following in /etc/postfix2/main.cf:
syslog_name = postfix2
queue_directory = /var/spool/postfix2
alternate_config_directories = /etc/postfix

The only thing to do now is to bind the instances to separate IP addresses (or even different ports but I didn’t try that).
In my case I’ve changed the inet_interfaces value into mail1 for the first instance and mail2 for the second one. mail1 and mail2 are entries in my /etc/hosts for 192.168.0.4 and 192.168.0.5. They are both in use by my mail server.

The only difficult part was to hack the /etc/init.d/postfix script to start and stop both instances. I’ve posted my init script and it takes care of the multiple instances.

When you run tools like ‘qshape‘ or ‘mailq‘ they all use the spool directory of the first instance. With some tools you can specify the -c parameter to change to an alternative config directory. For example ‘qshape -c /etc/postfix2‘ will analyze the queue of the second instance. But the tool ‘mailq‘ doesn’t take the parameter -c. There you can set the environment variable ‘MAIL_CONFIG’. Just ‘export MAIL_CONFIG=/etc/postfix2 && mailq‘ et voila. When you use the php mail() function just ‘putenv (”MAIL_CONFIG=/etc/postfix2″);‘ before using the mail() function and your mails will be sent thru the second instance.

Happy mailing,
Nicolas

The hacked postfix init script

While installing more tools to monitor the servers of our company I came across a really nice tool of 3ware: 3DM.

It monitors the status of your 3ware escalade RAID arrays and is able to send you an email upon an error.

The only annoying part is the installation on a Debian system: it only supports RedHat / Fedora and SUSE.

Because I need to install it on quite a few servers I don’t felt like doing it manually all the time. So I hacked the installer a bit in order to make it debian-proof.

You can download the package below. If you have any questions or comments, please let me know them.

Enjoy it,
Nicolas

3DM2 9.3.0.3 debian