• using RDS, a “Database as a Service”
• running your own MySQL server on a EC2 instance, with an EBS volume attached, a snapshot-enabled filesystem…

The comparison is made on features, cost and performances. I used the same MySQL version as RDS (5.1.42) and the same config. Performances are measured by tpcc-mysql from Percona and they are charted using Google Fusion Tables.

You’ll se that, altough MySQL + EC2 requires some additionnal DBA or sysadmin skills, it allows a lot of more tuning and a real performance boost (+270%).

So, if you’re planning to host MySQL databases on Amazon, feel free to take a look at these slides :

Note for french-speaking people : la présentation est disponible en français.

Feel free to Flattr this post at flattr.com, if you like it.

Here is how to quickly configure Awstats to take profit of your Qmail log files…

First of all, you need to convert rour /var/log/mail.log file to make it readable to Awstats (supposing you’re using Qmail and Vpopmail from Dotdeb and Awstats from Debian’s official repositories).

Hopefully, a tool is provided with the awstats package : /usr/share/doc/awstats/examples/maillogconvert.pl

You can take a look at its output by executing the following command :

/usr/share/doc/awstats/examples/maillogconvert.pl vadmin /var/log/mail.log 

(vadmin is here to consider the Vpopmail’s virtualdomain translation)

Great! We now just have to configure awstats to produce useful reports. Here is a sample configuration file (name it /etc/awstats/awstats.mail.conf, for example)… Please note the space after the pipe :

LogFile="/usr/share/doc/awstats/examples/maillogconvert.pl vadmin /var/log/mail.log | "
LogType=M
LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
SiteDomain="localhost"
HostAliases="127.0.0.1 @/etc/qmail/rcpthosts"
LevelForBrowsersDetection=0 # 0 disables Browsers detection.
LevelForOSDetection=0 # 0 disables OS detection.
LevelForRefererAnalyze=0 # 0 disables Origin detection.
LevelForRobotsDetection=0 # 0 disables Robots detection.
LevelForSearchEnginesDetection=0 # 0 disables Search engines detection.
LevelForKeywordsDetection=0 # 0 disables Keyphrases/Keywords detection.
LevelForFileTypesDetection=0 # 0 disables File types detection.
ShowSummary=HB
ShowMonthStats=HB
ShowDaysOfMonthStats=HB
ShowDaysOfWeekStats=HB
ShowHoursStats=HB
ShowDomainsStats=0
ShowHostsStats=HBL
ShowRobotsStats=0
ShowEMailSenders=HBML
ShowEMailReceivers=HBML
ShowSessionsStats=0
ShowPagesStats=0
ShowFileTypesStats=0
ShowOSStats=0
ShowBrowsersStats=0
ShowOriginStats=0
ShowKeyphrasesStats=0
ShowKeywordsStats=0
ShowMiscStats=0
ShowHTTPErrorsStats=0
ShowSMTPErrorsStats=1

Now, just add this line to /etc/cron.d/awstats to compute your logfiles every 10 minutes:

0,10,20,30,40,50 * * * * root [ -x /usr/lib/cgi-bin/awstats.pl ] && /usr/lib/cgi-bin/awstats.pl -config=mail -update &>/dev/null

(Let’s enjoy a 10-minutes break or brew a good coffee, waiting for the stats to be calculated…)

Your mailserver’s statistics will shortly be available at this URL : http://yourserver/cgi-bin/awstats.pl?config=mail.

Feel free to Flattr this post at flattr.com, if you like it.

Then, this article will show you how to build packages from your favorite PECL extensions in a strict Debian way, using the dh-make-php package.

Imagine we want to package the Fileinfo extension for PHP5…

First, install all the needed packages (libmagic-dev is only needed here to build Fileinfo) :

# apt-get install dh-make-php php5-dev build-essential libmagic-dev

Then, export some needed environment variables, download the package

# export DEBFULLNAME="Guillaume Plessis"
# export DEBEMAIL="gui@php.net"
# pecl download fileinfo
downloading Fileinfo-1.0.4.tgz ...
Starting to download Fileinfo-1.0.4.tgz (5,835 bytes)
.....done: 5,835 bytes
File /tmp/Fileinfo-1.0.4.tgz downloaded

Ok, let’s summon the magical dh-make-pecl command with the appropriate arguments :

# dh-make-pecl --only 5 Fileinfo-1.0.4.tgz
Creating debian source package: php-fileinfo-1.0.4
Upstream is: Ilia Alshanetsky
Guessing Maintainer: Guillaume Plessis

No error message… Great! dh-make-pecl did its job, e.g. unarchived the tarball and created the php-fileinfo-1.0.4/debian/ subdirectory that contains all the packaging stuff.
“–only 5” restricts the building process to PHP5 packages (PHP4 is obsolete!)

Let’s build the package!

# cd php-fileinfo-1.0.4/
# ./debian/rules binary
xsltproc --nonet --novalid debian/changelog.xsl package.xml > debian/Changelog
touch build-stamp
dh_testdir
...
dh_md5sums
dh_builddeb
dpkg-deb: building package `php5-fileinfo' in `../php5-fileinfo_1.0.4-1_i386.deb'.

That’s it! Let’s take a look at the content of the package…

# dpkg -c ../php5-fileinfo_1.0.4-1_i386.deb
drwxr-xr-x root/root         0 2008-09-20 10:10 ./
drwxr-xr-x root/root         0 2008-09-20 10:10 ./usr/
drwxr-xr-x root/root         0 2008-09-20 10:10 ./usr/lib/
drwxr-xr-x root/root         0 2008-09-20 10:10 ./usr/lib/php5/
drwxr-xr-x root/root         0 2008-09-20 10:10 ./usr/lib/php5/20060613+lfs/
-rw-r--r-- root/root     10296 2008-09-20 10:10 ./usr/lib/php5/20060613+lfs/fileinfo.so
drwxr-xr-x root/root         0 2008-09-20 10:10 ./usr/share/
drwxr-xr-x root/root         0 2008-09-20 10:10 ./usr/share/doc/
drwxr-xr-x root/root         0 2008-09-20 10:10 ./usr/share/doc/php5-fileinfo/
-rw-r--r-- root/root       202 2008-09-20 10:09 ./usr/share/doc/php5-fileinfo/README.Debian
-rw-r--r-- root/root        26 2004-02-13 01:06 ./usr/share/doc/php5-fileinfo/CREDITS
-rw-r--r-- root/root      3466 2008-09-20 10:09 ./usr/share/doc/php5-fileinfo/copyright
-rw-r--r-- root/root       159 2008-09-20 10:09 ./usr/share/doc/php5-fileinfo/changelog.Debian.gz
-rw-r--r-- root/root       865 2005-11-24 23:07 ./usr/share/doc/php5-fileinfo/fileinfo.php
-rw-r--r-- root/root       149 2008-09-20 10:10 ./usr/share/doc/php5-fileinfo/changelog.gz

and its metadata :

# dpkg -I ../php5-fileinfo_1.0.4-1_i386.deb
 new debian package, version 2.0.
 size 12740 bytes: control archive= 4848 bytes.
     940 bytes,    31 lines   *  config               #!/bin/sh
     607 bytes,    20 lines      control
     522 bytes,     7 lines      md5sums
    1218 bytes,    58 lines   *  postinst             #!/bin/sh
     206 bytes,     8 lines   *  postrm               #!/bin/sh
    1773 bytes,    63 lines   *  prerm                #!/bin/sh
    8758 bytes,    67 lines      templates
 Package: php5-fileinfo
 Version: 1.0.4-1
 Section: web
 Priority: optional
 Architecture: i386
 Depends: libc6 (>= 2.3.6-6), libmagic1, debconf (>= 0.5) | debconf-2.0, php5 | php5-cli
 Installed-Size: 100
 Maintainer: Guillaume Plessis
 Source: php-fileinfo
 Description: Fileinfo module for PHP 5
  libmagic bindings
  .
  This extension allows retrieval of information regarding vast
  majority of file.
  This information may include dimensions, quality, length etc...
  .
  Additionally it can also be used to retrieve the mime type for a
  particular
  file and for text files proper language encoding.
  .

Great job! Now install the package and enjoy…

For more information and specific needs, take a look at “dh-make-pecl” and “man dh-make-pecl“.

Feel free to Flattr this post at flattr.com, if you like it.

By default, Qmail listens to all the available IP address on the machine (0.0.0.0). It is possible to change this behaviour to bind Qmail to a specific IP.

We suppose that you are using Qmail from Dotdeb and launching it using the provided init script, /etc/init.d/qmail. Just edit it and change these lines :

          sh -c "start-stop-daemon --start --quiet --user qmaild \
          --pidfile /var/run/tcpserver_smtpd.pid --make-pidfile \
          --exec /usr/bin/tcpserver -- -H -P -R -l 0 \
          -u `id -u qmaild` -g `id -g nobody` -x /etc/tcp.smtp.cdb 0 smtp \
          $rblsmtpd /usr/sbin/qmail-smtpd 2>&1 \
          | $logger &"

by those ones :

          sh -c "start-stop-daemon --start --quiet --user qmaild \
          --pidfile /var/run/tcpserver_smtpd.pid --make-pidfile \
          --exec /usr/bin/tcpserver -- -H -P -R -l 0 \
          -u `id -u qmaild` -g `id -g nobody` -x /etc/tcp.smtp.cdb xxx.xxx.xxx.xxx smtp \
          $rblsmtpd /usr/sbin/qmail-smtpd 2>&1 \
          | $logger &"

(Just replace xxx.xxx.xxx.xxx by your IP address)

Now, when you list your listening dameon, you should see tcpserver listening to xxx.xxx.xxx.xxx:25 instead of 0.0.0.0:25.

machine# netstat -apn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 xxx.xxx.xxx.xxx:25      0.0.0.0:*               LISTEN      21175/tcpserver

Feel free to Flattr this post at flattr.com, if you like it.

TCMalloc is faster than the glibc 2.3 malloc (available as a separate library called ptmalloc2) and other mallocs that I have tested. ptmalloc2 takes approximately 300 nanoseconds to execute a malloc/free pair on a 2.8 GHz P4 (for small objects). The TCMalloc implementation takes approximately 50 nanoseconds for the same operation pair.

Its deployment in your LAMP stack can speed up your MySQL servers, since it enhances memory allocation on threaded applications with the downside of larger memory footprints.

Here is how to use it easily…

Installation

First of all, be sure you use Debian 5.0 (a.k.a. “Lenny”) or later. Then install the minimal tcmalloc library from Dotdeb :

apt-get install libtcmalloc-minimal0

Then, since the mainstream MySQL packages are not compiled against tcmalloc, you’ll have to trick your OS’ dynamic linker by adding the following line at the top of your /etc/init.d/mysql init script :

export LD_PRELOAD="/usr/lib/libtcmalloc_minimal.so.0"

After relauching your MySQL server using the modified init script, you’ll take profit from tcmalloc’s faster memory allocation.

Results

At the time of writing this article, I didn’t make benchmarks, but some reported that they had a ~15-20% performance gain. Please leave comments about your experience.

Feel free to Flattr this post at flattr.com, if you like it.

rsync -a --delete rsync.dotdeb.org::packages/ /your/local/path/

Then, if you wish your mirror to appear on the mirrors list, send me its full URL.

Note 1 : mirroring once a day should be enough. More frequent updates could lead to a ban without any warning.
Note 2 : Using anything else than Rsync (Wget, HTTrack…) could lead to a ban without any warning.

Feel free to Flattr this post at flattr.com, if you like it.

Memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load…

Many well-known huge architecture (Facebook, Livejournal, Youtube…) are using it as memory caching to reduce the load on their servers. It can also be used to share PHP sessions among several servers. Let’s see how…

Installation

The first thing is to install the memcached server on your Debian server :

apt-get install memcached

Then, since the memcache PECL extension now provides its own session handler, it’s easy to plug PHP and memcached servers. Just install the appropriate extension (from Dotdeb) :

apt-get install php5-memcache

and change some of your PHP settings :

session.save_handler = files
; session.save_path = "N;/path"

to :

session.save_handler = memcache
; change server:port to fit your needs...
session.save_path="tcp://server:port?persistent=1&weight=1&timeout=1&retry_interval=15"

That’s all! After relaunching your Apache2 server, your PHP sessions will be stored on the memcached server.

Feel free to Flattr this post at flattr.com, if you like it.

Installation

First of all, be sure to have Qmail installed from Dotdeb with a version number greater than 1.03-37.dotdeb.1.
Then, since we’ll use a MySQL backend to share the greylisting database between several servers, be sure to have MySQL installed somewhere on your boxes.

In our example, we’ll create the database (we’ll name it relaydelay) and grant access to user@host with the password “password“. You’re free to change these values to fit your needs…

CREATE DATABASE relaydelay;
GRANT ALL PRIVILEGES ON `relaydelay`.* TO 'user'@'localhost' IDENTIFIED BY 'password';

Then, we create the needed tables :

CREATE TABLE relaytofrom
(
  id              bigint          NOT NULL        auto_increment,
  relay_ip        char(16),
  mail_from       varchar(255),
  rcpt_to         varchar(255),
  block_expires   datetime        NOT NULL,
 
  record_expires  datetime        NOT NULL,
  blocked_count   bigint          default 0 NOT NULL,
  passed_count    bigint          default 0 NOT NULL,
  aborted_count   bigint          default 0 NOT NULL,
  origin_type     enum("MANUAL","AUTO") NOT NULL,
  create_time     datetime        NOT NULL,
  last_update     timestamp       NOT NULL,
 
  primary key(id),
  key(relay_ip),
  key(mail_from(20)),
  key(rcpt_to(20))
);
 
CREATE TABLE dns_name
(
  relay_ip      varchar(18)       NOT NULL,
  relay_name    varchar(255)      NOT NULL,
  last_update   timestamp         NOT NULL,
  primary key(relay_ip),
  key(relay_name(20))
);
 
CREATE TABLE mail_log
(
  id              bigint          NOT NULL        auto_increment,
  relay_ip        varchar(16)     NOT NULL,
  relay_name      varchar(255),
  dns_mismatch    bool            NOT NULL,
  mail_from       varchar(255)    NOT NULL,
  rcpt_to         varchar(255)    NOT NULL,
  rcpt_host       varchar(80)     NOT NULL,
  create_time     datetime        NOT NULL,
 
  primary key(id),
  key(relay_ip),
  key(mail_from(20)),
  key(rcpt_to(20))
);

We now have to put the appropriate settings in the /etc/init.d/qmail init script…

export MYSQLHOST="localhost"
export MYSQLUSER="user"
export MYSQLPASS="password"
export MYSQLDB="relaydelay"
export BLOCK_EXPIRE=5           # minutes until email is accepted
export RECORD_EXPIRE=600        # minutes until record expires
export RECORD_EXPIRE_GOOD=36
export LOCAL_SCAN_DEBUG=0

and in the /usr/sbin/greylisting-delete-expired :

MYSQLHOST="localhost"
MYSQLUSER="user"
MYSQLPASS="password"
MYSQLDB="relaydelay"

We can now relaunch Qmail and enable or disable the greylisting with a simple symbolic link :

/etc/init.d/qmail
ln -s /usr/bin/qmail-envelope-scanner /usr/sbin/qmail-envelope-scanner

Usage

Example wildcard whitelists for subnets :

INSERT INTO relaytofrom VALUES (0,"127.0.0.1"   ,NULL,NULL,"0000-00-00 00:00:00","9999-12-31 23:59:59",0,0,0,"MANUAL",NOW(),NOW());
INSERT INTO relaytofrom VALUES (0,"192.168"     ,NULL,NULL,"0000-00-00 00:00:00","9999-12-31 23:59:59",0,0,0,"MANUAL",NOW(),NOW());

Example wildcard whitelist entry for a recieved domain or subdomain

INSERT INTO relaytofrom VALUES (0,NULL,NULL,"sub.domain.com","0000-00-00 00:00:00","9999-12-31 23:59:59",0,0,0,"MANUAL",NOW(),NOW());

Feel free to Flattr this post at flattr.com, if you like it.

The problem

It is not possible to enable both PHP4 and PHP5 Apache2 modules on the same webserver, it causes shared objects collision. Then, the idea is to enable PHP5 as Apache2 module on one side and PHP4 as CGI on the other side.

Please note that this solution is not a good idea anywhere else but on developpement servers and should be temporary because of the performance and security issues of PHP as CGI.

Installation

First, we have to install the appropriate PHP packages and their dependencies (from Dotdeb, of course) :

apt-get install libapache2-mod-php5 php4-cgi

Apache2 is now configured by default to parse .php files with the mod_php5 module.

But, using mod_actions, we can tell Apache2 to parse .php files with the php4 CGI. First, enable mod_actions :

a2enmod actions
/etc/init.d/apache2 force-reload

Now, by adding the following two lines in a Virtualhost or Directory (…) context or directly in the main Apache2 configuration file, we’ll switch from PHP5 to PHP4 :

AddHandler php-script .php
Action php-script /cgi-bin/php4

(It could be necessary to reload your apache configuration :

/etc/init.d/apache2 force-reload

)

The result
To check that all works fine, create a file, called phpinfo.php, containing the following code :

Depending on the presence (or not) of the two above magical lines, a HTTP request on the script will show a PHP4 (or PHP5)-typical output.

Feel free to Flattr this post at flattr.com, if you like it.

Installation

First of all, we suppose that :

• you installed the last Qmail and Vpopmail packages from Dotdeb
• you’re launching them using the bundled init scripts (with tcpserver)

Configuration

To enable SMTP-auth, you just have to edit the /etc/init.d/qmail init script and replace these lines :

sh -c "start-stop-daemon --start --quiet --user qmaild
  --pidfile /var/run/tcpserver_smtpd.pid --make-pidfile
  --exec /usr/bin/tcpserver -- -R
  -u `id -u qmaild` -g `id -g nobody` -x /etc/tcp.smtp.cdb 0 smtp
  $rblsmtpd /usr/sbin/qmail-smtpd 2>&1
  | $logger &"

by these ones :

sh -c "start-stop-daemon --start --quiet --user qmaild
    --pidfile /var/run/tcpserver_smtpd.pid --make-pidfile
    --exec /usr/bin/tcpserver -- -R
    -u `id -u qmaild` -g `id -g nobody` -x /etc/tcp.smtp.cdb 0 smtp
    $rblsmtpd /usr/sbin/qmail-smtpd /usr/sbin/vchkpw /bin/true 2>&1
    | $logger &"

Then, we have to setuid /usr/sbin/vchkpw :

chmod u+s /usr/sbin/vchkpw

and restart Qmail :

/etc/init.d/qmail stop
/etc/init.d/qmail start

The result

You will now be able to use the SMTP PLAINTEXT authentification. You just have to configure your favorite mail transport agent…

Feel free to Flattr this post at flattr.com, if you like it.