Dotdeb » Mail

Calculate statistics from your Qmail logfiles using Awstats

Guillaume Plessis — Mon, 13 Oct 2008 15:06:35 +0000

Awstats is a commonly used program to calculate web statistics from your webserver logfiles. It can detect useragents, referers, unique visitors… But one of its another feature is to build usage reports from your mail server’s logfiles, as seen on this demo.

Here is how to quickly configure Awstats to take profit of your Qmail log files…

First of all, you need to convert rour /var/log/mail.log file to make it readable to Awstats (supposing you’re using Qmail and Vpopmail from Dotdeb and Awstats from Debian’s official repositories).

Hopefully, a tool is provided with the awstats package : /usr/share/doc/awstats/examples/maillogconvert.pl

You can take a look at its output by executing the following command :

/usr/share/doc/awstats/examples/maillogconvert.pl vadmin /var/log/mail.log

(vadmin is here to consider the Vpopmail’s virtualdomain translation)

Great! We now just have to configure awstats to produce useful reports. Here is a sample configuration file (name it /etc/awstats/awstats.mail.conf, for example)… Please note the space after the pipe :

LogFile="/usr/share/doc/awstats/examples/maillogconvert.pl vadmin /var/log/mail.log | "
LogType=M
LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
SiteDomain="localhost"
HostAliases="127.0.0.1 @/etc/qmail/rcpthosts"
LevelForBrowsersDetection=0 # 0 disables Browsers detection.
LevelForOSDetection=0 # 0 disables OS detection.
LevelForRefererAnalyze=0 # 0 disables Origin detection.
LevelForRobotsDetection=0 # 0 disables Robots detection.
LevelForSearchEnginesDetection=0 # 0 disables Search engines detection.
LevelForKeywordsDetection=0 # 0 disables Keyphrases/Keywords detection.
LevelForFileTypesDetection=0 # 0 disables File types detection.
ShowSummary=HB
ShowMonthStats=HB
ShowDaysOfMonthStats=HB
ShowDaysOfWeekStats=HB
ShowHoursStats=HB
ShowDomainsStats=0
ShowHostsStats=HBL
ShowRobotsStats=0
ShowEMailSenders=HBML
ShowEMailReceivers=HBML
ShowSessionsStats=0
ShowPagesStats=0
ShowFileTypesStats=0
ShowOSStats=0
ShowBrowsersStats=0
ShowOriginStats=0
ShowKeyphrasesStats=0
ShowKeywordsStats=0
ShowMiscStats=0
ShowHTTPErrorsStats=0
ShowSMTPErrorsStats=1

Now, just add this line to /etc/cron.d/awstats to compute your logfiles every 10 minutes:

0,10,20,30,40,50 * * * * root [ -x /usr/lib/cgi-bin/awstats.pl ] && /usr/lib/cgi-bin/awstats.pl -config=mail -update &>/dev/null

(Let’s enjoy a 10-minutes break or brew a good coffee, waiting for the stats to be calculated…)

Your mailserver’s statistics will shortly be available at this URL : http://yourserver/cgi-bin/awstats.pl?config=mail.

Vpopmail 5.4.25 updated

Guillaume Plessis — Thu, 09 Oct 2008 12:16:21 +0000

The Vpopmail 5.4.25 packages has been updated to fix a problem with passwords longer than 8 characters.

The /etc/init.d/vpopmail script has been updated to launch simultaneously :

the (enabled by default) POP3 server
the (disabled by default) POP3-SSL server
the (disabled by default) Vpopmaild server

depending on the settings in /etc/vpopmail/options.

Qmailadmin 1.2.11 updated

Guillaume Plessis — Mon, 29 Sep 2008 11:59:05 +0000

I just updated Qmailadmin 1.2.11 for Etch amd64/i386 to include some changes :

instead of a checkbox to enable/disable spam filtering on pop accounts, there is now a list to chose what to do (no filtering, marking spams, deleting spams, learn spam, learn ham). This allow anyone to easily train your bayesian filters.
Spamassassin is now recommanded and will replace bogofilter in a near future
Clamassassin replaces clamfilter.pl due to performance issues. Please DO update the /etc/procmailrc-* files during the upgrade to ensure virus are scanned.

Vpopmail 5.4.25 is now packaged

Guillaume Plessis — Wed, 24 Sep 2008 21:55:32 +0000

Vpopmail 5.4.25 has been packaged for Etch amd64/i386 to fix some annoying bugs. It’s safe and recommanded to upgrade your servers to this version.

For more information, read the official Changelog.

ClamAV removed from Dotdeb

Guillaume Plessis — Mon, 22 Sep 2008 08:50:36 +0000

FYI, ClamAV has been removed from Dotdeb, since it is more actively maintained on Debian Volatile.

Volatile is now part of the Debian project and will be activated by default on Debian Lenny. Its goal is to provide up-to-date packages of some specific pieces of software (ClamAV, Spamassassin for example). Please use it instead of Dotdeb for your mail-filtering packages.

How to bind your Qmail server to a specific IP address

Guillaume Plessis — Wed, 27 Aug 2008 13:04:43 +0000

Just a little tip :

By default, Qmail listens to all the available IP address on the machine (0.0.0.0). It is possible to change this behaviour to bind Qmail to a specific IP.

We suppose that you are using Qmail from Dotdeb and launching it using the provided init script, /etc/init.d/qmail. Just edit it and change these lines :

          sh -c "start-stop-daemon --start --quiet --user qmaild \
          --pidfile /var/run/tcpserver_smtpd.pid --make-pidfile \
          --exec /usr/bin/tcpserver -- -H -P -R -l 0 \
          -u `id -u qmaild` -g `id -g nobody` -x /etc/tcp.smtp.cdb 0 smtp \
          $rblsmtpd /usr/sbin/qmail-smtpd 2>&1 \
          | $logger &"

by those ones :

          sh -c "start-stop-daemon --start --quiet --user qmaild \
          --pidfile /var/run/tcpserver_smtpd.pid --make-pidfile \
          --exec /usr/bin/tcpserver -- -H -P -R -l 0 \
          -u `id -u qmaild` -g `id -g nobody` -x /etc/tcp.smtp.cdb xxx.xxx.xxx.xxx smtp \
          $rblsmtpd /usr/sbin/qmail-smtpd 2>&1 \
          | $logger &"

(Just replace xxx.xxx.xxx.xxx by your IP address)

Now, when you list your listening dameon, you should see tcpserver listening to xxx.xxx.xxx.xxx:25 instead of 0.0.0.0:25.

machine# netstat -apn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 xxx.xxx.xxx.xxx:25      0.0.0.0:*               LISTEN      21175/tcpserver

How to enable greylisting on your Qmail server

Guillaume Plessis — Sun, 24 Aug 2008 18:26:52 +0000

Greylisting is very useful to avoid most of the incoming spam on your mail server. The Qmail packages shipped on Dotdeb have built-in MySQL-based greylisting. Here is how to enable it…

Installation

First of all, be sure to have Qmail installed from Dotdeb with a version number greater than 1.03-37.dotdeb.1.
Then, since we’ll use a MySQL backend to share the greylisting database between several servers, be sure to have MySQL installed somewhere on your boxes.

In our example, we’ll create the database (we’ll name it relaydelay) and grant access to user@host with the password “password“. You’re free to change these values to fit your needs…

CREATE DATABASE relaydelay;
GRANT ALL PRIVILEGES ON `relaydelay`.* TO 'user'@'localhost' IDENTIFIED BY 'password';

Then, we create the needed tables :

CREATE TABLE relaytofrom
(
  id              bigint          NOT NULL        auto_increment,
  relay_ip        char(16),
  mail_from       varchar(255),
  rcpt_to         varchar(255),
  block_expires   datetime        NOT NULL,
 
  record_expires  datetime        NOT NULL,
  blocked_count   bigint          default 0 NOT NULL,
  passed_count    bigint          default 0 NOT NULL,
  aborted_count   bigint          default 0 NOT NULL,
  origin_type     enum("MANUAL","AUTO") NOT NULL,
  create_time     datetime        NOT NULL,
  last_update     timestamp       NOT NULL,
 
  primary key(id),
  key(relay_ip),
  key(mail_from(20)),
  key(rcpt_to(20))
);
 
CREATE TABLE dns_name
(
  relay_ip      varchar(18)       NOT NULL,
  relay_name    varchar(255)      NOT NULL,
  last_update   timestamp         NOT NULL,
  primary key(relay_ip),
  key(relay_name(20))
);
 
CREATE TABLE mail_log
(
  id              bigint          NOT NULL        auto_increment,
  relay_ip        varchar(16)     NOT NULL,
  relay_name      varchar(255),
  dns_mismatch    bool            NOT NULL,
  mail_from       varchar(255)    NOT NULL,
  rcpt_to         varchar(255)    NOT NULL,
  rcpt_host       varchar(80)     NOT NULL,
  create_time     datetime        NOT NULL,
 
  primary key(id),
  key(relay_ip),
  key(mail_from(20)),
  key(rcpt_to(20))
);

We now have to put the appropriate settings in the /etc/init.d/qmail init script…

export MYSQLHOST="localhost"
export MYSQLUSER="user"
export MYSQLPASS="password"
export MYSQLDB="relaydelay"
export BLOCK_EXPIRE=5           # minutes until email is accepted
export RECORD_EXPIRE=600        # minutes until record expires
export RECORD_EXPIRE_GOOD=36
export LOCAL_SCAN_DEBUG=0

and in the /usr/sbin/greylisting-delete-expired :

MYSQLHOST="localhost"
MYSQLUSER="user"
MYSQLPASS="password"
MYSQLDB="relaydelay"

We can now relaunch Qmail and enable or disable the greylisting with a simple symbolic link :

/etc/init.d/qmail
ln -s /usr/bin/qmail-envelope-scanner /usr/sbin/qmail-envelope-scanner

Usage

Example wildcard whitelists for subnets :

INSERT INTO relaytofrom VALUES (0,"127.0.0.1"   ,NULL,NULL,"0000-00-00 00:00:00","9999-12-31 23:59:59",0,0,0,"MANUAL",NOW(),NOW());
INSERT INTO relaytofrom VALUES (0,"192.168"     ,NULL,NULL,"0000-00-00 00:00:00","9999-12-31 23:59:59",0,0,0,"MANUAL",NOW(),NOW());

Example wildcard whitelist entry for a recieved domain or subdomain

INSERT INTO relaytofrom VALUES (0,NULL,NULL,"sub.domain.com","0000-00-00 00:00:00","9999-12-31 23:59:59",0,0,0,"MANUAL",NOW(),NOW());

How to enable the SMTP authentification on your Qmail server

Guillaume Plessis — Sun, 24 Aug 2008 12:44:02 +0000

The Qmail package from Dotdeb has been built with SMTP-auth features (but disabled by default). Here is the way to use them…

Installation

First of all, we suppose that :

you installed the last Qmail and Vpopmail packages from Dotdeb
you’re launching them using the bundled init scripts (with tcpserver)

Configuration

To enable SMTP-auth, you just have to edit the /etc/init.d/qmail init script and replace these lines :

sh -c "start-stop-daemon --start --quiet --user qmaild
  --pidfile /var/run/tcpserver_smtpd.pid --make-pidfile
  --exec /usr/bin/tcpserver -- -R
  -u `id -u qmaild` -g `id -g nobody` -x /etc/tcp.smtp.cdb 0 smtp
  $rblsmtpd /usr/sbin/qmail-smtpd 2>&1
  | $logger &"

by these ones :

sh -c "start-stop-daemon --start --quiet --user qmaild
    --pidfile /var/run/tcpserver_smtpd.pid --make-pidfile
    --exec /usr/bin/tcpserver -- -R
    -u `id -u qmaild` -g `id -g nobody` -x /etc/tcp.smtp.cdb 0 smtp
    $rblsmtpd /usr/sbin/qmail-smtpd /usr/sbin/vchkpw /bin/true 2>&1
    | $logger &"

Then, we have to setuid /usr/sbin/vchkpw :

chmod u+s /usr/sbin/vchkpw

and restart Qmail :

/etc/init.d/qmail stop
/etc/init.d/qmail start

The result

You will now be able to use the SMTP PLAINTEXT authentification. You just have to configure your favorite mail transport agent…