HowTo Mail Backup - Ikujam » Historique » Version 2
Version 1 (iku jam, 24/01/2012 13:09) → Version 2/19 (iku jam, 29/01/2012 20:33)
h1. HowTo Mail Backup - Ikujam
{{>toc}}
*WIP...*
# Presentation
# Requirements
# dns setup on host
## master for tld .test
## static IPs for vms
# 2 vms mail1.test mail2.test
## postfix
## postfixadmin
## postgresql
## pgpool for postgres replication
## inotify-based sync on maildir
## roundcube as mail client
## mailman as list manager
# test setup
## default case
## failover scenario
h2. Presentation
several projects with mail servers
request of certain stability, needed documentation
free software user, activist and contributor
idea is to produce a complete test environment with vms on a single machine
CC-NC-SA
h2. Requirements
to follow you need some linux admin skills:
* basic shell (bash)
* at least basic knowledge of debian package system (install & setup packages with apt-get, manage services)
* able to setup ssh public key authentication
* i don't like nano, feel free to use it - or another editor - instead of vi
h3. Host system
* debian
* qemu-kvm
* bind
This howto hwtoo uses
# cat /etc/debian_version
wheezy/sid
# uname -a
Linux master 3.1.0-1-amd64 #1 SMP Sun Dec 11 20:36:41 UTC 2011 x86_64 GNU/Linux
h3. Mail Server VMs
* debian
* debian packages for the different software
root@mail1:~# echo "mail1" > /etc/hostname
root@mail1:~# apt-get install inotify-tools rsync openssh-server pgpool javascript-common apache2 libapache2-mod-php5 roundcube postgresql postfix postfix-pgsql mailman roundcube-pgsql libc-client2007e mlock php5-imap postgrey courier-authlib-postgresql sasl2-bin courier-authdaemon libsasl2-modules-sql courier-imap-ssl --no-install-recommends
* use default options for roundcube, courier & mailman for now
** ident authentication
** dbconfig
** pgsql as database choice
** mailman language as you prefer
* install postfixadmin :
root@mail1:~# lynx 'http://downloads.sourceforge.net/project/postfixadmin/postfixadmin/postfixadmin-2.3.4/postfixadmin_2.3.4_all.deb'
* use default options for now
* just as of personal habit, some tools i use
root@mail1:~# apt-get install lynx less mc vim
root@mail1:~# cat /etc/debian_version
wheezy/sid
root@mail1:~# uname -a
Linux mail1.test 3.1.0-1-amd64 #1 SMP Tue Jan 10 05:01:58 UTC 2012 x86_64 GNU/Linux
root@mail2:~# cat /etc/debian_version
wheezy/sid
root@mail2:~# uname -a
Linux mail2 3.1.0-1-amd64 #1 SMP Fri Dec 23 16:37:11 UTC 2011 x86_64 GNU/Linux
root@mail2:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.122.3
netmask 255.255.255.0
network 192.168.122.0
broadcast 192.168.122.255
gateway 192.168.122.1
h2. dns setup on host
root@quadebian:/etc/bind# cat db.192.168.122
;
; BIND reverse data file for test
;
$TTL 604800
@ IN SOA master.test. root.master.test. (
1 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS master.test.
1 IN PTR master.test.
2 IN PTR mail1.test.
3 IN PTR mail2.test.
root@quadebian:/etc/bind# cat db.test
;
; BIND data file for test
;
$TTL 604800
@ IN SOA master.test. info.master.test. (
2 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS master.test.
test. IN MX 10 mail1.test.
test. IN MX 20 mail2.test.
master IN A 192.168.122.1
mail1 IN A 192.168.122.2
mail2 IN A 192.168.122.3
root@quadebian:/etc/bind# named-checkzone test db.test
zone test/IN: loaded serial 2
OK
* pass kvm dns server in forward mode on host node (default net config)
root@quadebian:/etc/bind# virsh
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh # net-dumpxml default
<network>
<name>default</name>
<uuid>0529cc34-c2ad-9663-0f42-5b338b14a6e4</uuid>
<forward mode='nat'/>
<bridge name='virbr0' stp='on' delay='0' />
<mac address='52:54:00:37:85:D8'/>
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254' />
</dhcp>
</ip>
</network>
h3. vm dns config
* change requires to reaffect NICs via virt-manager
** remove nic (and /etc/udev/rules.d/70-persistent-net.rules - it keeps track of different nics on the system, avoids getting eth2/3/4...)
** create new nic on default network
** reboot vm
** test connectivity & bind (set nameserver to 192.168.122.1 in /etc/resolv.conf)
h3. tests to do
* open http://mail1.test/roundcube & http://mail1.test/postfixadmin in a browser
** roundcube -> 404
** postfixadmin -> ok
* dns
root@quadebian:/etc/bind# dig mx test
; <<>> DiG 9.7.3 <<>> mx test
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26405
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 3
;; QUESTION SECTION:
;test. IN MX
;; ANSWER SECTION:
test. 604800 IN MX 20 mail2.test.
test. 604800 IN MX 10 mail1.test.
;; AUTHORITY SECTION:
test. 604800 IN NS master.test.
;; ADDITIONAL SECTION:
mail1.test. 604800 IN A 192.168.122.2
mail2.test. 604800 IN A 192.168.122.3
master.test. 604800 IN A 192.168.122.1
;; Query time: 2 msec
;; SERVER: 10.11.12.126#53(10.11.12.126)
;; WHEN: Tue Jan 24 09:55:25 2012
;; MSG SIZE rcvd: 135
h2. Server configuration
h3. postfix
root@mail2:/etc/postfix# mv main.cf main.cf.debian
root@mail2:/etc/postfix# vi main.cf
root@mail2:/etc/postfix# mkdir pgsql
root@mail2:/etc/postfix# vi pgsql/virtual_alias_maps.cf
root@mail2:/etc/postfix# vi pgsql/virtual_domain_maps.cf
root@mail2:/etc/postfix# vi pgsql/relay_domains.cf
root@mail2:/etc/postfix# vi pgsql/virtual_mailbox_limits.cf
root@mail2:/etc/postfix# vi pgsql/virtual_mailbox_maps.cf
root@mail2:/etc/courier# vi /etc/mailname
root@mail2:/etc/courier# cat /etc/postfix/transport
lists.test mailman:
root@mail2:/etc/courier# postmap /etc/postfix/transport
root@mail2:/etc/postfix# scp -r . mail1.test:/etc/postfix/
root@mail1:/etc/postfix# vi main.cf
# change following line :
mydestination = test,mail1.test,localhost.test, localhost
h3. saslauthd
* change /etc/default/saslauthd
START=yes
MECHANISMS="rimap"
OPTIONS="-c -r -O localhost -m /var/run/saslauthd"
h3. postfixadmin
*Only on mail1* : mail2 will be synced through logshipping/PITR slony ;)
* open
http://mail1.test/postfixadmin/setup.php
* set password and replace specified line in /etc/postfixadmin/config.inc.php :
$CONF['setup_password'] = 'changeme';
* create superadmin account using a local or valid email address (if you have internet access)
* modify /usr/share/postfixadmin/functions.inc.php
** this is in order to allow local domains, e.g. @.test@
_lignes 232++_
<pre>
if (!preg_match ('/^([-0-9A-Z]+\.)+' . '([0-9A-Z]){2,6}$/i', ($domain)))
{
if (!preg_match ('/^([-0-9A-Z]){3,16}$/i', ($domain)))
{
flash_error(sprintf($PALANG['pInvalidDomainRegex'], htmlentities($domain)));
return false;
}
}
</pre>
h3. courier
root@mail1:/etc/courier# vi authdaemonrc
root@mail2:/etc/courier# mv authpgsqlrc authpgsqlrc.debian
root@mail2:/etc/courier# vi authpgsqlrc
root@mail2:/etc/courier# mv imapd imapd.debian
root@mail2:/etc/courier# vi imapd
root@mail2:/etc/courier# mv imapd-ssl imapd-ssl.debian
root@mail2:/etc/courier# vi imapd-ssl
h3. roundcube
* activate webapp
** uncomment two alias directives inside /etc/apache2/conf.d/roundcube
** adapt config :
$rcmail_config['default_host'] = 'localhost';
$rcmail_config['smtp_server'] = 'localhost';
* /etc/init.d/apache2 reload
h3. ssh
* generate pair of keys on mail1 & mail2
# su mail
$ bash
mail@mail2:/etc/postfix$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/mail/.ssh/id_rsa):
Created directory '/var/mail/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/mail/.ssh/id_rsa.
Your public key has been saved in /var/mail/.ssh/id_rsa.pub.
The key fingerprint is:
b9:bf:63:05:c0:9f:4f:07:82:d9:fd:79:99:cf:20:20 mail@mail2
The key's randomart image is:
+--[ RSA 2048]----+
| . + . |
| E + o |
| + + o .o|
| .+ o =o.|
| S + o +.|
| . o o|
| . . |
| .o |
| .oo |
+-----------------+
* add mail1's public key to mail1's authorized keys
mail@mail1:/$ cp /var/mail/.ssh/id_rsa.pub /var/mail/.ssh/authorized_keys
* add mail1's public key to mail2's authorized keys
mail@mail2:/$ vi /var/mail/.ssh/authorized_keys
mail@mail2:/$ chmod 0600 /var/mail/.ssh/authorized_keys
* test connection
mail@mail1:/etc/courier$ ssh mail2.test
The authenticity of host 'mail2.test (192.168.122.3)' can't be established.
ECDSA key fingerprint is cb:a6:dd:64:03:ba:45:61:a3:b8:14:3a:05:89:ab:b3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mail2.test,192.168.122.3' (ECDSA) to the list of known hosts.
Linux mail2 3.1.0-1-amd64 #1 SMP Fri Dec 23 16:37:11 UTC 2011 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
$ hostname
mail2
$ logout
h3. inotify/rsync
# gen ssh key for user @mail@ on mail1 & copy public key to mail2
* create sync script
mail@mail1:/etc/courier$ vi ~/sync.sh
<pre>
#!/bin/sh
BASEDIR="$1"
REMOTE_HOST="$2"
RSYNC_OPTIONS="-rtlavz -e ssh --delete"
# Initial sync
rsync ${RSYNC_OPTIONS} ${BASEDIR}/ ${REMOTE_HOST}:${BASEDIR}
# Wait for events to trigger rsync
inotifywait --format '%e %w' -e close_write -e move -e create -e delete -qmr $BASEDIR | while read EVENT
do
# Fork off rsync proc to do sync
rsync ${RSYNC_OPTIONS} ${BASEDIR}/ ${REMOTE_HOST}:${BASEDIR} &
done
</pre>
root@mail1:/etc/courier# mkdir /var/log/mail
root@mail1:/etc/courier# chown mail:mail /var/log/mail
root@mail1:/etc/courier# vi /etc/rc.local
## ajout de la ligne :
su mail -l -c " nohup sh ~/sync.sh /var/mail/ mail2.test 2>&1 >> /var/log/mail/sync.log &"
root@mail1:/etc/courier# sh /etc/rc.local
nohup: ignoring input and redirecting stderr to stdout
root@mail1:/etc/courier# su mail
mail@mail1:/etc/courier$ chmod 0700 ~/sync.sh
h3. postgresql PITR
# gen ssh key for user @postgres@ on mail1 & copy public key to mail2
http://www.postgresql.org/docs/8.4/static/continuous-archiving.html
http://wiki.postgresql.org/wiki/Warm_Standby
* stop postresql on
mail@mail1:~$ sh sync.sh /var/mail/ mail2.test
* do a full sync of the database
# su postgres
$ rsync -a /var/lib/postgresql/9.1/main/ postgres@mail2.test://var/lib/postgresql/9.1/main/
on mail1 :
archive_command = 'rsync -a /var/lib/postgresql/9.1/main/%p postgres@mail2.test://var/lib/postgresql/9.1/wal/pg_xlog/%f'
* restart postgresql
on mail2 :
root@mail2:~#
root@mail1:/etc/courier# mkdir /var/lib/postgresql/9.1/wal
root@mail2:~# /var/log/mail
root@mail1:/etc/courier# chown postgres:postgres mail:mail /var/lib/postgresql/9.1/wal
root@mail2:/var/lib/postgresql/9.1/main# /var/log/mail
root@mail1:/etc/courier# vi recovery.conf
restore_command = 'cp /var/lib/postgresql/9.1/wal/pg_xlog/%f "%p"'
* on first startup,
* recovery.conf will be renamed to recovery.done after recovery
** rename recovery.done to recovery.conf and restart postgresql to sync with latest logs from master.
/etc/rc.local
h3. pgpool
TODO
h2. putting pieces together
* recover postfixadmin on mail1 password from @/etc/postfixadmin/config.inc.php@ /etc/postfixadmin/config.inc.php :
$CONF['database_password'] = 'GENERATED PASSWORD';
* apply it to @/etc/postfixadmin/config.inc.php@ on mail2
* apply it to the different files (mail1 & mail2):
:
for i in /etc/postfix/pgsql/virtual_alias_maps.cf /etc/postfix/pgsql/virtual_domain_maps.cf /etc/postfix/pgsql/relay_domains.cf /etc/postfix/pgsql/virtual_mailbox_limits.cf /etc/postfix/pgsql/virtual_mailbox_maps.cf ; do sed -i "s/PASSWORD/GENERATED PASSWORD/" $i ; done
vi /etc/courier/authpgsqlrc
* restart courier authdaemon :
/etc/init.d/courier-authdaemon restart
* create account via postfixadmin
** login to http://mail1.test/postfixadmin/login.php
** add domain (Domain list -> new domain)
*** domain name : "test"
** add mailbox (Virtual list -> add mailbox)
* verify domain & mailbox creation
* send testmail in commandline on master (apt-get install bsd-mailx)
* verify replication of maildir on mail2
* roundcube
** connect on http://mail1.test/roundcube with test@test
** send test mail to outside (may be rejected/filtered as spam since "test" emaildomain isn't valid, should work with a public MX DNS entry)
h3. vacation/responder
root@mail1:~# apt-get install git-core --no-install-recommends
root@mail1:~# cd /usr/share/roundcube/plugins/ && git clone https://github.com/bhuisgen/rc-vacation.git vacation
root@mail1:/usr/share/roundcube/plugins# mkdir /etc/roundcube/plugins/vacation
root@mail1:/usr/share/roundcube/plugins# ln -s /usr/share/roundcube/plugins/vacation/config.inc.php /etc/roundcube/plugins/vacation/
root@mail1:/usr/share/roundcube/plugins# cd vacation/
root@mail1:/usr/share/roundcube/plugins/vacation# cp config.inc.php.dist config.inc.php
root@mail1:/usr/share/roundcube/plugins/vacation# vi config.inc.php
root@mail1:/usr/share/roundcube/plugins/vacation# ln -s /usr/share/roundcube/plugins/vacation/ /var/lib/roundcube/plugins/
* edit /etc/roundcube/main.inc.php
$rcmail_config['vacation_sql_dsn'] =
'pgsql://postfixadmin:PASSWORD@localhost/postfixadmin';
* test in roundcube settings, you should have a new tab "vacation/répondeur"
h2. failover
* in case of a failover of mail1, mail2 should be available to receive mails and provide access to all the mails that were on mail1
** when mail1 comes back up online, it needs to synchronize with mail2 before
* in case of a failover of mail2, mail1 should not be impacted
h2. References
http://chiliproject.tetaneutral.net/projects/tetaneutral/wiki/Serveur_Mail_tetalab
http://www.kutukupret.com/2011/06/28/postfix-one-way-maildir-replication-backup-using-inotify-and-rsync/
http://www.postgresql.org/docs/9.1/interactive/continuous-archiving.html
http://www.pgpool.net/pgpool-web/contrib_docs/pgpool-II_for_beginners.pdf