Backup » Historique » Version 21
Version 20 (Nicolas BERTRAND, 11/01/2017 16:31) → Version 21/24 (Laurent GUERBY, 27/06/2017 10:59)
{{>toc}}
h1. Backup
h2. Liens
* [[Backup_infra]]
* [[Apt_Backports_TTNN]] borg debian packaging
* https://www.reddit.com/r/linux/comments/42feqz/i_asked_here_for_the_optimal_backup_solution_and/
h2. BackupPC
* http://backuppc.sourceforge.net/
h2. Attic
* https://attic-backup.org/
* https://lists.tetaneutral.net/pipermail/technique/2015-September/001971.html
** Logiciel de backup : choix de attic
h2. Borg
* https://github.com/borgbackup
* http://readthedocs.org/projects/borgbackup/
* http://puppet.tetaneutral.net/pool/main/b/borgbackup/
* http://puppet.tetaneutral.net/dists/
Pour un peu automatiser + cron, on peut utiliser borgmatic:
* https://github.com/witten/borgmatic
h2. Borg tips
* On debian systems to avoid backuping cache (apt & others) use borg --exclude-caches and do once :
<pre>
echo Signature: 8a477f597d28d172789f06886806bc55 > /var/cache/CACHEDIR.TAG
</pre>
http://borgbackup.readthedocs.io/en/stable/usage.html#borg-create
http://www.brynosaurus.com/cachedir/spec.html
h2. Borg script
*Note 20160428* : le script est probablement inutile cf https://github.com/borgbackup/borg/issues/994
Pour une machine qui va etre eteinte et rallumée de maniere non controlable.
Creation initiale avec un user normal capable de ssh sur MACHINE:PORT :
<pre>
borg init --encryption keyfile ssh://USER@MACHINE:PORT/some/where/borg/NICK-repo
</pre>
On A 2 types d'encryption keyfile et repokey
* en keyfile: la clef est stockée 'localement' est doit aussi être backupé. mode "passphrase + key"
* repokey : la clef est stocké dans le repo only: mode "passphrase only"
<pre>
11.1026 < guerby> [09:39:25] zorun, tu as une ref dans la doc ? ce que j'ai trouvé http://borgbackup.readthedocs.io/en/stable/quickstart.html#encrypted-repos
11.1026 < guerby> [09:39:38] "so you still have the key in case it gets corrupted or lost. Also keep your passphrase at a safe place."
11.1026 < zorun> [09:50:10] guerby: cherche « repokey » dans https://borgbackup.readthedocs.io/en/stable/usage.html
11.1026 < zorun> [09:50:19] # Local repository (default is to use encryption in repokey mode)
11.1026 < zorun> [09:50:27] If you want “passphrase-only” security, use the repokey mode. The key will be stored inside the repository (in its “config” file). In above mentioned attack scenario, the attacker will have the key (but not the passphrase).
11.1026 < zorun> [09:50:43] If you want “passphrase and having-the-key” security, use the keyfile mode. The key will be stored in your home directory (in .config/borg/keys). In the attack scenario, the attacker who has just access to your repo won’t have the key (and also not the passphrase).
11.1026 < taziden> [09:51:21] et la méthode par défaut, c'est repokey
</pre>
Et setup cron + script :
<pre>
# crontab -l
@reboot /root/cron-borg.sh
# cat /root/cron-borg.sh
#!/bin/bash
export LANG=en_US.UTF-8
mkdir -p /root/borg >& /dev/null
sleep 300
echo === start === $(date) >> /root/borg/cron.log
NICK=myhost
REPO=ssh://USER@MACHINE:PORT/some/where/borg/${NICK}-repo
export BORG_PASSPHRASE=lalalala
if [ -f /root/borg/stamp ]; then
STAMP=$(cat /root/borg/stamp)
borg break-lock $REPO
else
STAMP=$(date '+%Y%m%dT%H%M%S')
if [ -f /root/borg/previous-stamp ]; then
PREVIOUS_STAMP=$(cat /root/borg/previous-stamp)
while [ "${STAMP%T*}" = "${PREVIOUS_STAMP%T*}" ]; do
STAMP=$(date '+%Y%m%dT%H%M%S')
echo === delay === $(date) >> /root/borg/cron.log
sleep 1h
done
fi
echo $STAMP > /root/borg/stamp
fi
borg create --compression lz4 --stats --verbose \
--exclude /root/borg --exclude '/home/*/.cache' --exclude-caches --one-file-system \
${REPO}::${NICK}-$STAMP / >> /root/borg/log-$STAMP 2>> /root/borg/err-$STAMP
res=$?
if [ $res -eq 0 -o $res eq 1 ]; then
mv -f /root/borg/stamp /root/borg/previous-stamp >& /dev/null
rm -f /root/borg/stamp >& /dev/null
fi
echo === done === $res === $(date) >> /root/borg/cron.log
exec "$0"
</pre>
h2. Migration Attic vers Borg
* https://github.com/borgbackup/borg/pull/231
* old: https://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/puppet-backup
* new: https://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/puppetmaster/revisions/master/entry/modules/ttnn/manifests/backup.pp
** git history BackupPC => Attic => Borg
<pre>
# apt-get -t jessie-backports install borgbackup
$ cd /backup/attic/
$ borg upgrade <repo>
$ borg check --repair <repo>
$ mv <repo> ../borg/
$ chown -R backupinfra: /backup/borg/<repo>
</pre>
Dans le module puppet, le changement le plus important est le parametre compression explicite pour correspondre au défaut de attic create :
<pre>
attic create ... <repo> -> borg create --compression zlib,6 ... <repo>
</pre>
Sinon le prochain backup sera non compressé, et aucun nouveau chucks ne correspondra aux anciens -> perte de la dedup. "zlib,6" étant le niveau de compression de attic.
https://github.com/jborg/attic/issues/299
http://borgbackup.readthedocs.org/en/stable/usage.html#environment-variables
<pre>
export ATTIC_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
</pre>
h1. Backup
h2. Liens
* [[Backup_infra]]
* [[Apt_Backports_TTNN]] borg debian packaging
* https://www.reddit.com/r/linux/comments/42feqz/i_asked_here_for_the_optimal_backup_solution_and/
h2. BackupPC
* http://backuppc.sourceforge.net/
h2. Attic
* https://attic-backup.org/
* https://lists.tetaneutral.net/pipermail/technique/2015-September/001971.html
** Logiciel de backup : choix de attic
h2. Borg
* https://github.com/borgbackup
* http://readthedocs.org/projects/borgbackup/
* http://puppet.tetaneutral.net/pool/main/b/borgbackup/
* http://puppet.tetaneutral.net/dists/
Pour un peu automatiser + cron, on peut utiliser borgmatic:
* https://github.com/witten/borgmatic
h2. Borg tips
* On debian systems to avoid backuping cache (apt & others) use borg --exclude-caches and do once :
<pre>
echo Signature: 8a477f597d28d172789f06886806bc55 > /var/cache/CACHEDIR.TAG
</pre>
http://borgbackup.readthedocs.io/en/stable/usage.html#borg-create
http://www.brynosaurus.com/cachedir/spec.html
h2. Borg script
*Note 20160428* : le script est probablement inutile cf https://github.com/borgbackup/borg/issues/994
Pour une machine qui va etre eteinte et rallumée de maniere non controlable.
Creation initiale avec un user normal capable de ssh sur MACHINE:PORT :
<pre>
borg init --encryption keyfile ssh://USER@MACHINE:PORT/some/where/borg/NICK-repo
</pre>
On A 2 types d'encryption keyfile et repokey
* en keyfile: la clef est stockée 'localement' est doit aussi être backupé. mode "passphrase + key"
* repokey : la clef est stocké dans le repo only: mode "passphrase only"
<pre>
11.1026 < guerby> [09:39:25] zorun, tu as une ref dans la doc ? ce que j'ai trouvé http://borgbackup.readthedocs.io/en/stable/quickstart.html#encrypted-repos
11.1026 < guerby> [09:39:38] "so you still have the key in case it gets corrupted or lost. Also keep your passphrase at a safe place."
11.1026 < zorun> [09:50:10] guerby: cherche « repokey » dans https://borgbackup.readthedocs.io/en/stable/usage.html
11.1026 < zorun> [09:50:19] # Local repository (default is to use encryption in repokey mode)
11.1026 < zorun> [09:50:27] If you want “passphrase-only” security, use the repokey mode. The key will be stored inside the repository (in its “config” file). In above mentioned attack scenario, the attacker will have the key (but not the passphrase).
11.1026 < zorun> [09:50:43] If you want “passphrase and having-the-key” security, use the keyfile mode. The key will be stored in your home directory (in .config/borg/keys). In the attack scenario, the attacker who has just access to your repo won’t have the key (and also not the passphrase).
11.1026 < taziden> [09:51:21] et la méthode par défaut, c'est repokey
</pre>
Et setup cron + script :
<pre>
# crontab -l
@reboot /root/cron-borg.sh
# cat /root/cron-borg.sh
#!/bin/bash
export LANG=en_US.UTF-8
mkdir -p /root/borg >& /dev/null
sleep 300
echo === start === $(date) >> /root/borg/cron.log
NICK=myhost
REPO=ssh://USER@MACHINE:PORT/some/where/borg/${NICK}-repo
export BORG_PASSPHRASE=lalalala
if [ -f /root/borg/stamp ]; then
STAMP=$(cat /root/borg/stamp)
borg break-lock $REPO
else
STAMP=$(date '+%Y%m%dT%H%M%S')
if [ -f /root/borg/previous-stamp ]; then
PREVIOUS_STAMP=$(cat /root/borg/previous-stamp)
while [ "${STAMP%T*}" = "${PREVIOUS_STAMP%T*}" ]; do
STAMP=$(date '+%Y%m%dT%H%M%S')
echo === delay === $(date) >> /root/borg/cron.log
sleep 1h
done
fi
echo $STAMP > /root/borg/stamp
fi
borg create --compression lz4 --stats --verbose \
--exclude /root/borg --exclude '/home/*/.cache' --exclude-caches --one-file-system \
${REPO}::${NICK}-$STAMP / >> /root/borg/log-$STAMP 2>> /root/borg/err-$STAMP
res=$?
if [ $res -eq 0 -o $res eq 1 ]; then
mv -f /root/borg/stamp /root/borg/previous-stamp >& /dev/null
rm -f /root/borg/stamp >& /dev/null
fi
echo === done === $res === $(date) >> /root/borg/cron.log
exec "$0"
</pre>
h2. Migration Attic vers Borg
* https://github.com/borgbackup/borg/pull/231
* old: https://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/puppet-backup
* new: https://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/puppetmaster/revisions/master/entry/modules/ttnn/manifests/backup.pp
** git history BackupPC => Attic => Borg
<pre>
# apt-get -t jessie-backports install borgbackup
$ cd /backup/attic/
$ borg upgrade <repo>
$ borg check --repair <repo>
$ mv <repo> ../borg/
$ chown -R backupinfra: /backup/borg/<repo>
</pre>
Dans le module puppet, le changement le plus important est le parametre compression explicite pour correspondre au défaut de attic create :
<pre>
attic create ... <repo> -> borg create --compression zlib,6 ... <repo>
</pre>
Sinon le prochain backup sera non compressé, et aucun nouveau chucks ne correspondra aux anciens -> perte de la dedup. "zlib,6" étant le niveau de compression de attic.
https://github.com/jborg/attic/issues/299
http://borgbackup.readthedocs.org/en/stable/usage.html#environment-variables
<pre>
export ATTIC_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
</pre>