Backup and recovery using Restic

Backups are encrypted and done daily at around 4 am via restic to storage1 and offsite to a Hetzner storage box paid by sandro.

storage1 houses a restic-server protected by basic auth. When configuring a new host make sure that the restic repository password and the basic auth credentials for the restic server are different.

Restore is done easiest by mounting the backup with restic mount /mnt and then using rsync -a /mnt/snapshots/latest/path /path to copy the files from the latest snapshot back to their original place.

Disaster disk recovery

In case of possible critical data loss call a member of AG Infra and cooperate to minimize the risk of accidental mistakes.

When a disk is failing and already has several bad sectors in SMART, then it should be replaced ASAP. ZFS snapshots cannot be copied with zfs send/receive when they contain a bad sector and will abort when encountering them. A half send snapshot can be resumed but it is not useful and cannot be mounted. Because of that it is best to use ddrescue to the bare block device to get the data of the disk ASAP. ddrescue can copy to another block device or to a file. A map file should always be supllied!

ddrescue /dev/sdx1 /dev/sdz1 /map

or

ddrescue /dev/sdx1 /disk.img /map

After that rsync can be used to copy the data of the bad file system. If there are bad blocks, some files cannot be copied/recovered. They need to be recovered via other methods. Rsync should be run with the following arguments: rsync -a --info=progress2 /mnt/old /mnt/new