My backup system

Backups are important and I think everyone will agree on this. Previously, I had a really poor backup system which was just a bunch of bash scripts running every day.

I wanted to change that so I duck’d around and found out about an amazing open source backup tool called restic. It’s a full fledged backup tool for your bare metal OS. They have an amazing documentation which made everything easy to setup.

Initial Configuration

Alright. Let’s first install restic. It’s available in most of the native package managers. For me, it’s ubuntu:

sudo apt install restic

So, how does restic work? Well, a very high level overview of it would be something like this:

We start with a repository, which is encrypted with a password. After that we do restic backup which creates a snapshot inside that repository. We can create as many snapshots as we’d like, given that we have the required space to store it.

Let’s run some commands

There’s this quickstart guide which does a wonderful job for explaining the basic operations that can be performed with restic.

Side note: We will be automating the whole backup process and setup monitoring in the later part. Setting the environment variables in the beginning gave me some issues which is why I will set it a bit later.

Initialize the repository:

restic init -r restic-repo

Backup a directory:

restic -r restic-repo backup test-backup

List snapshots:

restic -r restic-repo snapshots

Designing the backup setup

It’s not a full proof backup design but it’s good enough for my use case. For now, I will be sending backups to only 1 server. Eventually, I want to have multiple locations an one centralized server which takes backup of all the systems.

I will be creating 2 repositories. One main repository and the one copy of it which will be transferred to a different location.

Let’s start by initializing the main repository:

restic -r server-backup init

Now, the copy of it:

restic -r server-backup-copy init

Save the password to environment variable, RESTIC_PASSWORD for repository 1 and RESTIC_PASSWORD2 for repository 2:

export RESTIC_PASSWORD=secret-password
export RESTIC_PASSWORD2=another-secret-password

Now, we will take a backup of our home directory. Before we do that, let’s look at the exclude flag from the documentation.

We want to exclude the restic repositories which are in the home directory. Make an exclude.txt file and specify the path of the directories

server-backup/
server-backup-copy/

After that:

restic -r server-backup backup . --exclude-file=excludes.txt

This will be running daily which will create snapshots everyday.

We can see the snapshots using:

restic -r server-backup snapshots

Now let’s look at the copy command:

For getting the password of 2nd repository, we will use the --password-command2 flag whose value would be the environment variable we set earlier.

Run the copy command:

restic copy -r server-backup --repo2 server-backup-copy --password-command2="echo $RESTIC_PASSWORD2"

We are done with the backup setup. One more thing I would like to do is, keep removing the old snapshots to save up some space.

There’s forget command which can be used to delete snapshots.

I would like to keep the snapshots of the last 5 days. We can use --keep-daily flag for keeping 5 day’s snapshots.

restic -r server-backup forget --keep-daily 5

We are done with the restic part. There’s still a lot you can do here like checking integrity of the backup, data verification, etc. The documentation will help you with that. For now, let’s move onto the second part.

Rsync and bash

I have uploaded all the scripts on this github repo: backup.

First, I will write down all the commands in a bash script:

restic -r server-backup backup . --exclude-file=excludes.txt
restic copy -r server-backup --repo2 server-backup-copy --password-command2="echo $RESTIC_PASSWORD2"
restic -r server-backup forget --keep-daily 5

Save it as restic.sh. We will use rsync for transferring the server-backup-copy repository to another server:

rsync -aP -e "ssh -i ~/.ssh/key" server-backup-copy 
root@server:/root/restic-backup --stats

Replace server with your desired hostname or IP.

Save it as rsync.sh.

After that, we will be using another bash script for running those bash scripts and use Telegram bot for sending updates. (Edit: Instead of a python script, I have resorted to a bash script becasue of weird shenanigans with python on a cronjob)

Create a Telegram Bot. And paste it’s token in the script.

#!/bin/bash

export RESTIC_PASSWORD= # password of original restic repo
export RESTIC_PASSWORD2= # password of copy restic reop

# Logging function
log() {
  timestamp=$(date "+%Y-%m-%d %H:%M:%S")
  echo "$timestamp - $1"
}

# Send Telegram notification
send_telegram_notification() {
  message="$1"
  BOT_TOKEN="token" # bot token
  GROUP_ID="-1234"  # chat_id

  curl -s --data "text=$1" --data "chat_id=$GROUP_ID" 'https://api.telegram.org/bot'$BOT_TOKEN'/sendMessage' > /dev/null
}

# Execute backup command
execute_backup() {
  log "INFO: Executing backup command..."
  cmd="sh $HOME/restic.sh"
  eval "$cmd" >> $HOME/log.txt

  if [ $? -eq 0 ]; then
    log "INFO: Backup command executed successfully."
    return 0
  else
    log "ERROR: Backup command failed to execute."
    return 1
  fi
}

# Execute rsync command
execute_rsync() {
  log "INFO: Executing rsync command..."
  cmd="sh $HOME/rsync.sh"
  eval "$cmd"

  if [ $? -eq 0 ]; then
    log "INFO: rsync command executed successfully."
    return 0
  else
    log "ERROR: rsync command failed to execute."
    return 1
  fi
}

# Main
log "INFO: Starting backup script..."

if execute_backup; then
  backup_message="Backup job completed at $(date "+%Y-%m-%d %H:%M:%S"). Doing rsync transfer now."
  log "INFO: $backup_message"
  send_telegram_notification "$backup_message"
else
  log "ERROR: Backup job failed to execute."
  send_telegram_notification "Backup job failed to execute."
  exit 1
fi

if execute_rsync; then
  rsync_message="rsync transfer complete to crunchbits server."
  log "INFO: $rsync_message"
  send_telegram_notification "$rsync_message"
else
  log "ERROR: rsync transfer failed."
  send_telegram_notification "rsync transfer failed."
fi

log "INFO: Script completed."

It’s a fairly straightforward bash script. First, it will execute the restic.sh script and wait for it to finish.

After that, it will run rsync.sh and perform the same operations.

All that’s left to do is put it in the cronjob so that it will run everyday (for those unfamiliar with cronjob, checkout cronhelp):

crontab -e

0 2 * * * * sh backup.sh

Well, that’s it. Thank you for sticking till the end!

Keep experimenting!