UsBackup - Septimiu Ujica

README

Description

UsBackup is a backup software that allows files, datasets, vms, configs to be backed up in pull mode (via ssh). It is designed to run as a background process and to be as simple as possible.

It features a simple configuration file that allows you to configure different sources, storages and jobs. It also features a simple reporting system that can send reports via various handlers after the backup is finished.

Replication is also supported, allowing you to replicate existing backups to other locations. This is useful for offsite backups or as a part of the 3-2-1 backup strategy.

Files can be backed up from local storage or from remote storage, also they can be backed up using incremental, full or archive mode. (The first one being the most efficient in terms of space and time)

The recommended way to configure hosts is to use ssh keys. Passwords are supported but not recommended as they will be stored as plain text in the configuration file.

WARNING! If using UsBackup prior to version 2.0, please note that the configuration file format and backup directories structure have changed and they are not compatible with versions 1.x and 0.x. Please check the config.sample.yml file for the new format and migrate your configuration file accordingly. Version 1.x is still available in the legacy branch.

Features

Backup files (incremental, full, archive)
Backup FTP/FTPS sources (incremental, full)
Backup OpenWRT config
Backup Truenas config
Backup ZFS datasets
Backup Home Assistant config
Backup Proxmox VMs
Remote or local sources
Replication to local or remote hosts
Source configuration
Storage configuration
Job configuration
Notification configuration
Pre / post backup commands

Software requirements

python3
rsync
tar
ssh
sshpass (if using passwords for remote hosts – not recommended)

Installation

1. As a package

pip install --upgrade <git-repo>

git clone <git-repo>
cd <git-repo>
python setup.py install

2. As a standalone script

git clone <git-repo>

Usage

UsBackup can be used in 3 ways:

1. As a package (if installed globally)

/usr/bin/usbackup <parameters>

2. As a package (if installed in a virtualenv)

<path-to-venv>/bin/usbackup <parameters>

3. As a standalone script

<git-clone-dir>/run.py <parameters>

Check “Command line arguments” section for more information about the available parameters.

Command line arguments

usbackup [-h] [--config CONFIG_FILES] [--log LOG_FILE] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--version] {configtest,du,backup} ...

options:
  -h, --help            show this help message and exit
  --config CONFIG_FILES
                        Alternative config file(s)
  --log LOG_FILE        Log file where to write logs
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Log level
  --version             shows UsBackup version

Commands:
  {configtest,du,backup}
    daemon              Run as daemon and perform actions based on configured jobs

    run                 Run a job based on the provided paramaters
      options:
        --type {backup,replication}
                              The type of the job to run. Available types: backup, replication. Default: backup
        --dest DEST           Destination storage to be used when performing the job
        --replicate REPLICATE
                              Source storage to read the data from when performing the replication job - required when the job type is replication, otherwise ignored
        --replication-mode {incremental,full,archive}
                              Replication mode. Available options: incremental, full, archive. Default: incremental
        --limit LIMIT         List of sources for the job (if no sources are provided, all sources will be included, except the ones in the exclude list)
        --exclude EXCLUDE     List of sources to exclude from the job
        --retention-policy RETENTION_POLICY
                              Retention policy. last=<NR>,hourly=<NR>,daily=<daNRys>,weekly=<NR>,monthly=<NR>,yearly=<NR>. Example: --retention-policy last=6,hourly=24,daily=7,weekly=4,monthly=12,yearly=1
        --notification-policy NOTIFICATION_POLICY
                              Notification policy. Available options: never, always, on-failure. Default: always
        --concurrency CONCURRENCY
                              Concurrency. Number of concurrent hosts to backup. Default: 1

    configtest         Test configuration file

    stats              Show some stats
      options:
        --json      Output the stats in JSON format

Examples

Run a manual backup job

Backup all sources to storage1. No retention policy is applied – all versions are kept indefinitely:

usbackup run --dest storage1

Backup only specific sources:

usbackup run --dest storage1 --limit host1 --limit host2

Backup all sources except one:

usbackup run --dest storage1 --exclude host3

Backup with a custom retention policy (keep last 5, 7 daily, 4 weekly):

usbackup run --dest storage1 --retention-policy last=5,daily=7,weekly=4

Run a manual replication job

Replicate the latest backup from storage1 to storage2 using incremental mode (default):

usbackup run --type replication --replicate storage1 --dest storage2

Replicate using archive mode (packages the latest backup version into an archive.tar.gz per version):

usbackup run --type replication --replicate storage1 --dest storage2 --replication-mode archive

Replicate only specific sources with a retention policy:

usbackup run --type replication --replicate storage1 --dest storage2 --limit host1 --retention-policy last=3,daily=7

Note: The destination storage for replication jobs can be a remote host over SSH, just like any other storage. Configure it with a host in the storages section of the config file.

Configuration file

For a sample configuration file see config.sample.yml file. Aditionally, you can copy the file to /etc/usbackup/config.yml, /etc/opt/usbackup/config.yml or ~/.config/usbackup/config.yml (or where you want as long as you provide the --config parameter) and adjust the values to your needs.

A secrets.env file can also be created in the same directory as the config file to store sensitive information such as passwords and API keys. This file should be in the format of key=value pairs, one per line and referenced in the config file using the ${VAR_NAME} syntax.

For details on how to configure the file, see the config.sample.yml file.

The main sections of the configuration file are:

sources Sources are the representation of each host that needs to be backed up and contains the instructions of what to back up on that host.
storages Storages are the representation of the different backup destinations.
jobs Jobs are the glue that binds sources and storages together and defines when to run the backup and how many backups to keep.
notifiers Notifiers are the different methods of sending notifications after the backup is finished.

Valid format for hosts:

<user>:<password>@<host>:<port>

With all the fields except host being optional. If no user is specified, the root user will be used. If no port is specified, the default port will be used for that service. If no password is specified, it will try to use ssh keys (recomended way).

Note! Using passwords is not recommended as they will be stored as plain text in the configuration file, instead use ssh keys for file transfers / configs.

Systemd service

To run UsBackup as a service, have it start on boot and restart on failure, create a systemd service file in /etc/systemd/system/usbackup.service and copy the content from usbackup.sample.service file, adjusting the ExecStart parameter based on the installation method.

After that, run the following commands:

systemctl daemon-reload
systemctl enable usbackup.service
systemctl start usbackup.service

Disclaimer

This software is provided as is, without any warranty. Use at your own risk. The author is not responsible for any damage caused by this software.

License

This software is licensed under the GNU GPL v3 license. See the LICENSE file for more information.