SAB - Stand Alone Backup for lab clients

SAB is intended for backing up lab clients at NTNU. Key features is:

Central storage
Disk oriented with revisions
Stand alone Linux in memory backup and storage process
Compression based on blanking unused blocks, blockwise diff and gzip
Based on open formats and processes
Keep it simple and maintainable

Process overview

The lifecycle of a backup set is generally like this

Backup - Copy a disk from the client to the server
Curatation - Storing and process data and metadata
Restore - Copy an stored image to a client disk
Discard - Removal when not longer needed

Backup and restore

Backup and restore is done in the primary user interface. It is text based in the stand alone Linux in memory system. The design goal for the user interface is to reduce the workload of the operator to a minimum for normal tasks.

Quick and responsive interface
As little manual typing as possible
Drop and forget, no need to "follow up" the ongoing process

The minimalistic Linux i memory system put some constrains on the user interface, and prominent limitations is:

prompt based
no national keyboard support

Some configurations struggle booting the Linux in memory system. Alternatively you may run from Ubuntu live or similar.

Curatation

The curation is done in background by offline processes on the server side

Checkin

A checkin process is polling the incoming "reception" folder frequently. It does a few basic checks on image completeness before storing it in its permanent storage location. Incomplete images is put aside for inspection [TBD: and the user is notified].

Checkin is done frequently [every 60 seconds] to facillitate immediate restore for transferring data to a replacement disk.

Free block zeroing

This process try to zero unused blocks using the linux tools ntfsclone [NTFS] and zerofree [EXT]. This is done on each partition separatly, and the old and new file system is mounted ro and compared. If any problem arise, the original partitions data is used. Free blocks not belonging to any partition or disk label area is zeroed.

This process is not yet considered completely failsafe, and a diff file is kept so the original image may be rebuild.

Disks without a recognized partition table is used unaltered, and the unzero diff is a small noop.

Delta images

An older image is stored as a block diff of a newer version. This way only changed blocks is stored, but a sequence of diffs may have to be applied to rebuild an old image. This way

the most recent image is allways readily available and not dependant on any other.
the oldest image may allways be discarded without breaking any dependencys.

The application of the diffs is done transparent to the user, but may affect restore time.

[TBD: Discarding]

Care should be taken when discarding to avoid broken dependencys. Before discarding an image, any dependensies should be resolved by rebuilding all dependent images, and optionally be diffed to another mor recent version.

Storage

The images is stored grouped together with other versions of the same disk, based on the partition table identifier. The metadata is stored in a YAML. The metedata is stored in a SQl database as well for quick search and lookup.

Access control

Storing images require ssh access to the storage server, as well as write access to the incoming "reception" folder.

Restore access require ssh access to the storage server, and read access to the storage folders. [TBD: Read access is not required if the database assosiate the logged in user with the image in other ways.] The storage tree may be read by SMB by users with the appropiate access.

User interface

The primary user interface is running in the stand alone Linux in memory system. Prerequesite for this is a USB [TBD: or PXE] boot medium with the system. The target system must also be configured to boot of the medium.

The USB stick may preferably be removed as soon the system is booted.

After booting the stand alone system, the user interface will present a few options, a short menu and prompt for required information. The first promt accept some special values for the options or menu. Anything else will start the default Backup process.

There is no "back" in the user interface. If you entered anything wrong, press ctrl-c to restart the process.

Backup

The backup process consist of four prompts:

System cabinet label(s):

Here you enter any system identification label. For NTNU PC's this is typically like "NTNU12345", "KB-1234" "HIST1234". For other computers some other name may be used to identify.

Select one of the above local disks:

Above this prompt is the output of fdisk -l. The answer will be matched against the highlighted "Disk:" lines, and the prompt will reapear until you have uniquelly identified one disk.

If there is only one disk in the system, this will automaticly be selected.

Username:

Enter your NTNU username. This will be used to log into the backup server.

When username is entered, ssh is started. This will give a warning about adding the host.

During the process the system is listening for network port announcements. If none has been been recieved so far you will be asked to wait a little more or proceed. Hit enter to proceed.

Password:

This prompt is presented by ssh. The user interface do not store passwords anywhere, so you'll be asked about it every time.

What happens next...

On sucessful login the server responds with a unique session id. This may later be recognized as version id of the image.

The user interface then start uploading the image along with metadata, both collected and typed in. A progress indication is sent from the server at intervalls.

When the upload is finished a checksum and a small menue is presented. The user interface may then run a checksum of the local disk and/or reboot, depending on options set.

Default is a reboot after 60s, so normally you may just leave the process to it self as soon the session id is presented. [TBD: If anything fails after that, you'll be notified by mail.]

Restore

Type "!" on the "System cabinet label(s):" prompt to start restore. You'll then be asked to identify the image to restore and the disk to OVERWRITE.

Add tags []:

Here you type in tags to search for one at a time. This does a database search for backed up disks. Any disk matching all entered tags is listed. The tags may include username, cabinet label etc. You may use "%" as wildcard, so "%1234" may match both "NTNU01234" and "KB-1234". You may enter "<" to empty the list. When only one disk is matced it proceed with some data about the matched disk and a list of all available versions.

Select version:

Type in the index number or the full version id. The list is chronological, so 0 is always the newest version.

Select one of the above local disks:

This is similar to the backup phase, but now you select the disk to OVERWRITE.

When a disk is selected, information about both the image and target disk is presented. Please take time to verify that is in line with your intentions!

Username and password

The username and password prompt is presented similar to during backup.

If you request a image that is not yet "zeroed", the server will inform you about it. This should mainly happen when transferring data to a new disk. Press enter to proceed with the original image.

What happens next...

When you are successfully logged in the server will emit the selected image wich the user interface writes directly to the selected disk. There will be some progress messages at intervalls. At the end the server will emit the checksum of the transmitted image and the original checksum for comparison. [TBD: You should have the same options at the end as during backup.]

Implemetation

User interface

systembackup.sh

Implement the above described user interface and is run on the client, normally by the Linux in memory system. The script interacts with the server/storage side of the SAB system by two server scripts over ssh.

Up and download scripts

Started on the server over SSH with standard SSH password query.

incoming.pl

Recieves the image prepended with a metadata header.

The header comprise the metadata enclosed in two identical separator lines. The separator should not appear in the metdata. The current client use the sha256 checksum of the metata for separator. The header is thus formated like this:

sha256 of metadata
metadata
sha256 of metadata

The image is then transferred binary and the conection closed at the end.

A session id is printed to STDOUT before STDIN is read. This id is used in filenames in the incoming folder, and later as verion numbers in image groups. Diagnostics is printed to STDOUT, including the sha256 sum of the recieved image.

The metadata is formatted with header lines in the format of "=== key" followed by the data assosiated with the key. The data is either gatered with promps to the user, or the output of different commands on the client like fdisk.

The following files is written to the incoming folder.

* _sid_.err - Copy of any diagnostic messages.
* _sid_.head - the recieved header including the separator lined
* _sid_.img - the recieved image
* _sid_.sha256 - the sha256 sum of the recieved image

outgoing.pl

Send the requested image on stdout.

A prompt based dialogue over STDERR and STDIN is used to select the wanted image version. The image is transferred binary over STDOUT. On error no data will be sent, ensuring no data wil be overwritten on the clients target device.

An optional parameter with info about the client target device may be supplied for manual verificatione before starting the transfer. This typically contain output of fdisk for the target device.

Curation

import.pl

Import backupsets into storage and database.

Metadata Database

Installation

The instalation has to ensure:

linmemboot.sh and StandAloneBackup.sh from SAB is available in the linmem StandAloneBackup environment with all neccesary support files.
incoming.pl and outgoing.pl is found where StandAloneBackup.sh expect it to be
import.pl, zero.pl and delta.pl is run regularely from cron.
.pm files from SAB/lib is available to the above .pl scripts
A file store with an "incoming" and a "storage" folder:
- import.pl, zero.pl and delta.pl and have read and write access to the file store
- incoming.pl has write permission to the incoming folder
- outgoing.pl have read(files)/execute(directorys) access to the store tree
The metadata database tables exists:
- import.pl, zero.pl and delta.pl have read and write access
- outgoing.pl has read access

NTNU

Server role fagit-sab-server (nvfaglig09)

Ensure the following is present under /fagit/fagit-images/images/SAB/

incoming/
storage/

Ensure the following is present

/local/git/
- sab - clone of ssh://git@git.it.ntnu.no:7999/fagit/sab.git
/usr/local/lib/site_perl/
- BlockDiff.pm -> /local/git/sab/BlockDiff.pm
- DBItransaction.pm -> /local/git/sab/DBItransaction.pm
- ImageHeader.pm -> /local/git/sab/ImageHeader.pm
- ImageHeaderSection -> /local/git/sab/ImageHeaderSection
- ImageHeaderSection.pm -> /local/git/sab/ImageHeaderSection.pm
- SABDB.pm -> /local/git/sab/SABDB.pm
- SABset.pm -> /local/git/sab/SABset.pm
- Schema.pm -> /local/git/sab/Schema.pm
- SQLStruct.pm -> /local/git/sab/SQLStruct.pm
/local/bin/
- incoming.pl -> /local/git/sab/incoming.pl
- outgoing.pl -> /local/git/sab/outgoing.pl
/local/admin/bin/
- import.pl -> /local/git/sab/import.pl
- zero.pl -> /local/git/sab/zero.pl
- delta.pl -> /local/git/sab/delta.pl

Ensure the following is run in cron as root

/local/admin/bin/import.pl - run every minute
/local/admin/bin/zero.pl - run every minute
/local/admin/bin/delta.pl - run once every night

Client role fagit-sab-klient (vaktmester) (TBD)

The client script StandAloneBackup.sh should be copied to https://www.it.ntnu.no/linmem/src/i386/StandAloneBackup/ along with its linmemboot.sh.

The rest https://www.it.ntnu.no/linmem/src/i386/StandAloneBackup/ is files required to run the client script in the linmem environment.

Block Diff

A simple block diff format is developed for this purpose.

A diff of two images A and B consist of a sequence of 512 bytes block groups. The first block in each group is a bitmap representing the corresponding 4096 blocks of the sources. Each bit tell if the corresponding 512 byte blocks of the sources is equal. The rest of the group contain the data blocks of B for any zero bits (B differ from A). End of file B is marked with a zero bit with no corresponding data block.

See https://git.it.ntnu.no/projects/FAGIT/repos/blockdiff/ for implemetation details and distribution of BlockDiff.pm

Linux in memory

The linux in memory platform is consist of a kernel and rdinit image pair delivered by a disk image with syslinux [TBD: or PXE]. This is based on the standard Ubuntu live distribution, but the boot sequence is intercepted to

Load and extract tar set from net.
Run a shell script delivered by the tar set.

The user interface is run from this script. The rest of the tar set is utillitys etc the user interface need.

See https://www.it.ntnu.no/linmem/ for implementation details

Using SAB with Ubuntu live

Download Ubuntu desktop iso from Ubuntu
Transfer to USB-stick with
- Windows: rufus
- Linux: dd bs=1M if=Ubuntu-... of=/dev/sdX
Boot USB stick
Right click desktop and select "Open in terminal"
sudo -i
wget https://www.it.ntnu.no/lab/SAB/StandAloneBackup.sh
chmod 755 StandAloneBackup.sh
./StandAloneBackup.sh
Proceed as described in SAB readme

forskit-rse/SAB

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages