Ramil Valitov requested to merge ram/ispconfig3:feature/backup-formats into stable-3.1 Mar 31, 2020

New compression formats and archive containers

For databases

zip (deflate)
gzip
bzip2
xz
zip (bzip2)
7z (LZMA)
7z (LZMA2)
7z (PPMd)
7z (BZip2)

For web files

zip (deflate)
tar (gzip)
tar (bzip2)
tar (xz)
RAR (fastest compression)
RAR (fast compression)
RAR (normal compression)
RAR (good compression)
RAR (best compression)
tar + 7z (LZMA)
tar + 7z (LZMA2)
tar + 7z (PPMd)
tar + 7z (BZip2)

Requirements

The following packages should be installed for support of all archive formats:

p7zip-full, required for 7z
tar, required for tar-based containers
bzip2, required for bzip2
xz-utils, reguired for xz
rar, requried for RAR
pigz, optional for faster gzip compression

If some packages are missing, then a notification will be displayed, example:

Notes

For backward compatibility with previous versions of ISPConfig: in "default" mode ISPConfig has the following behaviour: if backup mode is set as web user, then "zip" format is used for web files, otherwise backup mode is set as root user and "tar (gzip)" format is used; database is compressed in "gzip" format.
Only "tar" based formats and "rar" preserve file ownership and permissions of web files and guarantee correct restore.

Encryption

Encryption is available only for the following backup formats:

7z
RAR
zip (not secure!)

If any other format is used then encryption can't be applied and encryption settings are ignored. You can safely change the password at anytime: the "Restore" button will still work for old backups encrypted with a previous password. You don't need to type a password to restore backups using the "Restore" button. Password is required when you extract the archives manually. Please, don't forget your password, because it's impossible to restore it.

Encrypted files can be restored by ISPConfig, and can be opened in Linux and Windows. In Windows you need to install 7z (free) or WinRAR (has trial mode), both of them do support encrypted files and many archive containers.

Manual backups via ISPConfig interface

Two new buttons are available:

make backup of database
make backup of web files

The total number of backups is always counted, both for automatic and manual backups:

If this number is exceeded, then old backups will be erased automatically. You don't need to have "Backup interval" enabled in order to make manual backups. But you need to have "Number of backup copies" > 0, in order to be able to save at least 1 backup file. The "Number of backup copies" specifies a total number of backup files that you can have. For example, if you want to store a backup of a database and web files, then the "Number of backup copies" should be equal to 2 or greater.

Information about backup process is reported in a more fancy and eye-catching way than it used to be before:

All manual backups files have manual- prefix.

Manual backups via shell script

A new schell script is added to make manual backups.

Currently ISPConfig can make only automatic cron based backups and there's no way to manually trigger a backup process for desired website. There's a workaround to use a cron_debug.php:

php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php

However such approach will create backups for all websites and will not make backups for websites that are inactive or have backup option disabled.

The script has the following features:

make backup only of a single website specified by user
it can make a backup of database, web files or both
the backup works even for websites that are inactive or have a backup feature disabled
the backup script respects the maximum backup count option and removes previous old backups if required
the backups produced by the script become visible in the ISPConfig interface

Usage:
	php /usr/local/ispconfig/server/backup-now.php --id=<4> [--type=<all>]
Options:
	--id		id of the website to backup.
	--type		backup type: all, web or mysql. Default is all.

The id is obtained from the ISPConfig interface:

Screenshot example:

All backups performed by this script are considered to be done in "manual" mode.

List of backup files

The table of files shows more info about the files: compression methods, encryption, scheduler:

Final notes

The restore code tries to restore permissions even for zip format, and is more stable. It has more logging (error and debug), so the user can get more information if something goes wrong.

Related issues, etc.

Closes #3947 (closed) (manual backups)
Closes #3489 (closed) (encrypted backups)
Closes #5554 (closed) (log error when database backup fails)
Closes #2891 (closed) (add option to change backups archive type)
Fixes #4346 (closed) (backup dir not unmounted after download)

Special thanks to @florian030 for his valuable comments and sharing his approach.

New code for backups with lots of new features