Simple, configuration-driven backup software for servers and workstations

Ruamel 0.15.5 (released about 13 hours ago) not working correctly in conjunction with pykwalify. With a default config, when borgmatic starts

pykwalify.errors.RuleError: <RuleError: error code 4: Value for keyword 'map/mapping' is not a dict: Path: '/'>

Installing ruamel.yaml==0.15.4 seem to solve the issue.

source_directories_glob can be used to enable glob support (defaults to disabled). Let me know what needs to be changed (option names or other stuff) for this patch to get accepted.

Hi @witten ,

this PR adds the ability to capture and parse the output of borg itself. In my case, I wanted to export those metrics to a Prometheus Node exporter Textfile Collector, so I added that module.

However it should be easy to add different metrics modules now, easily.

Hope you like it and it will find its way into master :)

Cheers

Borg supports checking a subset of archives in a repo by providing a prefix --prefix PREFIX when doing check. This PR adds functionality to use the same prefix defined in retention config to be passed as this param to borg.

http://borgbackup.readthedocs.io/en/stable/usage/check.html

When the database name is set to 'all', borgmatic uses pg_dumpall instead of pg_dump to dump the database. pg_dumpall outputs a plain-text format and pg_restore does not accept that format. To run these plain-text outputs, psql is needed instead.

This pull request implements that. Thanks!

I used https://github.com/cdrx/docker-pyinstaller for some hints on how to do this.

It builds standalone binaries much like the borg project distributes. Creating a standalone build of borgmatic means it's possible use borg via borgmatic on a machine by simply copying two files onto it.

I've only created a build for Linux amd64; there's no reason other platforms couldn't be supported. The docker-pyinstaller repo I linked above will provide a bit of guidance to anyone looking to add support for other platforms.

Hi :wave:,

the following pull request adds the optional flag verify_ssl to the Healthchecks monitoring configuration. The variable defaults to True.

Use case: When running a self-hosted Healthchecks instance with self-signed certificates the ping command fails with the error:

/etc/borgmatic.d/config.yaml: Pinging Healthchecks finish
/etc/borgmatic.d/config.yaml: Healthchecks error: HTTPSConnectionPool(host='<local_network_dns_redacted>', port=443): Max retries exceeded with url: /ping/1bb6a6f7-679f-4b14-8f47-4480b542a28a (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

Let me know what you think. Hope you don't mind me creating the pull request here at Github rather than over at torsion.org.

With --json option and mysql dump database TypeError appear : Traceback (most recent call last): File "/opt/borgmatic/bin/borgmatic", line 8, in sys.exit(main()) File "/opt/borgmatic/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 920, in main summary_logs = parse_logs + list(collect_configuration_run_summary_logs(configs, arguments)) File "/opt/borgmatic/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 813, in collect_configuration_run_summary_logs results = list(run_configuration(config_filename, config, arguments)) File "/opt/borgmatic/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 114, in run_configuration yield from run_actions( File "/opt/borgmatic/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 350, in run_actions json_output = borg_create.create_archive( File "/opt/borgmatic/lib/python3.8/site-packages/borgmatic/borg/create.py", line 323, in create_archive return execute_command_with_processes( File "/opt/borgmatic/lib/python3.8/site-packages/borgmatic/execute.py", line 262, in execute_command_with_processes log_outputs( File "/opt/borgmatic/lib/python3.8/site-packages/borgmatic/execute.py", line 102, in log_outputs logger.log(output_log_level, line) File "/usr/lib/python3.8/logging/init.py", line 1508, in log raise TypeError("level must be an integer") TypeError: level must be an integer

This PR adds support for user-defined shell scripts or single commands to be executed before and after a backup, or if an error has occurred. The intended use case is just as described in two existing issues [1, 2].

Using the "before_backup", "after_backup", and "on_error" sections in the config file, it is possible to list one or multiple commands to execute. For example, this can be used to send emails after successful / failed backups, check disk space, execute mysqldump prior to creating a backup, etc.

For a better overview I've split the implementation into 3 commits:

1. af38fdea01a145a6c3437df0193823bb04d20e21 enables the value of the "example" fields in schema.yaml to be false by default, e.g. when used in conjunction with type: bool. This comes in handy to demonstrate the use of hooks in the generated default config. As the user has to customize the commands to execute anyway, it seems advisable to disable the hooks by default.

2. 55fd81332455270a94392adda3502fafd0b8b655 is the actual implementation for hooks. For now, I've moved exec_cmd() into the separate file hook.py. As it's only 4 lines it might as well fit into borgmatic.py but I guess that's a matter of taste :-)

3. 7685613d280817c5efd954c715d12f339fa2753f removes "example" fields recursively from schema. It might be a bit overkill just for user hooks but I think it's useful anyway, considering that future config versions could add further "map" sections regardless of their position in the hierarchy.

It may not be missed that shell support in borgmatic also has some security implications:

• In the worst case scenario, a config file specifies a shell script that is also writable by attackers. For example, lets assume the specified script resides in the document root of a vHost user and the provided web application is not exactly secure. After successful exploitation an attacker might place his own commands into the script and wait for the next cron run of borgmatic. In that case, an attacker's payload would be executed with the same permissions as borgmatic. If borgmatic is called with higher-level permissions, the consequences could be fatal.

• In another scenario, imagine config files having insecure file permissions. An attacker couldn't only read the encryption passphrase but might also be able to insert arbitrary shell commands. The implications would be the same as above.

In summary, I think user hooks would be a very useful extension to borgmatic if used with care. Anyone enabling them should understand the security implications.

[1] https://tree.taiga.io/project/witten-borgmatic/issue/16 [2] https://tree.taiga.io/project/witten-borgmatic/issue/38

add pg_dump_command, pg_restore_command and psql_command configuration options. In this way, users can specify alternative commands on a per-database basis.

This is convenient if one needs to do backup/restore of different versions of postgres. A concrete example is the use of different versions of postgres running in docker containers for which one would want to use commands inside these containers while still being able to use pipes. Example:

hooks:
  postgresql_databases:
    - name: my_pg_database
      pg_dump_command: docker exec my_pg_database pg_dump
      pg_restore_command: docker exec my_pg_database pg_restore
      psql_command: docker exec my_pg_database psql

It partially fixes (#311) since it's just for the postgres hook.

Hallo witten,

I have found a little error in your remove_database_dumps function. If you have multiple dumps of a database type, the function parameter dump_path is overwritten in foreach loop with dump_file. In the second loop, the database name is appended a second time to dump_path.

This pull request fixed the error in the cleanup routine for database dumps.

regards Raphael