stv0g/restic
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

Compare commits

base: stv0g:master
Branches Tags
stv0g:master
stv0g:support-go111
stv0g:forget-explain
stv0g:prune-aggressive
stv0g:add-webdav-server
stv0g:v0.9.1
stv0g:v0.9.0
stv0g:v0.8.3
stv0g:v0.8.2
stv0g:v0.8.1
stv0g:v0.8.0
stv0g:v0.7.3
stv0g:v0.7.2
stv0g:v0.7.1
stv0g:v0.7.0
stv0g:v0.6.1
stv0g:v0.6.0
stv0g:v0.6.0-rc.1
stv0g:v0.5.0
stv0g:v0.5.0-rc.1
stv0g:v0.4.0
stv0g:v0.3.3
stv0g:v0.3.2
stv0g:v0.3.1
stv0g:v0.3.0
stv0g:v0.2.0
stv0g:v0.1.0-doc
stv0g:v0.1.0
stv0g:v0.0.1alpha
..
compare: stv0g:prune-aggressive
Branches Tags
stv0g:master
stv0g:support-go111
stv0g:forget-explain
stv0g:prune-aggressive
stv0g:add-webdav-server
stv0g:v0.9.1
stv0g:v0.9.0
stv0g:v0.8.3
stv0g:v0.8.2
stv0g:v0.8.1
stv0g:v0.8.0
stv0g:v0.7.3
stv0g:v0.7.2
stv0g:v0.7.1
stv0g:v0.7.0
stv0g:v0.6.1
stv0g:v0.6.0
stv0g:v0.6.0-rc.1
stv0g:v0.5.0
stv0g:v0.5.0-rc.1
stv0g:v0.4.0
stv0g:v0.3.3
stv0g:v0.3.2
stv0g:v0.3.1
stv0g:v0.3.0
stv0g:v0.2.0
stv0g:v0.1.0-doc
stv0g:v0.1.0
stv0g:v0.0.1alpha

1 commit
master ... prune-aggr

Author SHA1 Message Date
Alexander Neumann
47db2ab8ef Make prune more aggressive 2018-03-05 20:41:11 +01:00
6653 changed files with 4339067 additions and 69712 deletions
Show stats Expand all files Collapse all files

79
.github/ISSUE_TEMPLATE.md vendored Normal file
View file

@ -0,0 +1,79 @@
<!--
Welcome! - We kindly ask that you:
1. Fill out the issue template below - not doing so needs a good reason.
2. Use the forum if you have a question rather than a bug or feature request.
The forum is at: https://forum.restic.net
NOTE: Not filling out the issue template needs a good reason, as otherwise it
may take a lot longer to find the problem, not to mention it can take up a lot
more time which can otherwise be spent on development. Please also take the
time to help us debug the issue by collecting relevant information, even if
it doesn't seem to be relevant to you. Thanks!
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
## Output of `restic version`
## How did you run restic exactly?
<!--
This section should include at least:
* The complete command line and any environment variables you used to
configure restic's backend access. Make sure to replace sensitive values!
* The output of the commands, what restic prints gives may give us much
information to diagnose the problem!
-->
## What backend/server/service did you use to store the repository?
## Expected behavior
<!--
Describe what you'd like restic to do differently.
-->
## Actual behavior
<!--
In this section, please try to concentrate on observations, so only describe
what you observed directly.
-->
## Steps to reproduce the behavior
<!--
The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
## Do you have any idea what may have caused this?
## Do you have an idea how to solve the issue?
## Did restic help you or made you happy in any way?
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!
Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
-->

93
.github/ISSUE_TEMPLATE/Bug.md vendored
View file

@ -1,93 +0,0 @@
---
name: Bug report
about: Report a problem with restic to help us resolve it and improve
---
<!--
Welcome! - We kindly ask that you:
1. Fill out the issue template below - not doing so needs a good reason.
2. Use the forum if you have a question rather than a bug or feature request.
The forum is at: https://forum.restic.net
NOTE: Not filling out the issue template needs a good reason, as otherwise it
may take a lot longer to find the problem, not to mention it can take up a lot
more time which can otherwise be spent on development. Please also take the
time to help us debug the issue by collecting relevant information, even if
it doesn't seem to be relevant to you. Thanks!
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
Output of `restic version`
--------------------------
How did you run restic exactly?
-------------------------------
<!--
This section should include at least:
* The complete command line and any environment variables you used to
configure restic's backend access. Make sure to replace sensitive values!
* The output of the commands, what restic prints gives may give us much
information to diagnose the problem!
-->
What backend/server/service did you use to store the repository?
----------------------------------------------------------------
Expected behavior
-----------------
<!--
Describe what you'd like restic to do differently.
-->
Actual behavior
---------------
<!--
In this section, please try to concentrate on observations, so only describe
what you observed directly.
-->
Steps to reproduce the behavior
-------------------------------
<!--
The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
Do you have any idea what may have caused this?
-----------------------------------------------
Do you have an idea how to solve the issue?
-------------------------------------------
Did restic help you or made you happy in any way?
-------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!
Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
-->

57
.github/ISSUE_TEMPLATE/Feature.md vendored
View file

@ -1,57 +0,0 @@
---
name: Feature request
about: Suggest a new feature or enhancement for restic
---
<!--
Welcome! - We kindly ask that you:
1. Fill out the issue template below - not doing so needs a good reason.
2. Use the forum if you have a question rather than a bug or feature request.
The forum is at: https://forum.restic.net
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
Output of `restic version`
--------------------------
<!--
Please add the version of restic you're currently using here, this helps us
later to see what has changed in restic when we revisit this issue after some
time.
-->
What should restic do differently? Which functionality do you think we should add?
----------------------------------------------------------------------------------
<!--
Please describe the feature you'd like us to add here.
-->
What are you trying to do?
--------------------------
<!--
This section should contain a brief description what you're trying to do, which
would be possible after implementing the new feature.
-->
Did restic help you or made you happy in any way?
-------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!
Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
-->

13
.github/PULL_REQUEST_TEMPLATE.md vendored
View file

@ -1,5 +1,3 @@
<!--
Thank you very much for contributing code or documentation to restic! Please
fill out the following questions to make it easier for us to review your
@ -10,27 +8,24 @@ your time and add more commits. If you're done and ready for review, please
check the last box.
-->
What is the purpose of this change? What does it change?
--------------------------------------------------------
### What is the purpose of this change? What does it change?
<!--
Describe the changes here, as detailed as needed.
-->
Was the change discussed in an issue or in the forum before?
------------------------------------------------------------
### Was the change discussed in an issue or in the forum before?
<!--
Link issues and relevant forum posts here.
-->
Checklist
---------
### Checklist
- [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
- [ ] I have added tests for all changes in this PR
- [ ] I have added documentation for the changes (in the manual)
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/TEMPLATE))
- [ ] There's a new file in a subdir of `changelog/` (named after an unreleased version) that describe the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/changelog-entry.tmpl))
- [ ] I have run `gofmt` on the code in all commits
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits)
- [ ] I'm done, this Pull Request is ready for review

33
.travis.yml
View file

@ -1,20 +1,33 @@
language: go
sudo: false
go:
- "1.8.x"
- "1.9.x"
- "1.10"
os:
- linux
- osx
env:
matrix:
RESTIC_TEST_FUSE=0
matrix:
exclude:
- os: osx
go: "1.8.x"
- os: osx
go: "1.9.x"
- os: linux
go: "1.10"
include:
- os: linux
go: "1.9.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0 RESTIC_BUILD_SOLARIS=0
# only run fuse and cloud backends tests on Travis for the latest Go on Linux
- os: linux
go: "1.10.x"
go: "1.10"
sudo: true
- os: osx
go: "1.10.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
env:
RESTIC_TEST_FUSE=1
branches:
only:

330
CHANGELOG.md
View file

@ -1,322 +1,3 @@
Changelog for restic 0.9.1 (2018-06-10)
=======================================
The following sections list the changes in restic 0.9.1 relevant to
restic users. The changes are ordered by importance.
Summary
-------
* Fix #1801: Add limiting bandwidth to the rclone backend
* Fix #1822: Allow uploading large files to MS Azure
* Fix #1825: Correct `find` to not skip snapshots
* Fix #1833: Fix caching files on error
* Fix #1834: Resolve deadlock
Details
-------
* Bugfix #1801: Add limiting bandwidth to the rclone backend
The rclone backend did not respect `--limit-upload` or `--limit-download`. Oftentimes it's
not necessary to use this, as the limiting in rclone itself should be used because it gives much
better results, but in case a remote instance of rclone is used (e.g. called via ssh), it is still
relevant to limit the bandwidth from restic to rclone.
https://github.com/restic/restic/issues/1801
* Bugfix #1822: Allow uploading large files to MS Azure
Sometimes, restic creates files to be uploaded to the repository which are quite large, e.g.
when saving directories with many entries or very large files. The MS Azure API does not allow
uploading files larger that 256MiB directly, rather restic needs to upload them in blocks of
100MiB. This is now implemented.
https://github.com/restic/restic/issues/1822
* Bugfix #1825: Correct `find` to not skip snapshots
Under certain circumstances, the `find` command was found to skip snapshots containing
directories with files to look for when the directories haven't been modified at all, and were
already printed as part of a different snapshot. This is now corrected.
In addition, we've switched to our own matching/pattern implementation, so now things like
`restic find "/home/user/foo/**/main.go"` are possible.
https://github.com/restic/restic/issues/1825
https://github.com/restic/restic/issues/1823
* Bugfix #1833: Fix caching files on error
During `check` it may happen that different threads access the same file in the backend, which
is then downloaded into the cache only once. When that fails, only the thread which is
responsible for downloading the file signals the correct error. The other threads just assume
that the file has been downloaded successfully and then get an error when they try to access the
cached file.
https://github.com/restic/restic/issues/1833
* Bugfix #1834: Resolve deadlock
When the "scanning" process restic runs to find out how much data there is does not finish before
the backup itself is done, restic stops doing anything. This is resolved now.
https://github.com/restic/restic/issues/1834
https://github.com/restic/restic/pull/1835
Changelog for restic 0.9.0 (2018-05-21)
=======================================
The following sections list the changes in restic 0.9.0 relevant to
restic users. The changes are ordered by importance.
Summary
-------
* Fix #1608: Respect time stamp for new backup when reading from stdin
* Fix #1652: Ignore/remove invalid lock files
* Fix #1730: Ignore sockets for restore
* Fix #1684: Fix backend tests for rest-server
* Fix #1745: Correctly parse the argument to --tls-client-cert
* Enh #1433: Support UTF-16 encoding and process Byte Order Mark
* Enh #1561: Allow using rclone to access other services
* Enh #1665: Improve cache handling for `restic check`
* Enh #1721: Add `cache` command to list cache dirs
* Enh #1758: Allow saving OneDrive folders in Windows
* Enh #549: Rework archiver code
* Enh #1552: Use Google Application Default credentials
* Enh #1477: Accept AWS_SESSION_TOKEN for the s3 backend
* Enh #1648: Ignore AWS permission denied error when creating a repository
* Enh #1649: Add illumos/Solaris support
* Enh #1709: Improve messages `restic check` prints
* Enh #827: Add --new-password-file flag for non-interactive password changes
* Enh #1735: Allow keeping a time range of snaphots
* Enh #1782: Use default AWS credentials chain for S3 backend
Details
-------
* Bugfix #1608: Respect time stamp for new backup when reading from stdin
When reading backups from stdin (via `restic backup --stdin`), restic now uses the time stamp
for the new backup passed in `--time`.
https://github.com/restic/restic/issues/1608
https://github.com/restic/restic/pull/1703
* Bugfix #1652: Ignore/remove invalid lock files
This corrects a bug introduced recently: When an invalid lock file in the repo is encountered
(e.g. if the file is empty), the code used to ignore that, but now returns the error. Now, invalid
files are ignored for the normal lock check, and removed when `restic unlock --remove-all` is
run.
https://github.com/restic/restic/issues/1652
https://github.com/restic/restic/pull/1653
* Bugfix #1730: Ignore sockets for restore
We've received a report and correct the behavior in which the restore code aborted restoring a
directory when a socket was encountered. Unix domain socket files cannot be restored (they are
created on the fly once a process starts listening). The error handling was corrected, and in
addition we're now ignoring sockets during restore.
https://github.com/restic/restic/issues/1730
https://github.com/restic/restic/pull/1731
* Bugfix #1684: Fix backend tests for rest-server
The REST server for restic now requires an explicit parameter (`--no-auth`) if no
authentication should be allowed. This is fixed in the tests.
https://github.com/restic/restic/pull/1684
* Bugfix #1745: Correctly parse the argument to --tls-client-cert
Previously, the --tls-client-cert method attempt to read ARGV[1] (hardcoded) instead of the
argument that was passed to it. This has been corrected.
https://github.com/restic/restic/issues/1745
https://github.com/restic/restic/pull/1746
* Enhancement #1433: Support UTF-16 encoding and process Byte Order Mark
On Windows, text editors commonly leave a Byte Order Mark at the beginning of the file to define
which encoding is used (oftentimes UTF-16). We've added code to support processing the BOMs in
text files, like the exclude files, the password file and the file passed via `--files-from`.
This does not apply to any file being saved in a backup, those are not touched and archived as they
are.
https://github.com/restic/restic/issues/1433
https://github.com/restic/restic/issues/1738
https://github.com/restic/restic/pull/1748
* Enhancement #1561: Allow using rclone to access other services
We've added the ability to use rclone to store backup data on all backends that it supports. This
was done in collaboration with Nick, the author of rclone. You can now use it to first configure a
service, then restic manages the rest (starting and stopping rclone). For details, please see
the manual.
https://github.com/restic/restic/issues/1561
https://github.com/restic/restic/pull/1657
https://rclone.org
* Enhancement #1665: Improve cache handling for `restic check`
For safety reasons, restic does not use a local metadata cache for the `restic check` command,
so that data is loaded from the repository and restic can check it's in good condition. When the
cache is disabled, restic will fetch each tiny blob needed for checking the integrity using a
separate backend request. For non-local backends, that will take a long time, and depending on
the backend (e.g. B2) may also be much more expensive.
This PR adds a few commits which will change the behavior as follows:
* When `restic check` is called without any additional parameters, it will build a new cache in a
temporary directory, which is removed at the end of the check. This way, we'll get readahead for
metadata files (so restic will fetch the whole file when the first blob from the file is
requested), but all data is freshly fetched from the storage backend. This is the default
behavior and will work for almost all users.
* When `restic check` is called with `--with-cache`, the default on-disc cache is used. This
behavior hasn't changed since the cache was introduced.
* When `--no-cache` is specified, restic falls back to the old behavior, and read all tiny blobs
in separate requests.
https://github.com/restic/restic/issues/1665
https://github.com/restic/restic/issues/1694
https://github.com/restic/restic/pull/1696
* Enhancement #1721: Add `cache` command to list cache dirs
The command `cache` was added, it allows listing restic's cache directoriers together with
the last usage. It also allows removing old cache dirs without having to access a repo, via
`restic cache --cleanup`
https://github.com/restic/restic/issues/1721
https://github.com/restic/restic/pull/1749
* Enhancement #1758: Allow saving OneDrive folders in Windows
Restic now contains a bugfix to two libraries, which allows saving OneDrive folders in
Windows. In order to use the newer versions of the libraries, the minimal version required to
compile restic is now Go 1.9.
https://github.com/restic/restic/issues/1758
https://github.com/restic/restic/pull/1765
* Enhancement #549: Rework archiver code
The core archiver code and the complementary code for the `backup` command was rewritten
completely. This resolves very annoying issues such as 549. The first backup with this release
of restic will likely result in all files being re-read locally, so it will take a lot longer. The
next backup after that will be fast again.
Basically, with the old code, restic took the last path component of each to-be-saved file or
directory as the top-level file/directory within the snapshot. This meant that when called as
`restic backup /home/user/foo`, the snapshot would contain the files in the directory
`/home/user/foo` as `/foo`.
This is not the case any more with the new archiver code. Now, restic works very similar to what
`tar` does: When restic is called with an absolute path to save, then it'll preserve the
directory structure within the snapshot. For the example above, the snapshot would contain
the files in the directory within `/home/user/foo` in the snapshot. For relative
directories, it only preserves the relative path components. So `restic backup user/foo`
will save the files as `/user/foo` in the snapshot.
While we were at it, the status display and notification system was completely rewritten. By
default, restic now shows which files are currently read (unless `--quiet` is specified) in a
multi-line status display.
The `backup` command also gained a new option: `--verbose`. It can be specified once (which
prints a bit more detail what restic is doing) or twice (which prints a line for each
file/directory restic encountered, together with some statistics).
Another issue that was resolved is the new code only reads two files at most. The old code would
read way too many files in parallel, thereby slowing down the backup process on spinning discs a
lot.
https://github.com/restic/restic/issues/549
https://github.com/restic/restic/issues/1286
https://github.com/restic/restic/issues/446
https://github.com/restic/restic/issues/1344
https://github.com/restic/restic/issues/1416
https://github.com/restic/restic/issues/1456
https://github.com/restic/restic/issues/1145
https://github.com/restic/restic/issues/1160
https://github.com/restic/restic/pull/1494
* Enhancement #1552: Use Google Application Default credentials
Google provide libraries to generate appropriate credentials with various fallback
sources. This change uses the library to generate our GCS client, which allows us to make use of
these extra methods.
This should be backward compatible with previous restic behaviour while adding the
additional capabilities to auth from Google's internal metadata endpoints. For users
running restic in GCP this can make authentication far easier than it was before.
https://github.com/restic/restic/pull/1552
https://developers.google.com/identity/protocols/application-default-credentials
* Enhancement #1477: Accept AWS_SESSION_TOKEN for the s3 backend
Before, it was not possible to use s3 backend with AWS temporary security credentials(with
AWS_SESSION_TOKEN). This change gives higher priority to credentials.EnvAWS credentials
provider.
https://github.com/restic/restic/issues/1477
https://github.com/restic/restic/pull/1479
https://github.com/restic/restic/pull/1647
* Enhancement #1648: Ignore AWS permission denied error when creating a repository
It's not possible to use s3 backend scoped to a subdirectory(with specific permissions).
Restic doesn't try to create repository in a subdirectory, when 'bucket exists' of parent
directory check fails due to permission issues.
https://github.com/restic/restic/pull/1648
* Enhancement #1649: Add illumos/Solaris support
https://github.com/restic/restic/pull/1649
* Enhancement #1709: Improve messages `restic check` prints
Some messages `restic check` prints are not really errors, so from now on restic does not treat
them as errors any more and exits cleanly.
https://github.com/restic/restic/pull/1709
https://forum.restic.net/t/what-is-the-standard-procedure-to-follow-if-a-backup-or-restore-is-interrupted/571/2
* Enhancement #827: Add --new-password-file flag for non-interactive password changes
This makes it possible to change a repository password without being prompted.
https://github.com/restic/restic/issues/827
https://github.com/restic/restic/pull/1720
https://forum.restic.net/t/changing-repo-password-without-prompt/591
* Enhancement #1735: Allow keeping a time range of snaphots
We've added the `--keep-within` option to the `forget` command. It instructs restic to keep
all snapshots within the given duration since the newest snapshot. For example, running
`restic forget --keep-within 5m7d` will keep all snapshots which have been made in the five
months and seven days since the latest snapshot.
https://github.com/restic/restic/pull/1735
* Enhancement #1782: Use default AWS credentials chain for S3 backend
Adds support for file credentials to the S3 backend (e.g. ~/.aws/credentials), and reorders
the credentials chain for the S3 backend to match AWS's standard, which is static credentials,
env vars, credentials file, and finally remote.
https://github.com/restic/restic/pull/1782
Changelog for restic 0.8.3 (2018-02-26)
=======================================
@ -353,7 +34,6 @@ Details
https://github.com/restic/restic/issues/1641
https://github.com/restic/restic/pull/1643
https://forum.restic.net/t/help-fixing-repo-no-such-file/485/3
* Bugfix #1638: Handle errors listing files in the backend
@ -368,7 +48,6 @@ Details
operation on the backend should that fail. It is now corrected.
https://github.com/restic/restic/pull/1638
https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484
* Enhancement #1497: Add --read-data-subset flag to check command
@ -476,7 +155,6 @@ Details
of data loss, just minor inconvenience for our users.
https://github.com/restic/restic/pull/1589
https://forum.restic.net/t/error-loading-tree-check-prune-and-forget-gives-error-b2-backend/406
* Bugfix #1594: Google Cloud Storage: Use generic HTTP transport
@ -1015,7 +693,7 @@ Details
* Enhancement #1203: Print stats on all BSD systems when SIGINFO (ctrl+t) is received
https://github.com/restic/restic/pull/1203
https://github.com/restic/restic/pull/1082#issuecomment-326279920
https://github.com/restic/restic/pull/1082
* Enhancement #1205: Allow specifying time/date for a backup with `--time`
@ -1055,12 +733,12 @@ Details
* Enhancement #1055: Create subdirs below `data/` for local/sftp backends
The local and sftp backends now create the subdirs below `data/` on open/init. This way, restic
makes sure that they always exist. This is connected to an issue for the sftp server.
makes sure that they always exist. This is connected to an issue for the sftp server:
https://github.com/restic/restic/issues/1055
https://github.com/restic/rest-server/pull/11#issuecomment-309879710
https://github.com/restic/restic/pull/1077
https://github.com/restic/restic/pull/1105
https://github.com/restic/rest-server/pull/11#issuecomment-309879710
* Enhancement #1067: Allow loading credentials for s3 from IAM
@ -1072,7 +750,7 @@ Details
* Enhancement #1073: Add `migrate` cmd to migrate from `s3legacy` to `default` layout
The `migrate` command for changing the `s3legacy` layout to the `default` layout for s3
The `migrate` command for chaning the `s3legacy` layout to the `default` layout for s3
backends has been improved: It can now be restarted with `restic migrate --force s3_layout`
and automatically retries operations on error.

2
CONTRIBUTING.md
View file

@ -164,7 +164,7 @@ history and triaging bugs much easier.
Git commit messages have a very terse summary in the first line of the commit
message, followed by an empty line, followed by a more verbose description or a
List of changed things. For examples, please refer to the excellent [How to
Write a Git Commit Message](https://chris.beams.io/posts/git-commit/).
Write a Git Commit Message](http://chris.beams.io/posts/git-commit/).
If you change/add multiple different things that aren't related at all, try to
make several smaller commits. This is much easier to review. Using `git add -p`

27
GOVERNANCE.md
View file

@ -1,27 +0,0 @@
# restic project governance
## Overview
The restic project uses a governance model commonly described as Benevolent
Dictator For Life (BDFL). This document outlines our understanding of what this
means. It is derived from the [i3 window manager project
governance](https://raw.githubusercontent.com/i3/i3/next/.github/GOVERNANCE.md).
## Roles
* user: anyone who interacts with the restic project
* core contributor: a handful of people who have contributed significantly to
the project by any means (issue triage, support, documentation, code, etc.).
Core contributors are recognizable via GitHub’s "Member" badge.
* Benevolent Dictator For Life (BDFL): a single individual who makes decisions
when consensus cannot be reached. restic's current BDFL is [@fd0](https://github.com/fd0).
## Decision making process
In general, we try to reach consensus in discussions. In case consensus cannot
be reached, the BDFL makes a decision.
## Contribution process
The contribution process is described in a separate document called
[CONTRIBUTING](CONTRIBUTING.md).

424
Gopkg.lock generated
View file

@ -3,472 +3,234 @@
[[projects]]
branch = "master"
digest = "1:94e9caf404409a2990cfd22aca37d758494c098eff3e2c37fda1abed862e74dd"
name = "bazil.org/fuse"
packages = [
".",
"fs",
"fuseutil",
]
pruneopts = "UT"
revision = "65cc252bf6691cb3c7014bcb2c8dc29de91e3a7e"
packages = [".","fs","fuseutil"]
revision = "371fbbdaa8987b715bdd21d6adc4c9b20155f748"
[[projects]]
digest = "1:5c3894b2aa4d6bead0ceeea6831b305d62879c871780e7b76296ded1b004bc57"
name = "cloud.google.com/go"
packages = ["compute/metadata"]
pruneopts = "UT"
revision = "aad3f485ee528456e0768f20397b4d9dd941e755"
version = "v0.25.0"
revision = "767c40d6a2e058483c25fa193e963a22da17236d"
version = "v0.18.0"
[[projects]]
digest = "1:46ea9487304f4b3c787f54483ecb13a338d686dcd670db0ab1a112ed0ae2128e"
name = "github.com/Azure/azure-sdk-for-go"
packages = [
"storage",
"version",
]
pruneopts = "UT"
revision = "4e8cbbfb1aeab140cd0fa97fd16b64ee18c3ca6a"
version = "v19.1.0"
packages = ["storage"]
revision = "eae258195456be76b2ec9ad2ee2ab63cdda365d9"
version = "v12.2.0-beta"
[[projects]]
digest = "1:27d0cd1a78fc836f7c0f07749d029a5f7895c84ad066187b08b70e9d1830098e"
name = "github.com/Azure/go-autorest"
packages = [
"autorest",
"autorest/adal",
"autorest/azure",
"autorest/date",
"logger",
"version",
]
pruneopts = "UT"
revision = "dd94e014aaf16d1df746762e392aa201c1b4c461"
version = "v10.15.0"
packages = ["autorest","autorest/adal","autorest/azure","autorest/date"]
revision = "c2a68353555b68de3ee8455a4fd3e890a0ac6d99"
version = "v9.8.1"
[[projects]]
digest = "1:2209584c0f7c9b68c23374e659357ab546e1b70eec2761f03280f69a8fd23d77"
name = "github.com/cenkalti/backoff"
packages = ["."]
pruneopts = "UT"
revision = "2ea60e5f094469f9e65adb9cd103795b73ae743e"
version = "v2.0.0"
[[projects]]
digest = "1:7cb4fdca4c251b3ef8027c90ea35f70c7b661a593b9eeae34753c65499098bb1"
name = "github.com/cpuguy83/go-md2man"
packages = ["md2man"]
pruneopts = "UT"
revision = "20f5889cbdc3c73dbd2862796665e7c465ade7d1"
version = "v1.0.8"
[[projects]]
digest = "1:76dc72490af7174349349838f2fe118996381b31ea83243812a97e5a0fd5ed55"
name = "github.com/dgrijalva/jwt-go"
packages = ["."]
pruneopts = "UT"
revision = "06ea1031745cb8b3dab3f6a236daf2b0aa468b7e"
version = "v3.2.0"
[[projects]]
branch = "master"
digest = "1:6f9339c912bbdda81302633ad7e99a28dfa5a639c864061f1929510a9a64aa74"
name = "github.com/dustin/go-humanize"
packages = ["."]
pruneopts = "UT"
revision = "9f541cc9db5d55bce703bd99987c9d5cb8eea45e"
[[projects]]
digest = "1:c7edfbb6320d6a93240d663dc52bca92bed4c116abe54c35679eec4e7cc2bd77"
name = "github.com/elithrar/simple-scrypt"
packages = ["."]
pruneopts = "UT"
revision = "d150773194090feb6c897805a7bcea8d49544e2c"
version = "v1.3.0"
[[projects]]
digest = "1:fe8a03a8222d5b913f256972933d26d24ad7c8286692a42943bc01633cc8fce3"
name = "github.com/go-ini/ini"
packages = ["."]
pruneopts = "UT"
revision = "358ee7663966325963d4e8b2e1fbd570c5195153"
version = "v1.38.1"
[[projects]]
digest = "1:15042ad3498153684d09f393bbaec6b216c8eec6d61f63dff711de7d64ed8861"
name = "github.com/golang/protobuf"
packages = ["proto"]
pruneopts = "UT"
revision = "b4deda0973fb4c70b50d226b1af49f3da59f5265"
revision = "61153c768f31ee5f130071d08fc82b85208528de"
version = "v1.1.0"
[[projects]]
digest = "1:2e3c336fc7fde5c984d2841455a658a6d626450b1754a854b3b32e7a8f49a07a"
name = "github.com/google/go-cmp"
packages = [
"cmp",
"cmp/internal/diff",
"cmp/internal/function",
"cmp/internal/value",
]
pruneopts = "UT"
revision = "3af367b6b30c263d47e8895973edcca9a49cf029"
version = "v0.2.0"
name = "github.com/cpuguy83/go-md2man"
packages = ["md2man"]
revision = "1d903dcb749992f3741d744c0f8376b4bd7eb3e1"
version = "v1.0.7"
[[projects]]
name = "github.com/dgrijalva/jwt-go"
packages = ["."]
revision = "dbeaa9332f19a944acb5736b4456cfcc02140e29"
version = "v3.1.0"
[[projects]]
branch = "master"
name = "github.com/dustin/go-humanize"
packages = ["."]
revision = "bb3d318650d48840a39aa21a027c6630e198e626"
[[projects]]
name = "github.com/elithrar/simple-scrypt"
packages = ["."]
revision = "2325946f714c95de4a6088202c402fbdfa64163b"
version = "v1.2.0"
[[projects]]
name = "github.com/go-ini/ini"
packages = ["."]
revision = "32e4c1e6bc4e7d0d8451aa6b75200d19e37a536a"
version = "v1.32.0"
[[projects]]
branch = "master"
name = "github.com/golang/protobuf"
packages = ["proto"]
revision = "c65a0412e71e8b9b3bfd22925720d23c0f054237"
[[projects]]
digest = "1:870d441fe217b8e689d7949fef6e43efbc787e50f200cb1e70dbca9204a1d6be"
name = "github.com/inconshreveable/mousetrap"
packages = ["."]
pruneopts = "UT"
revision = "76626ae9c91c4f2a10f34cad8ce83ea42c93bb75"
version = "v1.0"
[[projects]]
digest = "1:190ff84d9b2ed6589088f178cba8edb4b8ecb334df4572421fb016be1ac20463"
name = "github.com/juju/ratelimit"
packages = ["."]
pruneopts = "UT"
revision = "59fac5042749a5afb9af70e813da1dd5474f0167"
version = "1.0.1"
[[projects]]
digest = "1:9cedee824c21326bd26950bd9e1ffe9dc4e7ca03dc8634d0e6f954ee6a383172"
branch = "master"
name = "github.com/kr/fs"
packages = ["."]
pruneopts = "UT"
revision = "1455def202f6e05b95cc7bfc7e8ae67ae5141eba"
version = "v0.1.0"
revision = "2788f0dbd16903de03cb8186e5c7d97b69ad387b"
[[projects]]
digest = "1:1faa76bd9bffce9c25eaca0597afb67bd05a21ae57fe4378154ce8855ef163d1"
name = "github.com/kurin/blazer"
packages = [
"b2",
"base",
"internal/b2assets",
"internal/b2types",
"internal/blog",
"x/window",
]
pruneopts = "UT"
revision = "caf65aa76491dc533bac68ad3243ce72fa4e0a0a"
version = "v0.5.1"
packages = ["b2","base","internal/b2types","internal/blog"]
revision = "cd0304efa98725679cf68422cefa328d3d96f2f4"
version = "v0.3.0"
[[projects]]
digest = "1:4e878df5f4e9fd625bf9c9aac77ef7cbfa4a74c01265505527c23470c0e40300"
name = "github.com/marstr/guid"
packages = ["."]
pruneopts = "UT"
revision = "8bd9a64bf37eb297b492a4101fb28e80ac0b290f"
version = "v1.1.0"
revision = "8bdf7d1a087ccc975cf37dd6507da50698fd19ca"
[[projects]]
digest = "1:d4d17353dbd05cb52a2a52b7fe1771883b682806f68db442b436294926bbfafb"
name = "github.com/mattn/go-isatty"
packages = ["."]
pruneopts = "UT"
revision = "0360b2af4f38e8d38c7fce2a9f4e702702d73a39"
version = "v0.0.3"
[[projects]]
digest = "1:95c73c666919be2843b955eafc83f58c136312b74f79c703152f4c4a95fd64dc"
name = "github.com/minio/minio-go"
packages = [
".",
"pkg/credentials",
"pkg/encrypt",
"pkg/s3signer",
"pkg/s3utils",
"pkg/set",
]
pruneopts = "UT"
revision = "70799fe8dae6ecfb6c7d7e9e048fce27f23a1992"
version = "v6.0.5"
packages = [".","pkg/credentials","pkg/encrypt","pkg/policy","pkg/s3signer","pkg/s3utils","pkg/set"]
revision = "14f1d472d115bac5ca4804094aa87484a72ced61"
version = "4.0.6"
[[projects]]
branch = "master"
digest = "1:8eb17c2ec4df79193ae65b621cd1c0c4697db3bc317fe6afdc76d7f2746abd05"
name = "github.com/mitchellh/go-homedir"
packages = ["."]
pruneopts = "UT"
revision = "3864e76763d94a6df2f9960b16a20a33da9f9a66"
revision = "b8bc1bf767474819792c23f32d8286a45736f1c6"
[[projects]]
digest = "1:928de5172dd3563964d1b88a4ee3775cf72e16f1efabb482ab6d0e0bab86ee69"
branch = "master"
name = "github.com/ncw/swift"
packages = ["."]
pruneopts = "UT"
revision = "b2a7479cf26fa841ff90dd932d0221cb5c50782d"
version = "v1.0.39"
revision = "ae9f0ea1605b9aa6434ed5c731ca35d83ba67c55"
[[projects]]
digest = "1:40e195917a951a8bf867cd05de2a46aaf1806c50cf92eebf4c16f78cd196f747"
name = "github.com/pkg/errors"
packages = ["."]
pruneopts = "UT"
revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
version = "v0.8.0"
[[projects]]
digest = "1:cfa0d7741863a0e1d30e0ccdd4b48a96a471cdb47892303de8b92c3713af3e77"
name = "github.com/pkg/profile"
packages = ["."]
pruneopts = "UT"
revision = "5b67d428864e92711fcbd2f8629456121a56d91f"
version = "v1.2.1"
[[projects]]
digest = "1:23ed92ba5d90a2dfe817f3895027ccef796e79c30be5125d48e17afdcc395d73"
name = "github.com/pkg/sftp"
packages = ["."]
pruneopts = "UT"
revision = "57673e38ea946592a59c26592b7e6fbda646975b"
version = "v1.8.0"
revision = "f6a9258a0f570c3a76681b897b6ded57cb0dfa88"
version = "1.2.0"
[[projects]]
digest = "1:0d67664e93e366f072ac9672feea29bfc63c9f90f005e9e8a0df1954153f5a14"
name = "github.com/pkg/xattr"
packages = ["."]
pruneopts = "UT"
revision = "ae385d07bb53f092fcc7daaf738d8513df084931"
version = "v0.3.1"
revision = "23c75e3f6c1d8b13b3dd905b011a7f38a06044b7"
version = "v0.2.1"
[[projects]]
digest = "1:13ecc4000f49cf0aa3ee56fffcc93119c8edffacfff08674c80d2757d8c33a83"
name = "github.com/restic/chunker"
packages = ["."]
pruneopts = "UT"
revision = "db83917be3b88cc307464b7d8a221c173e34a0db"
version = "v0.2.0"
[[projects]]
digest = "1:8bc629776d035c003c7814d4369521afe67fdb8efc4b5f66540d29343b98cf23"
name = "github.com/russross/blackfriday"
packages = ["."]
pruneopts = "UT"
revision = "55d61fa8aa702f59229e6cff85793c22e580eaf5"
version = "v1.5.1"
revision = "4048872b16cc0fc2c5fd9eacf0ed2c2fedaa0c8c"
version = "v1.5"
[[projects]]
digest = "1:274f67cb6fed9588ea2521ecdac05a6d62a8c51c074c1fccc6a49a40ba80e925"
name = "github.com/satori/go.uuid"
packages = ["."]
pruneopts = "UT"
revision = "f58768cc1a7a7e77a3bd49e98cdd21419399b6a3"
version = "v1.2.0"
[[projects]]
digest = "1:d867dfa6751c8d7a435821ad3b736310c2ed68945d05b50fb9d23aee0540c8cc"
name = "github.com/sirupsen/logrus"
packages = ["."]
pruneopts = "UT"
revision = "3e01752db0189b9157070a0e1668a620f9a85da2"
version = "v1.0.6"
revision = "d682213848ed68c0a260ca37d6dd5ace8423f5ba"
version = "v1.0.4"
[[projects]]
digest = "1:e01b05ba901239c783dfe56450bcde607fc858908529868259c9a8765dc176d0"
name = "github.com/spf13/cobra"
packages = [
".",
"doc",
]
pruneopts = "UT"
revision = "ef82de70bb3f60c65fb8eebacbb2d122ef517385"
version = "v0.0.3"
packages = [".","doc"]
revision = "7b2c5ac9fc04fc5efafb60700713d4fa609b777b"
version = "v0.0.1"
[[projects]]
digest = "1:9424f440bba8f7508b69414634aef3b2b3a877e522d8a4624692412805407bb7"
name = "github.com/spf13/pflag"
packages = ["."]
pruneopts = "UT"
revision = "583c0c0531f06d5278b7d917446061adc344b5cd"
version = "v1.0.1"
revision = "e57e3eeb33f795204c1ca35f56c44f83227c6e66"
version = "v1.0.0"
[[projects]]
branch = "master"
digest = "1:eefb1f49ec07e71206d4c9ea1a3e634cad331c2180733e4121b8ae39e8e92ecb"
name = "golang.org/x/crypto"
packages = [
"argon2",
"blake2b",
"curve25519",
"ed25519",
"ed25519/internal/edwards25519",
"internal/chacha20",
"internal/subtle",
"pbkdf2",
"poly1305",
"scrypt",
"ssh",
"ssh/terminal",
]
pruneopts = "UT"
revision = "c126467f60eb25f8f27e5a981f32a87e3965053f"
packages = ["curve25519","ed25519","ed25519/internal/edwards25519","internal/chacha20","pbkdf2","poly1305","scrypt","ssh","ssh/terminal"]
revision = "3d37316aaa6bd9929127ac9a527abf408178ea7b"
[[projects]]
branch = "master"
digest = "1:8356aa7bdcb10a210b814b64ff76d61de7c36ac4cb6263de3af5e3e2e546956d"
name = "golang.org/x/net"
packages = [
"context",
"context/ctxhttp",
"http/httpguts",
"http2",
"http2/hpack",
"idna",
]
pruneopts = "UT"
revision = "32f9bdbd7df18e8641d215e7ea68be88b971feb0"
packages = ["context","context/ctxhttp","idna","lex/httplex"]
revision = "5ccada7d0a7ba9aeb5d3aca8d3501b4c2a509fec"
[[projects]]
branch = "master"
digest = "1:bea0314c10bd362ab623af4880d853b5bad3b63d0ab9945c47e461b8d04203ed"
name = "golang.org/x/oauth2"
packages = [
".",
"google",
"internal",
"jws",
"jwt",
]
pruneopts = "UT"
revision = "3d292e4d0cdc3a0113e6d207bb137145ef1de42f"
packages = [".","google","internal","jws","jwt"]
revision = "b28fcf2b08a19742b43084fb40ab78ac6c3d8067"
[[projects]]
branch = "master"
digest = "1:39ebcc2b11457b703ae9ee2e8cca0f68df21969c6102cb3b705f76cca0ea0239"
name = "golang.org/x/sync"
packages = ["errgroup"]
pruneopts = "UT"
revision = "1d60e4601c6fd243af51cc01ddf169918a5407ca"
revision = "fd80eb99c8f653c847d294a001bdf2a3a6f768f5"
[[projects]]
branch = "master"
digest = "1:a220a85c72a6cb7339c412cb2b117019a7fd94007cdfffb3b5b1d058227a2bf8"
name = "golang.org/x/sys"
packages = [
"cpu",
"unix",
"windows",
]
pruneopts = "UT"
revision = "bd9dbc187b6e1dacfdd2722a87e83093c2d7bd6e"
[[projects]]
digest = "1:e5a8511f063c38c51ab9ab80e718e9149f692652aeb4e393a8c020dd1bf38ca2"
name = "golang.org/x/text"
packages = [
"collate",
"collate/build",
"encoding",
"encoding/internal",
"encoding/internal/identifier",
"encoding/unicode",
"internal/colltab",
"internal/gen",
"internal/tag",
"internal/triegen",
"internal/ucd",
"internal/utf8internal",
"language",
"runes",
"secure/bidirule",
"transform",
"unicode/bidi",
"unicode/cldr",
"unicode/norm",
"unicode/rangetable",
]
pruneopts = "UT"
revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
version = "v0.3.0"
packages = ["unix","windows"]
revision = "af50095a40f9041b3b38960738837185c26e9419"
[[projects]]
branch = "master"
digest = "1:fb983acae7bd9c3ed9aadc1b1241d9e559ed21dbf84c17a0dda663ca169ccd69"
name = "google.golang.org/api"
packages = [
"gensupport",
"googleapi",
"googleapi/internal/uritemplates",
"storage/v1",
]
pruneopts = "UT"
revision = "31ca0e01cd791f07750cb23fc99327721f753290"
name = "golang.org/x/text"
packages = ["collate","collate/build","internal/colltab","internal/gen","internal/tag","internal/triegen","internal/ucd","language","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
revision = "e19ae1496984b1c655b8044a65c0300a3c878dd3"
[[projects]]
branch = "master"
name = "google.golang.org/api"
packages = ["gensupport","googleapi","googleapi/internal/uritemplates","storage/v1"]
revision = "65b0d8655182691ad23b4fac11e6f7b897d9b634"
[[projects]]
digest = "1:c8907869850adaa8bd7631887948d0684f3787d0912f1c01ab72581a6c34432e"
name = "google.golang.org/appengine"
packages = [
".",
"internal",
"internal/app_identity",
"internal/base",
"internal/datastore",
"internal/log",
"internal/modules",
"internal/remote_api",
"internal/urlfetch",
"urlfetch",
]
pruneopts = "UT"
revision = "b1f26356af11148e710935ed1ac8a7f5702c7612"
version = "v1.1.0"
packages = [".","internal","internal/app_identity","internal/base","internal/datastore","internal/log","internal/modules","internal/remote_api","internal/urlfetch","urlfetch"]
revision = "150dc57a1b433e64154302bdc40b6bb8aefa313a"
version = "v1.0.0"
[[projects]]
branch = "v2"
digest = "1:5bb148b78468350091db2ffbb2370f35cc6dcd74d9378a31b1c7b86ff7528f08"
name = "gopkg.in/tomb.v2"
packages = ["."]
pruneopts = "UT"
revision = "d5d1b5820637886def9eef33e03a27a9f166942c"
[[projects]]
digest = "1:342378ac4dcb378a5448dd723f0784ae519383532f5e70ade24132c4c8693202"
name = "gopkg.in/yaml.v2"
packages = ["."]
pruneopts = "UT"
revision = "5420a8b6744d3b0345ab293f6fcba19c978f1183"
version = "v2.2.1"
revision = "d670f9405373e636a5a2765eea47fac0c9bc91a4"
[solve-meta]
analyzer-name = "dep"
analyzer-version = 1
input-imports = [
"bazil.org/fuse",
"bazil.org/fuse/fs",
"github.com/Azure/azure-sdk-for-go/storage",
"github.com/cenkalti/backoff",
"github.com/elithrar/simple-scrypt",
"github.com/google/go-cmp/cmp",
"github.com/juju/ratelimit",
"github.com/kurin/blazer/b2",
"github.com/mattn/go-isatty",
"github.com/minio/minio-go",
"github.com/minio/minio-go/pkg/credentials",
"github.com/ncw/swift",
"github.com/pkg/errors",
"github.com/pkg/profile",
"github.com/pkg/sftp",
"github.com/pkg/xattr",
"github.com/restic/chunker",
"github.com/spf13/cobra",
"github.com/spf13/cobra/doc",
"github.com/spf13/pflag",
"golang.org/x/crypto/poly1305",
"golang.org/x/crypto/scrypt",
"golang.org/x/crypto/ssh/terminal",
"golang.org/x/net/context",
"golang.org/x/net/context/ctxhttp",
"golang.org/x/net/http2",
"golang.org/x/oauth2/google",
"golang.org/x/sync/errgroup",
"golang.org/x/sys/unix",
"golang.org/x/text/encoding/unicode",
"google.golang.org/api/googleapi",
"google.golang.org/api/storage/v1",
"gopkg.in/tomb.v2",
]
inputs-digest = "336ac5c261c174cac89f9a7102b493f08edfbd51fd61d1673d1d2ec4132d80ab"
solver-name = "gps-cdcl"
solver-version = 1

4
Gopkg.toml
View file

@ -19,7 +19,3 @@
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"
[prune]
unused-packages = true
go-tests = true

21
README.rst
View file

@ -1,4 +1,4 @@
|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage| |Reviewed by Hound|
|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage|
Introduction
------------
@ -29,7 +29,7 @@ and add some data:
.. code-block:: console
$ restic --repo /tmp/backup backup ~/work
$ restic -r /tmp/backup backup ~/work
enter password for repository:
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
@ -57,7 +57,6 @@ Therefore, restic supports the following backends for storing backups natively:
- `BackBlaze B2 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2>`__
- `Microsoft Azure Blob Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>`__
- `Google Cloud Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage>`__
- And many other services via the `rclone <https://rclone.org>`__ `Backend <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone>`__
Design Principles
-----------------
@ -111,18 +110,10 @@ License
Restic is licensed under `BSD 2-Clause License <https://opensource.org/licenses/BSD-2-Clause>`__. You can find the
complete text in ``LICENSE``.
Sponsorship
-----------
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by `AppsCode <https://appscode.com>`__!
|AppsCode|
.. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
:target: https://restic.readthedocs.io/en/latest/?badge=latest
.. |Build Status| image:: https://travis-ci.com/restic/restic.svg?branch=master
:target: https://travis-ci.com/restic/restic
.. |Build Status| image:: https://travis-ci.org/restic/restic.svg?branch=master
:target: https://travis-ci.org/restic/restic
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true
:target: https://ci.appveyor.com/project/fd0/restic/branch/master
.. |Report Card| image:: https://goreportcard.com/badge/github.com/restic/restic
@ -131,7 +122,3 @@ Storage are sponsored by `AppsCode <https://appscode.com>`__!
:target: https://saythanks.io/to/restic
.. |TestCoverage| image:: https://codecov.io/gh/restic/restic/branch/master/graph/badge.svg
:target: https://codecov.io/gh/restic/restic
.. |AppsCode| image:: https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png
:target: https://appscode.com
.. |Reviewed by Hound| image:: https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg
:target: https://houndci.com

2
VERSION
View file

@ -1 +1 @@
0.9.1
0.8.3

10
build.go
View file

@ -53,7 +53,7 @@ var config = Config{
"github.com/restic/restic/internal/...",
"github.com/restic/restic/cmd/...",
},
MinVersion: GoVersion{Major: 1, Minor: 9, Patch: 0}, // minimum Go version supported
MinVersion: GoVersion{Major: 1, Minor: 8, Patch: 0}, // minimum Go version supported
}
// Config configures the build.
@ -70,7 +70,6 @@ var (
keepGopath bool
runTests bool
enableCGO bool
enablePIE bool
)
// specialDir returns true if the file begins with a special character ('.' or '_').
@ -226,11 +225,9 @@ func showUsage(output io.Writer) {
fmt.Fprintf(output, " -T --test run tests\n")
fmt.Fprintf(output, " -o --output set output file name\n")
fmt.Fprintf(output, " --enable-cgo use CGO to link against libc\n")
fmt.Fprintf(output, " --enable-pie use PIE buildmode\n")
fmt.Fprintf(output, " --goos value set GOOS for cross-compilation\n")
fmt.Fprintf(output, " --goarch value set GOARCH for cross-compilation\n")
fmt.Fprintf(output, " --goarm value set GOARM for cross-compilation\n")
fmt.Fprintf(output, " --tempdir dir use a specific directory for compilation\n")
}
func verbosePrintf(message string, args ...interface{}) {
@ -268,9 +265,6 @@ func build(cwd string, ver GoVersion, goos, goarch, goarm, gopath string, args .
a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", gopath))
a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", gopath))
}
if enablePIE {
a = append(a, "-buildmode=pie")
}
a = append(a, args...)
cmd := exec.Command("go", a...)
@ -492,8 +486,6 @@ func main() {
runTests = true
case "--enable-cgo":
enableCGO = true
case "--enable-pie":
enablePIE = true
case "--goos":
skipNext = true
targetGOOS = params[i+1]

4
changelog/0.7.1_2017-07-22/issue-1055
View file

@ -2,9 +2,9 @@ Enhancement: Create subdirs below `data/` for local/sftp backends
The local and sftp backends now create the subdirs below `data/` on
open/init. This way, restic makes sure that they always exist. This is
connected to an issue for the sftp server.
connected to an issue for the sftp server:
https://github.com/restic/restic/issues/1055
https://github.com/restic/rest-server/pull/11#issuecomment-309879710
https://github.com/restic/restic/issues/1055
https://github.com/restic/restic/pull/1077
https://github.com/restic/restic/pull/1105

2
changelog/0.7.1_2017-07-22/issue-1073
View file

@ -1,6 +1,6 @@
Enhancement: Add `migrate` cmd to migrate from `s3legacy` to `default` layout
The `migrate` command for changing the `s3legacy` layout to the `default` layout
The `migrate` command for chaning the `s3legacy` layout to the `default` layout
for s3 backends has been improved: It can now be restarted with `restic migrate
--force s3_layout` and automatically retries operations on error.

0
changelog/0.7.1_2017-07-22/pull-1080 → changelog/0.7.1_2017-07-22/plul-1080
View file

0
changelog/0.9.0_2018-05-21/pull-1647 → changelog/0.8.4/pull-1647
View file

4
changelog/0.9.0_2018-05-21/pull-1648 → changelog/0.8.4/pull-1648
View file

@ -1,6 +1,6 @@
Enhancement: Ignore AWS permission denied error when creating a repository
Enhancement: Ignore AWS permission denied error when creating a repository.
It's not possible to use s3 backend scoped to a subdirectory(with specific permissions).
Restic doesn't try to create repository in a subdirectory, when 'bucket exists' of parent directory check fails due to permission issues.
https://github.com/restic/restic/pull/1648
https://github.com/restic/restic/pull/1648

3
changelog/0.8.4/pull-1649 Normal file
View file

@ -0,0 +1,3 @@
Enhancement: Add illumos/Solaris support.
https://github.com/restic/restic/pull/1649

12
changelog/0.9.0_2018-05-21/issue-1433
View file

@ -1,12 +0,0 @@
Enhancement: Support UTF-16 encoding and process Byte Order Mark
On Windows, text editors commonly leave a Byte Order Mark at the beginning of
the file to define which encoding is used (oftentimes UTF-16). We've added code
to support processing the BOMs in text files, like the exclude files, the
password file and the file passed via `--files-from`. This does not apply to
any file being saved in a backup, those are not touched and archived as they
are.
https://github.com/restic/restic/issues/1433
https://github.com/restic/restic/issues/1738
https://github.com/restic/restic/pull/1748

10
changelog/0.9.0_2018-05-21/issue-1561
View file

@ -1,10 +0,0 @@
Enhancement: Allow using rclone to access other services
We've added the ability to use rclone to store backup data on all backends that
it supports. This was done in collaboration with Nick, the author of rclone.
You can now use it to first configure a service, then restic manages the rest
(starting and stopping rclone). For details, please see the manual.
https://github.com/restic/restic/issues/1561
https://github.com/restic/restic/pull/1657
https://rclone.org

7
changelog/0.9.0_2018-05-21/issue-1608
View file

@ -1,7 +0,0 @@
Bugfix: Respect time stamp for new backup when reading from stdin
When reading backups from stdin (via `restic backup --stdin`), restic now uses
the time stamp for the new backup passed in `--time`.
https://github.com/restic/restic/issues/1608
https://github.com/restic/restic/pull/1703

9
changelog/0.9.0_2018-05-21/issue-1652
View file

@ -1,9 +0,0 @@
Bugfix: Ignore/remove invalid lock files
This corrects a bug introduced recently: When an invalid lock file in the repo
is encountered (e.g. if the file is empty), the code used to ignore that, but
now returns the error. Now, invalid files are ignored for the normal lock
check, and removed when `restic unlock --remove-all` is run.
https://github.com/restic/restic/issues/1652
https://github.com/restic/restic/pull/1653

27
changelog/0.9.0_2018-05-21/issue-1665
View file

@ -1,27 +0,0 @@
Enhancement: Improve cache handling for `restic check`
For safety reasons, restic does not use a local metadata cache for the `restic
check` command, so that data is loaded from the repository and restic can check
it's in good condition. When the cache is disabled, restic will fetch each tiny
blob needed for checking the integrity using a separate backend request. For
non-local backends, that will take a long time, and depending on the backend
(e.g. B2) may also be much more expensive.
This PR adds a few commits which will change the behavior as follows:
* When `restic check` is called without any additional parameters, it will
build a new cache in a temporary directory, which is removed at the end of
the check. This way, we'll get readahead for metadata files (so restic will
fetch the whole file when the first blob from the file is requested), but
all data is freshly fetched from the storage backend. This is the default
behavior and will work for almost all users.
* When `restic check` is called with `--with-cache`, the default on-disc cache
is used. This behavior hasn't changed since the cache was introduced.
* When `--no-cache` is specified, restic falls back to the old behavior, and
read all tiny blobs in separate requests.
https://github.com/restic/restic/issues/1665
https://github.com/restic/restic/issues/1694
https://github.com/restic/restic/pull/1696

8
changelog/0.9.0_2018-05-21/issue-1721
View file

@ -1,8 +0,0 @@
Enhancement: Add `cache` command to list cache dirs
The command `cache` was added, it allows listing restic's cache directoriers
together with the last usage. It also allows removing old cache dirs without
having to access a repo, via `restic cache --cleanup`
https://github.com/restic/restic/issues/1721
https://github.com/restic/restic/pull/1749

11
changelog/0.9.0_2018-05-21/issue-1730
View file

@ -1,11 +0,0 @@
Bugfix: Ignore sockets for restore
We've received a report and correct the behavior in which the restore code
aborted restoring a directory when a socket was encountered. Unix domain socket
files cannot be restored (they are created on the fly once a process starts
listening). The error handling was corrected, and in addition we're now
ignoring sockets during restore.
https://github.com/restic/restic/issues/1730
https://github.com/restic/restic/pull/1731

8
changelog/0.9.0_2018-05-21/issue-1758
View file

@ -1,8 +0,0 @@
Enhancement: Allow saving OneDrive folders in Windows
Restic now contains a bugfix to two libraries, which allows saving OneDrive
folders in Windows. In order to use the newer versions of the libraries, the
minimal version required to compile restic is now Go 1.9.
https://github.com/restic/restic/issues/1758
https://github.com/restic/restic/pull/1765

43
changelog/0.9.0_2018-05-21/issue-549
View file

@ -1,43 +0,0 @@
Enhancement: Rework archiver code
The core archiver code and the complementary code for the `backup` command was
rewritten completely. This resolves very annoying issues such as 549. The first
backup with this release of restic will likely result in all files being
re-read locally, so it will take a lot longer. The next backup after that will
be fast again.
Basically, with the old code, restic took the last path component of each
to-be-saved file or directory as the top-level file/directory within the
snapshot. This meant that when called as `restic backup /home/user/foo`, the
snapshot would contain the files in the directory `/home/user/foo` as `/foo`.
This is not the case any more with the new archiver code. Now, restic works
very similar to what `tar` does: When restic is called with an absolute path to
save, then it'll preserve the directory structure within the snapshot. For the
example above, the snapshot would contain the files in the directory within
`/home/user/foo` in the snapshot. For relative directories, it only preserves
the relative path components. So `restic backup user/foo` will save the files
as `/user/foo` in the snapshot.
While we were at it, the status display and notification system was completely
rewritten. By default, restic now shows which files are currently read (unless
`--quiet` is specified) in a multi-line status display.
The `backup` command also gained a new option: `--verbose`. It can be specified
once (which prints a bit more detail what restic is doing) or twice (which
prints a line for each file/directory restic encountered, together with some
statistics).
Another issue that was resolved is the new code only reads two files at most.
The old code would read way too many files in parallel, thereby slowing down
the backup process on spinning discs a lot.
https://github.com/restic/restic/issues/549
https://github.com/restic/restic/issues/1286
https://github.com/restic/restic/issues/446
https://github.com/restic/restic/issues/1344
https://github.com/restic/restic/issues/1416
https://github.com/restic/restic/issues/1456
https://github.com/restic/restic/issues/1145
https://github.com/restic/restic/issues/1160
https://github.com/restic/restic/pull/1494

13
changelog/0.9.0_2018-05-21/pull-1552
View file

@ -1,13 +0,0 @@
Enhancement: Use Google Application Default credentials
Google provide libraries to generate appropriate credentials with various
fallback sources. This change uses the library to generate our GCS client, which
allows us to make use of these extra methods.
This should be backward compatible with previous restic behaviour while adding
the additional capabilities to auth from Google's internal metadata endpoints.
For users running restic in GCP this can make authentication far easier than it
was before.
https://github.com/restic/restic/pull/1552
https://developers.google.com/identity/protocols/application-default-credentials

3
changelog/0.9.0_2018-05-21/pull-1649
View file

@ -1,3 +0,0 @@
Enhancement: Add illumos/Solaris support
https://github.com/restic/restic/pull/1649

6
changelog/0.9.0_2018-05-21/pull-1684
View file

@ -1,6 +0,0 @@
Bugfix: Fix backend tests for rest-server
The REST server for restic now requires an explicit parameter (`--no-auth`) if
no authentication should be allowed. This is fixed in the tests.
https://github.com/restic/restic/pull/1684

7
changelog/0.9.0_2018-05-21/pull-1709
View file

@ -1,7 +0,0 @@
Enhancement: Improve messages `restic check` prints
Some messages `restic check` prints are not really errors, so from now on
restic does not treat them as errors any more and exits cleanly.
https://github.com/restic/restic/pull/1709
https://forum.restic.net/t/what-is-the-standard-procedure-to-follow-if-a-backup-or-restore-is-interrupted/571/2

7
changelog/0.9.0_2018-05-21/pull-1720
View file

@ -1,7 +0,0 @@
Enhancement: Add --new-password-file flag for non-interactive password changes
This makes it possible to change a repository password without being prompted.
https://github.com/restic/restic/issues/827
https://github.com/restic/restic/pull/1720
https://forum.restic.net/t/changing-repo-password-without-prompt/591

9
changelog/0.9.0_2018-05-21/pull-1735
View file

@ -1,9 +0,0 @@
Enhancement: Allow keeping a time range of snaphots
We've added the `--keep-within` option to the `forget` command. It instructs
restic to keep all snapshots within the given duration since the newest
snapshot. For example, running `restic forget --keep-within 5m7d` will keep all
snapshots which have been made in the five months and seven days since the
latest snapshot.
https://github.com/restic/restic/pull/1735

7
changelog/0.9.0_2018-05-21/pull-1746
View file

@ -1,7 +0,0 @@
Bugfix: Correctly parse the argument to --tls-client-cert
Previously, the --tls-client-cert method attempt to read ARGV[1] (hardcoded)
instead of the argument that was passed to it. This has been corrected.
https://github.com/restic/restic/issues/1745
https://github.com/restic/restic/pull/1746

7
changelog/0.9.0_2018-05-21/pull-1782
View file

@ -1,7 +0,0 @@
Enhancement: Use default AWS credentials chain for S3 backend
Adds support for file credentials to the S3 backend (e.g. ~/.aws/credentials),
and reorders the credentials chain for the S3 backend to match AWS's standard,
which is static credentials, env vars, credentials file, and finally remote.
https://github.com/restic/restic/pull/1782

9
changelog/0.9.1_2018-06-10/issue-1801
View file

@ -1,9 +0,0 @@
Bugfix: Add limiting bandwidth to the rclone backend
The rclone backend did not respect `--limit-upload` or `--limit-download`.
Oftentimes it's not necessary to use this, as the limiting in rclone itself
should be used because it gives much better results, but in case a remote
instance of rclone is used (e.g. called via ssh), it is still relevant to limit
the bandwidth from restic to rclone.
https://github.com/restic/restic/issues/1801

9
changelog/0.9.1_2018-06-10/issue-1822
View file

@ -1,9 +0,0 @@
Bugfix: Allow uploading large files to MS Azure
Sometimes, restic creates files to be uploaded to the repository which are
quite large, e.g. when saving directories with many entries or very large
files. The MS Azure API does not allow uploading files larger that 256MiB
directly, rather restic needs to upload them in blocks of 100MiB. This is now
implemented.
https://github.com/restic/restic/issues/1822

12
changelog/0.9.1_2018-06-10/issue-1825
View file

@ -1,12 +0,0 @@
Bugfix: Correct `find` to not skip snapshots
Under certain circumstances, the `find` command was found to skip snapshots
containing directories with files to look for when the directories haven't been
modified at all, and were already printed as part of a different snapshot. This
is now corrected.
In addition, we've switched to our own matching/pattern implementation, so now
things like `restic find "/home/user/foo/**/main.go"` are possible.
https://github.com/restic/restic/issues/1825
https://github.com/restic/restic/issues/1823

9
changelog/0.9.1_2018-06-10/issue-1833
View file

@ -1,9 +0,0 @@
Bugfix: Fix caching files on error
During `check` it may happen that different threads access the same file in the
backend, which is then downloaded into the cache only once. When that fails,
only the thread which is responsible for downloading the file signals the
correct error. The other threads just assume that the file has been downloaded
successfully and then get an error when they try to access the cached file.
https://github.com/restic/restic/issues/1833

8
changelog/0.9.1_2018-06-10/issue-1834
View file

@ -1,8 +0,0 @@
Bugfix: Resolve deadlock
When the "scanning" process restic runs to find out how much data there is does
not finish before the backup itself is done, restic stops doing anything. This
is resolved now.
https://github.com/restic/restic/issues/1834
https://github.com/restic/restic/pull/1835

8
changelog/CHANGELOG.tmpl
View file

@ -18,11 +18,11 @@ Details
{{ range $par := .Paragraphs }}
{{ wrap $par 80 3 }}
{{ end -}}
{{ range $url := .IssueURLs }}
{{ $url -}}
{{ range $id := .Issues }}
https://github.com/restic/restic/issues/{{ $id -}}
{{ end -}}
{{ range $url := .PRURLs }}
{{ $url -}}
{{ range $id := .PRs }}
https://github.com/restic/restic/pull/{{ $id -}}
{{ end -}}
{{ range $url := .OtherURLs }}
{{ $url -}}

0
changelog/TEMPLATE → changelog/changelog-entry.tmpl
View file

16
changelog/unreleased/issue-1854
View file

@ -1,16 +0,0 @@
Bugfix: Allow saving files/dirs on different fs with `--one-file-system`
restic now allows saving files/dirs on a different file system in a subdir
correctly even when `--one-file-system` is specified.
The first thing the restic archiver code does is to build a tree of the target
files/directories. If it detects that a parent directory is already included
(e.g. `restic backup /foo /foo/bar/baz`), it'll ignore the latter argument.
Without `--one-file-system`, that's perfectly valid: If `/foo` is to be
archived, it will include `/foo/bar/baz`. But with `--one-file-system`,
`/foo/bar/baz` may reside on a different file system, so it won't be included
with `/foo`.
https://github.com/restic/restic/issues/1854
https://github.com/restic/restic/pull/1855

6
changelog/unreleased/issue-1870
View file

@ -1,6 +0,0 @@
Bugfix: Fix restore with --include
We fixed a bug which prevented restic to restore files with an include filter.
https://github.com/restic/restic/issues/1870
https://github.com/restic/restic/pull/1900

12
changelog/unreleased/issue-1880
View file

@ -1,12 +0,0 @@
Bugfix: Use `--cache-dir` argument for `check` command
`check` command now uses a temporary sub-directory of the specified directory
if set using the `--cache-dir` argument. If not set, the cache directory is
created in the default temporary directory as before.
In either case a temporary cache is used to ensure the actual repository is
checked (rather than a local copy).
The `--cache-dir` argument was not used by the `check` command, instead a
cache directory was created in the temporary directory.
https://github.com/restic/restic/issues/1880

8
changelog/unreleased/issue-1893
View file

@ -1,8 +0,0 @@
Bugfix: Return error when exclude file cannot be read
A bug was found: when multiple exclude files were passed to restic and one of
them could not be read, an error was printed and restic continued, ignoring
even the existing exclude files. Now, an error message is printed and restic
aborts when an exclude file cannot be read.
https://github.com/restic/restic/issues/1893

8
changelog/unreleased/issue-1906
View file

@ -1,8 +0,0 @@
Enhancement: Add support for B2 application keys
Restic can now use so-called "application keys" which can be created in the B2
dashboard and were only introduced recently. In contrast to the "master key",
such keys can be restricted to a specific bucket and/or path.
https://github.com/restic/restic/issues/1906
https://github.com/restic/restic/pull/1914

4
changelog/unreleased/pull-1729
View file

@ -1,4 +0,0 @@
Enhancement: Add stats command to get information about a repository
https://github.com/restic/restic/issues/874
https://github.com/restic/restic/pull/1729

6
changelog/unreleased/pull-1772
View file

@ -1,6 +0,0 @@
Enhancement: Add restore --verify to verify restored file content
Restore will print error message if restored file content does not match
expected SHA256 checksum
https://github.com/restic/restic/pull/1772

6
changelog/unreleased/pull-1853
View file

@ -1,6 +0,0 @@
Enhancement: Add JSON output support to `restic key list`
This PR enables users to get the output of `restic key list` in JSON in addition
to the existing table format.
https://github.com/restic/restic/pull/1853

6
changelog/unreleased/pull-1861
View file

@ -1,6 +0,0 @@
Bugfix: Fix case-insensitive search with restic find
We've fixed the behavior for `restic find -i PATTERN`, which was
broken in v0.9.1.
https://github.com/restic/restic/pull/1861

8
changelog/unreleased/pull-1882
View file

@ -1,8 +0,0 @@
Enhancement: S3 backend: accept AWS_SESSION_TOKEN
Before, it was not possible to use s3 backend with AWS temporary security credentials(with AWS_SESSION_TOKEN).
This change gives higher priority to credentials.EnvAWS credentials provider.
https://github.com/restic/restic/issues/1477
https://github.com/restic/restic/pull/1479
https://github.com/restic/restic/pull/1647

9
changelog/unreleased/pull-1901
View file

@ -1,9 +0,0 @@
Enhancement: Update the Backblaze B2 library
We've updated the library we're using for accessing the Backblaze B2 service to
0.5.0 to include support for upcoming so-called "application keys". With this
feature, you can create access credentials for B2 which are restricted to e.g.
a single bucket or even a sub-directory of a bucket.
https://github.com/restic/restic/pull/1901
https://github.com/kurin/blazer

9
cmd/restic/background.go Normal file
View file

@ -0,0 +1,9 @@
// +build !linux
package main
// IsProcessBackground should return true if it is running in the background or false if not
func IsProcessBackground() bool {
//TODO: Check if the process are running in the background in other OS than linux
return false
}

4
internal/ui/termstatus/background_linux.go → cmd/restic/background_linux.go
View file

@ -1,4 +1,4 @@
package termstatus
package main
import (
"syscall"
@ -7,7 +7,7 @@ import (
"github.com/restic/restic/internal/debug"
)
// IsProcessBackground reports whether the current process is running in the background.
// IsProcessBackground returns true if it is running in the background or false if not
func IsProcessBackground() bool {
var pid int
_, _, err := syscall.Syscall(syscall.SYS_IOCTL, uintptr(syscall.Stdin), syscall.TIOCGPGRP, uintptr(unsafe.Pointer(&pid)))

5
cmd/restic/cleanup.go
View file

@ -64,10 +64,7 @@ func CleanupHandler(c <-chan os.Signal) {
fmt.Fprintf(stderr, "%ssignal %v received, cleaning up\n", ClearLine(), s)
code := 0
if s == syscall.SIGINT {
code = 130
} else {
if s != syscall.SIGINT {
code = 1
}

640
cmd/restic/cmd_backup.go
View file

@ -2,26 +2,21 @@ package main
import (
"bufio"
"bytes"
"context"
"io/ioutil"
"fmt"
"io"
"os"
"strconv"
"path"
"path/filepath"
"strings"
"time"
"github.com/spf13/cobra"
tomb "gopkg.in/tomb.v2"
"github.com/restic/restic/internal/archiver"
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/fs"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/textfile"
"github.com/restic/restic/internal/ui"
"github.com/restic/restic/internal/ui/termstatus"
)
var cmdBackup = &cobra.Command{
@ -47,16 +42,11 @@ given as the arguments.
return errors.Fatal("cannot use both `--stdin` and `--files-from -`")
}
var t tomb.Tomb
term := termstatus.New(globalOptions.stdout, globalOptions.stderr, globalOptions.Quiet)
t.Go(func() error { term.Run(t.Context(globalOptions.ctx)); return nil })
err := runBackup(backupOptions, globalOptions, term, args)
if err != nil {
return err
if backupOptions.Stdin {
return readBackupFromStdin(backupOptions, globalOptions, args)
}
t.Kill(nil)
return t.Wait()
return runBackup(backupOptions, globalOptions, args)
},
}
@ -100,6 +90,127 @@ func init() {
f.BoolVar(&backupOptions.WithAtime, "with-atime", false, "store the atime for all files and directories")
}
func newScanProgress(gopts GlobalOptions) *restic.Progress {
if gopts.Quiet {
return nil
}
p := restic.NewProgress()
p.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
if IsProcessBackground() {
return
}
PrintProgress("[%s] %d directories, %d files, %s", formatDuration(d), s.Dirs, s.Files, formatBytes(s.Bytes))
}
p.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
PrintProgress("scanned %d directories, %d files in %s\n", s.Dirs, s.Files, formatDuration(d))
}
return p
}
func newArchiveProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
if gopts.Quiet {
return nil
}
archiveProgress := restic.NewProgress()
var bps, eta uint64
itemsTodo := todo.Files + todo.Dirs
archiveProgress.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
if IsProcessBackground() {
return
}
sec := uint64(d / time.Second)
if todo.Bytes > 0 && sec > 0 && ticker {
bps = s.Bytes / sec
if s.Bytes >= todo.Bytes {
eta = 0
} else if bps > 0 {
eta = (todo.Bytes - s.Bytes) / bps
}
}
itemsDone := s.Files + s.Dirs
status1 := fmt.Sprintf("[%s] %s %s / %s %d / %d items %d errors ",
formatDuration(d),
formatPercent(s.Bytes, todo.Bytes),
formatBytes(s.Bytes), formatBytes(todo.Bytes),
itemsDone, itemsTodo,
s.Errors)
status2 := fmt.Sprintf("ETA %s ", formatSeconds(eta))
if w := stdoutTerminalWidth(); w > 0 {
maxlen := w - len(status2) - 1
if maxlen < 4 {
status1 = ""
} else if len(status1) > maxlen {
status1 = status1[:maxlen-4]
status1 += "... "
}
}
PrintProgress("%s%s", status1, status2)
}
archiveProgress.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
fmt.Printf("\nduration: %s\n", formatDuration(d))
}
return archiveProgress
}
func newArchiveStdinProgress(gopts GlobalOptions) *restic.Progress {
if gopts.Quiet {
return nil
}
archiveProgress := restic.NewProgress()
var bps uint64
archiveProgress.OnUpdate = func(s restic.Stat, d time.Duration, ticker bool) {
if IsProcessBackground() {
return
}
sec := uint64(d / time.Second)
if s.Bytes > 0 && sec > 0 && ticker {
bps = s.Bytes / sec
}
status1 := fmt.Sprintf("[%s] %s %s/s", formatDuration(d),
formatBytes(s.Bytes),
formatBytes(bps))
if w := stdoutTerminalWidth(); w > 0 {
maxlen := w - len(status1)
if maxlen < 4 {
status1 = ""
} else if len(status1) > maxlen {
status1 = status1[:maxlen-4]
status1 += "... "
}
}
PrintProgress("%s", status1)
}
archiveProgress.OnDone = func(s restic.Stat, d time.Duration, ticker bool) {
fmt.Printf("\nduration: %s\n", formatDuration(d))
}
return archiveProgress
}
// filterExisting returns a slice of all existing items, or an error if no
// items exist at all.
func filterExisting(items []string) (result []string, err error) {
@ -120,33 +231,78 @@ func filterExisting(items []string) (result []string, err error) {
return
}
// readFromFile will read all lines from the given filename and return them as
// a string array, if filename is empty readFromFile returns and empty string
// array. If filename is a dash (-), readFromFile will read the lines from the
// standard input.
func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string) error {
if len(args) != 0 {
return errors.Fatal("when reading from stdin, no additional files can be specified")
}
fn := opts.StdinFilename
if fn == "" {
return errors.Fatal("filename for backup from stdin must not be empty")
}
if filepath.Base(fn) != fn || path.Base(fn) != fn {
return errors.Fatal("filename is invalid (may not contain a directory, slash or backslash)")
}
if gopts.password == "" {
return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
}
repo, err := OpenRepository(gopts)
if err != nil {
return err
}
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
err = repo.LoadIndex(gopts.ctx)
if err != nil {
return err
}
r := &archiver.Reader{
Repository: repo,
Tags: opts.Tags,
Hostname: opts.Hostname,
}
_, id, err := r.Archive(gopts.ctx, fn, os.Stdin, newArchiveStdinProgress(gopts))
if err != nil {
return err
}
Verbosef("archived as %v\n", id.Str())
return nil
}
// readFromFile will read all lines from the given filename and write them to a
// string array, if filename is empty readFromFile returns and empty string
// array. If filename is a dash (-), readFromFile will read the lines from
// the standard input.
func readLinesFromFile(filename string) ([]string, error) {
if filename == "" {
return nil, nil
}
var (
data []byte
err error
)
if filename == "-" {
data, err = ioutil.ReadAll(os.Stdin)
} else {
data, err = textfile.Read(filename)
}
if err != nil {
return nil, err
var r io.Reader = os.Stdin
if filename != "-" {
f, err := os.Open(filename)
if err != nil {
return nil, err
}
defer f.Close()
r = f
}
var lines []string
scanner := bufio.NewScanner(bytes.NewReader(data))
scanner := bufio.NewScanner(r)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
// ignore empty lines
@ -167,58 +323,56 @@ func readLinesFromFile(filename string) ([]string, error) {
return lines, nil
}
// Check returns an error when an invalid combination of options was set.
func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
func runBackup(opts BackupOptions, gopts GlobalOptions, args []string) error {
if opts.FilesFrom == "-" && gopts.password == "" {
return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
}
if opts.Stdin {
if opts.FilesFrom != "" {
return errors.Fatal("--stdin and --files-from cannot be used together")
}
if len(args) > 0 {
return errors.Fatal("--stdin was specified and files/dirs were listed as arguments")
}
fromfile, err := readLinesFromFile(opts.FilesFrom)
if err != nil {
return err
}
return nil
}
// merge files from files-from into normal args so we can reuse the normal
// args checks and have the ability to use both files-from and args at the
// same time
args = append(args, fromfile...)
if len(args) == 0 {
return errors.Fatal("nothing to backup, please specify target files/dirs")
}
target := make([]string, 0, len(args))
for _, d := range args {
if a, err := filepath.Abs(d); err == nil {
d = a
}
target = append(target, d)
}
target, err = filterExisting(target)
if err != nil {
return err
}
// rejectFuncs collect functions that can reject items from the backup
var rejectFuncs []RejectFunc
// collectRejectFuncs returns a list of all functions which may reject data
// from being saved in a snapshot
func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectFunc, err error) {
// allowed devices
if opts.ExcludeOtherFS && !opts.Stdin {
f, err := rejectByDevice(targets)
if opts.ExcludeOtherFS {
f, err := rejectByDevice(target)
if err != nil {
return nil, err
return err
}
fs = append(fs, f)
}
// exclude restic cache
if repo.Cache != nil {
f, err := rejectResticCache(repo)
if err != nil {
return nil, err
}
fs = append(fs, f)
rejectFuncs = append(rejectFuncs, f)
}
// add patterns from file
if len(opts.ExcludeFiles) > 0 {
excludes, err := readExcludePatternsFromFiles(opts.ExcludeFiles)
if err != nil {
return nil, err
}
opts.Excludes = append(opts.Excludes, excludes...)
opts.Excludes = append(opts.Excludes, readExcludePatternsFromFiles(opts.ExcludeFiles)...)
}
if len(opts.Excludes) > 0 {
fs = append(fs, rejectByPattern(opts.Excludes))
rejectFuncs = append(rejectFuncs, rejectByPattern(opts.Excludes))
}
if opts.ExcludeCaches {
@ -228,37 +382,124 @@ func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets
for _, spec := range opts.ExcludeIfPresent {
f, err := rejectIfPresent(spec)
if err != nil {
return nil, err
return err
}
fs = append(fs, f)
rejectFuncs = append(rejectFuncs, f)
}
return fs, nil
repo, err := OpenRepository(gopts)
if err != nil {
return err
}
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
// exclude restic cache
if repo.Cache != nil {
f, err := rejectResticCache(repo)
if err != nil {
return err
}
rejectFuncs = append(rejectFuncs, f)
}
err = repo.LoadIndex(gopts.ctx)
if err != nil {
return err
}
var parentSnapshotID *restic.ID
// Force using a parent
if !opts.Force && opts.Parent != "" {
id, err := restic.FindSnapshot(repo, opts.Parent)
if err != nil {
return errors.Fatalf("invalid id %q: %v", opts.Parent, err)
}
parentSnapshotID = &id
}
// Find last snapshot to set it as parent, if not already set
if !opts.Force && parentSnapshotID == nil {
id, err := restic.FindLatestSnapshot(gopts.ctx, repo, target, []restic.TagList{}, opts.Hostname)
if err == nil {
parentSnapshotID = &id
} else if err != restic.ErrNoSnapshotFound {
return err
}
}
if parentSnapshotID != nil {
Verbosef("using parent snapshot %v\n", parentSnapshotID.Str())
}
Verbosef("scan %v\n", target)
selectFilter := func(item string, fi os.FileInfo) bool {
for _, reject := range rejectFuncs {
if reject(item, fi) {
return false
}
}
return true
}
stat, err := archiver.Scan(target, selectFilter, newScanProgress(gopts))
if err != nil {
return err
}
arch := archiver.New(repo)
arch.Excludes = opts.Excludes
arch.SelectFilter = selectFilter
arch.WithAccessTime = opts.WithAtime
arch.Warn = func(dir string, fi os.FileInfo, err error) {
// TODO: make ignoring errors configurable
Warnf("%s\rwarning for %s: %v\n", ClearLine(), dir, err)
}
timeStamp := time.Now()
if opts.TimeStamp != "" {
timeStamp, err = time.Parse(TimeFormat, opts.TimeStamp)
if err != nil {
return errors.Fatalf("error in time option: %v\n", err)
}
}
_, id, err := arch.Snapshot(gopts.ctx, newArchiveProgress(gopts, stat), target, opts.Tags, opts.Hostname, parentSnapshotID, timeStamp)
if err != nil {
return err
}
Verbosef("snapshot %s saved\n", id.Str())
return nil
}
// readExcludePatternsFromFiles reads all exclude files and returns the list of
// exclude patterns. For each line, leading and trailing white space is removed
// and comment lines are ignored. For each remaining pattern, environment
// variables are resolved. For adding a literal dollar sign ($), write $$ to
// the file.
func readExcludePatternsFromFiles(excludeFiles []string) ([]string, error) {
getenvOrDollar := func(s string) string {
if s == "$" {
return "$"
}
return os.Getenv(s)
}
func readExcludePatternsFromFiles(excludeFiles []string) []string {
var excludes []string
for _, filename := range excludeFiles {
err := func() (err error) {
data, err := textfile.Read(filename)
file, err := fs.Open(filename)
if err != nil {
return err
}
defer func() {
// return pre-close error if there was one
if errClose := file.Close(); err == nil {
err = errClose
}
}()
scanner := bufio.NewScanner(bytes.NewReader(data))
scanner := bufio.NewScanner(file)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
@ -272,228 +513,15 @@ func readExcludePatternsFromFiles(excludeFiles []string) ([]string, error) {
continue
}
line = os.Expand(line, getenvOrDollar)
line = os.ExpandEnv(line)
excludes = append(excludes, line)
}
return scanner.Err()
}()
if err != nil {
return nil, err
Warnf("error reading exclude patterns: %v:", err)
return nil
}
}
return excludes, nil
}
// collectTargets returns a list of target files/dirs from several sources.
func collectTargets(opts BackupOptions, args []string) (targets []string, err error) {
if opts.Stdin {
return nil, nil
}
fromfile, err := readLinesFromFile(opts.FilesFrom)
if err != nil {
return nil, err
}
// merge files from files-from into normal args so we can reuse the normal
// args checks and have the ability to use both files-from and args at the
// same time
args = append(args, fromfile...)
if len(args) == 0 && !opts.Stdin {
return nil, errors.Fatal("nothing to backup, please specify target files/dirs")
}
targets = args
targets, err = filterExisting(targets)
if err != nil {
return nil, err
}
return targets, nil
}
// parent returns the ID of the parent snapshot. If there is none, nil is
// returned.
func findParentSnapshot(ctx context.Context, repo restic.Repository, opts BackupOptions, targets []string) (parentID *restic.ID, err error) {
// Force using a parent
if !opts.Force && opts.Parent != "" {
id, err := restic.FindSnapshot(repo, opts.Parent)
if err != nil {
return nil, errors.Fatalf("invalid id %q: %v", opts.Parent, err)
}
parentID = &id
}
// Find last snapshot to set it as parent, if not already set
if !opts.Force && parentID == nil {
id, err := restic.FindLatestSnapshot(ctx, repo, targets, []restic.TagList{}, opts.Hostname)
if err == nil {
parentID = &id
} else if err != restic.ErrNoSnapshotFound {
return nil, err
}
}
return parentID, nil
}
func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Terminal, args []string) error {
err := opts.Check(gopts, args)
if err != nil {
return err
}
targets, err := collectTargets(opts, args)
if err != nil {
return err
}
timeStamp := time.Now()
if opts.TimeStamp != "" {
timeStamp, err = time.Parse(TimeFormat, opts.TimeStamp)
if err != nil {
return errors.Fatalf("error in time option: %v\n", err)
}
}
var t tomb.Tomb
p := ui.NewBackup(term, gopts.verbosity)
// use the terminal for stdout/stderr
prevStdout, prevStderr := gopts.stdout, gopts.stderr
defer func() {
gopts.stdout, gopts.stderr = prevStdout, prevStderr
}()
gopts.stdout, gopts.stderr = p.Stdout(), p.Stderr()
if s, ok := os.LookupEnv("RESTIC_PROGRESS_FPS"); ok {
fps, err := strconv.Atoi(s)
if err == nil && fps >= 1 {
if fps > 60 {
fps = 60
}
p.MinUpdatePause = time.Second / time.Duration(fps)
}
}
t.Go(func() error { return p.Run(t.Context(gopts.ctx)) })
p.V("open repository")
repo, err := OpenRepository(gopts)
if err != nil {
return err
}
p.V("lock repository")
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
// rejectFuncs collect functions that can reject items from the backup
rejectFuncs, err := collectRejectFuncs(opts, repo, targets)
if err != nil {
return err
}
p.V("load index files")
err = repo.LoadIndex(gopts.ctx)
if err != nil {
return err
}
parentSnapshotID, err := findParentSnapshot(gopts.ctx, repo, opts, targets)
if err != nil {
return err
}
if parentSnapshotID != nil {
p.V("using parent snapshot %v\n", parentSnapshotID.Str())
}
selectFilter := func(item string, fi os.FileInfo) bool {
for _, reject := range rejectFuncs {
if reject(item, fi) {
return false
}
}
return true
}
var targetFS fs.FS = fs.Local{}
if opts.Stdin {
p.V("read data from stdin")
targetFS = &fs.Reader{
ModTime: timeStamp,
Name: opts.StdinFilename,
Mode: 0644,
ReadCloser: os.Stdin,
}
targets = []string{opts.StdinFilename}
}
sc := archiver.NewScanner(targetFS)
sc.Select = selectFilter
sc.Error = p.ScannerError
sc.Result = p.ReportTotal
p.V("start scan on %v", targets)
t.Go(func() error { return sc.Scan(t.Context(gopts.ctx), targets) })
arch := archiver.New(repo, targetFS, archiver.Options{})
arch.Select = selectFilter
arch.WithAtime = opts.WithAtime
arch.Error = p.Error
arch.CompleteItem = p.CompleteItemFn
arch.StartFile = p.StartFile
arch.CompleteBlob = p.CompleteBlob
if parentSnapshotID == nil {
parentSnapshotID = &restic.ID{}
}
snapshotOpts := archiver.SnapshotOptions{
Excludes: opts.Excludes,
Tags: opts.Tags,
Time: timeStamp,
Hostname: opts.Hostname,
ParentSnapshot: *parentSnapshotID,
}
uploader := archiver.IndexUploader{
Repository: repo,
Start: func() {
p.VV("uploading intermediate index")
},
Complete: func(id restic.ID) {
p.V("uploaded intermediate index %v", id.Str())
},
}
t.Go(func() error {
return uploader.Upload(gopts.ctx, t.Context(gopts.ctx), 30*time.Second)
})
p.V("start backup on %v", targets)
_, id, err := arch.Snapshot(gopts.ctx, targets, snapshotOpts)
if err != nil {
return errors.Fatalf("unable to save snapshot: %v", err)
}
p.Finish()
p.P("snapshot %s saved\n", id.Str())
// cleanly shutdown all running goroutines
t.Kill(nil)
// let's see if one returned an error
err = t.Wait()
if err != nil {
return err
}
return nil
return excludes
}

122
cmd/restic/cmd_cache.go
View file

@ -1,122 +0,0 @@
package main
import (
"fmt"
"path/filepath"
"sort"
"time"
"github.com/restic/restic/internal/cache"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/fs"
"github.com/spf13/cobra"
)
var cmdCache = &cobra.Command{
Use: "cache",
Short: "Operate on local cache directories",
Long: `
The "cache" command allows listing and cleaning local cache directories.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runCache(cacheOptions, globalOptions, args)
},
}
// CacheOptions bundles all options for the snapshots command.
type CacheOptions struct {
Cleanup bool
MaxAge uint
}
var cacheOptions CacheOptions
func init() {
cmdRoot.AddCommand(cmdCache)
f := cmdCache.Flags()
f.BoolVar(&cacheOptions.Cleanup, "cleanup", false, "remove old cache directories")
f.UintVar(&cacheOptions.MaxAge, "max-age", 30, "max age in `days` for cache directories to be considered old")
}
func runCache(opts CacheOptions, gopts GlobalOptions, args []string) error {
if len(args) > 0 {
return errors.Fatal("the cache command has no arguments")
}
if gopts.NoCache {
return errors.Fatal("Refusing to do anything, the cache is disabled")
}
var (
cachedir = gopts.CacheDir
err error
)
if cachedir == "" {
cachedir, err = cache.DefaultDir()
if err != nil {
return err
}
}
if opts.Cleanup || gopts.CleanupCache {
oldDirs, err := cache.OlderThan(cachedir, time.Duration(opts.MaxAge)*24*time.Hour)
if err != nil {
return err
}
if len(oldDirs) == 0 {
Verbosef("no old cache dirs found\n")
return nil
}
Verbosef("remove %d old cache directories\n", len(oldDirs))
for _, item := range oldDirs {
dir := filepath.Join(cachedir, item.Name())
err = fs.RemoveAll(dir)
if err != nil {
Warnf("unable to remove %v: %v\n", dir, err)
}
}
return nil
}
tab := NewTable()
tab.Header = fmt.Sprintf("%-14s %-16s %s", "Repository ID", "Last Used", "Old")
tab.RowFormat = "%-14s %-16s %s"
dirs, err := cache.All(cachedir)
if err != nil {
return err
}
if len(dirs) == 0 {
Printf("no cache dirs found, basedir is %v\n", cachedir)
return nil
}
sort.Slice(dirs, func(i, j int) bool {
return dirs[i].ModTime().Before(dirs[j].ModTime())
})
for _, entry := range dirs {
var old string
if cache.IsOld(entry.ModTime(), time.Duration(opts.MaxAge)*24*time.Hour) {
old = "yes"
}
tab.Rows = append(tab.Rows, []interface{}{
entry.Name()[:10],
fmt.Sprintf("%d days ago", uint(time.Since(entry.ModTime()).Hours()/24)),
old,
})
}
tab.Write(gopts.stdout)
return nil
}

2
cmd/restic/cmd_cat.go
View file

@ -58,7 +58,7 @@ func runCat(gopts GlobalOptions, args []string) error {
// find snapshot id with prefix
id, err = restic.FindSnapshot(repo, args[1])
if err != nil {
return errors.Fatalf("could not find snapshot: %v\n", err)
return err
}
}
}

67
cmd/restic/cmd_check.go
View file

@ -2,7 +2,6 @@ package main
import (
"fmt"
"io/ioutil"
"os"
"strconv"
"strings"
@ -12,7 +11,6 @@ import (
"github.com/restic/restic/internal/checker"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/fs"
"github.com/restic/restic/internal/restic"
)
@ -50,7 +48,7 @@ func init() {
f := cmdCheck.Flags()
f.BoolVar(&checkOptions.ReadData, "read-data", false, "read all data blobs")
f.StringVar(&checkOptions.ReadDataSubset, "read-data-subset", "", "read subset n of m data packs (format: `n/m`)")
f.StringVar(&checkOptions.ReadDataSubset, "read-data-subset", "", "read subset of data packs")
f.BoolVar(&checkOptions.CheckUnused, "check-unused", false, "find unused blobs")
f.BoolVar(&checkOptions.WithCache, "with-cache", false, "use the cache")
}
@ -119,58 +117,15 @@ func newReadProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
return readProgress
}
// prepareCheckCache configures a special cache directory for check.
//
// * if --with-cache is specified, the default cache is used
// * if the user explicitly requested --no-cache, we don't use any cache
// * if the user provides --cache-dir, we use a cache in a temporary sub-directory of the specified directory and the sub-directory is deleted after the check
// * by default, we use a cache in a temporary directory that is deleted after the check
func prepareCheckCache(opts CheckOptions, gopts *GlobalOptions) (cleanup func()) {
cleanup = func() {}
if opts.WithCache {
// use the default cache, no setup needed
return cleanup
}
if gopts.NoCache {
// don't use any cache, no setup needed
return cleanup
}
cachedir := gopts.CacheDir
// use a cache in a temporary directory
tempdir, err := ioutil.TempDir(cachedir, "restic-check-cache-")
if err != nil {
// if an error occurs, don't use any cache
Warnf("unable to create temporary directory for cache during check, disabling cache: %v\n", err)
gopts.NoCache = true
return cleanup
}
gopts.CacheDir = tempdir
Verbosef("using temporary cache in %v\n", tempdir)
cleanup = func() {
err := fs.RemoveAll(tempdir)
if err != nil {
Warnf("error removing temporary cache directory: %v\n", err)
}
}
return cleanup
}
func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
if len(args) != 0 {
return errors.Fatal("check has no arguments")
}
cleanup := prepareCheckCache(opts, &gopts)
AddCleanupHandler(func() error {
cleanup()
return nil
})
if !opts.WithCache {
// do not use a cache for the checker
gopts.NoCache = true
}
repo, err := OpenRepository(gopts)
if err != nil {
@ -200,7 +155,7 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
}
if dupFound {
Printf("This is non-critical, you can run `restic rebuild-index' to correct this\n")
Printf("\nrun `restic rebuild-index' to correct this\n")
}
if len(errs) > 0 {
@ -211,26 +166,16 @@ func runCheck(opts CheckOptions, gopts GlobalOptions, args []string) error {
}
errorsFound := false
orphanedPacks := 0
errChan := make(chan error)
Verbosef("check all packs\n")
go chkr.Packs(gopts.ctx, errChan)
for err := range errChan {
if checker.IsOrphanedPack(err) {
orphanedPacks++
Verbosef("%v\n", err)
continue
}
errorsFound = true
fmt.Fprintf(os.Stderr, "%v\n", err)
}
if orphanedPacks > 0 {
Verbosef("%d additional files were found in the repo, which likely contain duplicate data.\nYou can run `restic prune` to correct this.\n", orphanedPacks)
}
Verbosef("check snapshots, trees and blobs\n")
errChan = make(chan error)
go chkr.Structure(gopts.ctx, errChan)

152
cmd/restic/cmd_find.go
View file

@ -3,6 +3,7 @@ package main
import (
"context"
"encoding/json"
"path/filepath"
"strings"
"time"
@ -10,9 +11,7 @@ import (
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/filter"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/walker"
)
var cmdFind = &cobra.Command{
@ -95,7 +94,7 @@ type statefulOutput struct {
hits int
}
func (s *statefulOutput) PrintJSON(path string, node *restic.Node) {
func (s *statefulOutput) PrintJSON(prefix string, node *restic.Node) {
type findNode restic.Node
b, err := json.Marshal(struct {
// Add these attributes
@ -112,7 +111,7 @@ func (s *statefulOutput) PrintJSON(path string, node *restic.Node) {
Content byte `json:"content,omitempty"`
Subtree byte `json:"subtree,omitempty"`
}{
Path: path,
Path: filepath.Join(prefix, node.Name),
Permissions: node.Mode.String(),
findNode: (*findNode)(node),
})
@ -139,22 +138,22 @@ func (s *statefulOutput) PrintJSON(path string, node *restic.Node) {
s.hits++
}
func (s *statefulOutput) PrintNormal(path string, node *restic.Node) {
func (s *statefulOutput) PrintNormal(prefix string, node *restic.Node) {
if s.newsn != s.oldsn {
if s.oldsn != nil {
Verbosef("\n")
}
s.oldsn = s.newsn
Verbosef("Found matching entries in snapshot %s\n", s.oldsn.ID().Str())
Verbosef("Found matching entries in snapshot %s\n", s.oldsn.ID())
}
Printf(formatNode(path, node, s.ListLong) + "\n")
Printf(formatNode(prefix, node, s.ListLong) + "\n")
}
func (s *statefulOutput) Print(path string, node *restic.Node) {
func (s *statefulOutput) Print(prefix string, node *restic.Node) {
if s.JSON {
s.PrintJSON(path, node)
s.PrintJSON(prefix, node)
} else {
s.PrintNormal(path, node)
s.PrintNormal(prefix, node)
}
}
@ -175,75 +174,74 @@ func (s *statefulOutput) Finish() {
// Finder bundles information needed to find a file or directory.
type Finder struct {
repo restic.Repository
pat findPattern
out statefulOutput
ignoreTrees restic.IDSet
repo restic.Repository
pat findPattern
out statefulOutput
notfound restic.IDSet
}
func (f *Finder) findInTree(ctx context.Context, treeID restic.ID, prefix string) error {
if f.notfound.Has(treeID) {
debug.Log("%v skipping tree %v, has already been checked", prefix, treeID)
return nil
}
debug.Log("%v checking tree %v\n", prefix, treeID)
tree, err := f.repo.LoadTree(ctx, treeID)
if err != nil {
return err
}
var found bool
for _, node := range tree.Nodes {
debug.Log(" testing entry %q\n", node.Name)
name := node.Name
if f.pat.ignoreCase {
name = strings.ToLower(name)
}
m, err := filepath.Match(f.pat.pattern, name)
if err != nil {
return err
}
if m {
if !f.pat.oldest.IsZero() && node.ModTime.Before(f.pat.oldest) {
debug.Log(" ModTime is older than %s\n", f.pat.oldest)
continue
}
if !f.pat.newest.IsZero() && node.ModTime.After(f.pat.newest) {
debug.Log(" ModTime is newer than %s\n", f.pat.newest)
continue
}
debug.Log(" found match\n")
found = true
f.out.Print(prefix, node)
}
if node.Type == "dir" {
if err := f.findInTree(ctx, *node.Subtree, filepath.Join(prefix, node.Name)); err != nil {
return err
}
}
}
if !found {
f.notfound.Insert(treeID)
}
return nil
}
func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error {
debug.Log("searching in snapshot %s\n for entries within [%s %s]", sn.ID(), f.pat.oldest, f.pat.newest)
if sn.Tree == nil {
return errors.Errorf("snapshot %v has no tree", sn.ID().Str())
}
f.out.newsn = sn
return walker.Walk(ctx, f.repo, *sn.Tree, f.ignoreTrees, func(nodepath string, node *restic.Node, err error) (bool, error) {
if err != nil {
return false, err
}
if node == nil {
return false, nil
}
normalizedNodepath := nodepath
if f.pat.ignoreCase {
normalizedNodepath = strings.ToLower(nodepath)
}
foundMatch, err := filter.Match(f.pat.pattern, normalizedNodepath)
if err != nil {
return false, err
}
var (
ignoreIfNoMatch = true
errIfNoMatch error
)
if node.Type == "dir" {
childMayMatch, err := filter.ChildMatch(f.pat.pattern, normalizedNodepath)
if err != nil {
return false, err
}
if !childMayMatch {
ignoreIfNoMatch = true
errIfNoMatch = walker.SkipNode
} else {
ignoreIfNoMatch = false
}
}
if !foundMatch {
return ignoreIfNoMatch, errIfNoMatch
}
if !f.pat.oldest.IsZero() && node.ModTime.Before(f.pat.oldest) {
debug.Log(" ModTime is older than %s\n", f.pat.oldest)
return ignoreIfNoMatch, errIfNoMatch
}
if !f.pat.newest.IsZero() && node.ModTime.After(f.pat.newest) {
debug.Log(" ModTime is newer than %s\n", f.pat.newest)
return ignoreIfNoMatch, errIfNoMatch
}
debug.Log(" found match\n")
f.out.Print(nodepath, node)
return false, nil
})
return f.findInTree(ctx, *sn.Tree, string(filepath.Separator))
}
func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
@ -291,10 +289,10 @@ func runFind(opts FindOptions, gopts GlobalOptions, args []string) error {
defer cancel()
f := &Finder{
repo: repo,
pat: pat,
out: statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON},
ignoreTrees: restic.NewIDSet(),
repo: repo,
pat: pat,
out: statefulOutput{ListLong: opts.ListLong, JSON: globalOptions.JSON},
notfound: restic.NewIDSet(),
}
for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, opts.Snapshots) {
if err = f.findInSnapshot(ctx, sn); err != nil {

5
cmd/restic/cmd_forget.go
View file

@ -33,7 +33,6 @@ type ForgetOptions struct {
Weekly int
Monthly int
Yearly int
Within restic.Duration
KeepTags restic.TagLists
Host string
@ -59,7 +58,6 @@ func init() {
f.IntVarP(&forgetOptions.Weekly, "keep-weekly", "w", 0, "keep the last `n` weekly snapshots")
f.IntVarP(&forgetOptions.Monthly, "keep-monthly", "m", 0, "keep the last `n` monthly snapshots")
f.IntVarP(&forgetOptions.Yearly, "keep-yearly", "y", 0, "keep the last `n` yearly snapshots")
f.VarP(&forgetOptions.Within, "keep-within", "", "keep snapshots that were created within `duration` before the newest (e.g. 1y5m7d)")
f.Var(&forgetOptions.KeepTags, "keep-tag", "keep snapshots with this `taglist` (can be specified multiple times)")
// Sadly the commonly used shortcut `H` is already used.
@ -172,7 +170,6 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
Weekly: opts.Weekly,
Monthly: opts.Monthly,
Yearly: opts.Yearly,
Within: opts.Within,
Tags: opts.KeepTags,
}
@ -181,8 +178,6 @@ func runForget(opts ForgetOptions, gopts GlobalOptions, args []string) error {
}
if !policy.Empty() {
Verbosef("Applying Policy: %v\n", policy)
for k, snapshotGroup := range snapshotGroups {
var key key
if json.Unmarshal([]byte(k), &key) != nil {

95
cmd/restic/cmd_key.go
View file

@ -2,11 +2,7 @@ package main
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
"os"
"strings"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
@ -27,84 +23,37 @@ The "key" command manages keys (passwords) for accessing the repository.
},
}
var newPasswordFile string
func init() {
cmdRoot.AddCommand(cmdKey)
flags := cmdKey.Flags()
flags.StringVarP(&newPasswordFile, "new-password-file", "", "", "the file from which to load a new password")
}
type keyInfo struct {
Current bool `json:"current"`
ID string `json:"id"`
UserName string `json:"userName"`
HostName string `json:"hostName"`
Created string `json:"created"`
}
func listKeys(ctx context.Context, s *repository.Repository) error {
tab := NewTable()
tab.Header = fmt.Sprintf(" %-10s %-10s %-10s %s", "ID", "User", "Host", "Created")
tab.RowFormat = "%s%-10s %-10s %-10s %s"
func (ki keyInfo) CurrentStr() string {
if ki.Current {
return "*"
}
return " "
}
func listKeys(ctx context.Context, s *repository.Repository, gopts GlobalOptions) error {
var (
appendKey func(keyInfo)
printKeys func() error
)
switch gopts.JSON {
case true:
var keys []keyInfo
appendKey = func(key keyInfo) {
keys = append(keys, key)
}
printKeys = func() error {
return json.NewEncoder(gopts.stdout).Encode(keys)
}
default:
tab := NewTable()
tab.Header = fmt.Sprintf(" %-10s %-10s %-10s %s", "ID", "User", "Host", "Created")
tab.RowFormat = "%s%-10s %-10s %-10s %s"
appendKey = func(key keyInfo) {
tab.Rows = append(tab.Rows, []interface{}{key.CurrentStr(), key.ID, key.UserName, key.HostName, key.Created})
}
printKeys = func() error {
return tab.Write(globalOptions.stdout)
}
}
if err := s.List(ctx, restic.KeyFile, func(id restic.ID, size int64) error {
err := s.List(ctx, restic.KeyFile, func(id restic.ID, size int64) error {
k, err := repository.LoadKey(ctx, s, id.String())
if err != nil {
Warnf("LoadKey() failed: %v\n", err)
return nil
}
key := keyInfo{
Current: id.String() == s.KeyName(),
ID: id.Str(),
UserName: k.Username,
HostName: k.Hostname,
Created: k.Created.Format(TimeFormat),
var current string
if id.String() == s.KeyName() {
current = "*"
} else {
current = " "
}
appendKey(key)
tab.Rows = append(tab.Rows, []interface{}{current, id.Str(),
k.Username, k.Hostname, k.Created.Format(TimeFormat)})
return nil
}); err != nil {
})
if err != nil {
return err
}
return printKeys()
return tab.Write(globalOptions.stdout)
}
// testKeyNewPassword is used to set a new password during integration testing.
@ -115,10 +64,6 @@ func getNewPassword(gopts GlobalOptions) (string, error) {
return testKeyNewPassword, nil
}
if newPasswordFile != "" {
return loadPasswordFromFile(newPasswordFile)
}
// Since we already have an open repository, temporary remove the password
// to prompt the user for the passwd.
newopts := gopts
@ -203,7 +148,7 @@ func runKey(gopts GlobalOptions, args []string) error {
return err
}
return listKeys(ctx, repo, gopts)
return listKeys(ctx, repo)
case "add":
lock, err := lockRepo(repo)
defer unlockRepo(lock)
@ -237,11 +182,3 @@ func runKey(gopts GlobalOptions, args []string) error {
return nil
}
func loadPasswordFromFile(pwdFile string) (string, error) {
s, err := ioutil.ReadFile(pwdFile)
if os.IsNotExist(err) {
return "", errors.Fatalf("%s does not exist", pwdFile)
}
return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
}

6
cmd/restic/cmd_list.go
View file

@ -18,7 +18,7 @@ The "list" command allows listing objects in the repository based on type.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runList(cmd, globalOptions, args)
return runList(globalOptions, args)
},
}
@ -26,9 +26,9 @@ func init() {
cmdRoot.AddCommand(cmdList)
}
func runList(cmd *cobra.Command, opts GlobalOptions, args []string) error {
func runList(opts GlobalOptions, args []string) error {
if len(args) != 1 {
return errors.Fatal("type not specified, usage: " + cmd.Use)
return errors.Fatal("type not specified")
}
repo, err := OpenRepository(opts)

35
cmd/restic/cmd_ls.go
View file

@ -2,12 +2,13 @@ package main
import (
"context"
"path/filepath"
"github.com/spf13/cobra"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/walker"
)
var cmdLs = &cobra.Command{
@ -45,6 +46,25 @@ func init() {
flags.StringArrayVar(&lsOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot ID is given")
}
func printTree(ctx context.Context, repo *repository.Repository, id *restic.ID, prefix string) error {
tree, err := repo.LoadTree(ctx, *id)
if err != nil {
return err
}
for _, entry := range tree.Nodes {
Printf("%s\n", formatNode(prefix, entry, lsOptions.ListLong))
if entry.Type == "dir" && entry.Subtree != nil {
if err = printTree(ctx, repo, entry.Subtree, filepath.Join(prefix, entry.Name)); err != nil {
return err
}
}
}
return nil
}
func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
if len(args) == 0 && opts.Host == "" && len(opts.Tags) == 0 && len(opts.Paths) == 0 {
return errors.Fatal("Invalid arguments, either give one or more snapshot IDs or set filters.")
@ -64,18 +84,7 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args) {
Verbosef("snapshot %s of %v at %s):\n", sn.ID().Str(), sn.Paths, sn.Time)
err := walker.Walk(ctx, repo, *sn.Tree, nil, func(nodepath string, node *restic.Node, err error) (bool, error) {
if err != nil {
return false, err
}
if node == nil {
return false, nil
}
Printf("%s\n", formatNode(nodepath, node, lsOptions.ListLong))
return false, nil
})
if err != nil {
if err = printTree(gopts.ctx, repo, sn.Tree, string(filepath.Separator)); err != nil {
return err
}
}

1
cmd/restic/cmd_mount.go
View file

@ -1,4 +1,3 @@
// +build !netbsd
// +build !openbsd
// +build !solaris
// +build !windows

16
cmd/restic/cmd_prune.go
View file

@ -271,6 +271,22 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
Verbosef("will delete %d packs and rewrite %d packs, this frees %s\n",
len(removePacks), len(rewritePacks), formatBytes(uint64(removeBytes)))
if len(removePacks) != 0 {
bar = newProgressMax(!gopts.Quiet, uint64(len(removePacks)), "packs deleted")
bar.Start()
for packID := range removePacks {
h := restic.Handle{Type: restic.DataFile, Name: packID.String()}
err = repo.Backend().Remove(ctx, h)
if err != nil {
Warnf("unable to remove file %v from the repository\n", packID.Str())
}
bar.Report(restic.Stat{Blobs: 1})
}
bar.Done()
}
removePacks = restic.NewIDSet()
var obsoletePacks restic.IDSet
if len(rewritePacks) != 0 {
bar = newProgressMax(!gopts.Quiet, uint64(len(rewritePacks)), "packs rewritten")

11
cmd/restic/cmd_restore.go
View file

@ -5,7 +5,6 @@ import (
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/filter"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/restorer"
"github.com/spf13/cobra"
)
@ -34,7 +33,6 @@ type RestoreOptions struct {
Host string
Paths []string
Tags restic.TagLists
Verify bool
}
var restoreOptions RestoreOptions
@ -50,7 +48,6 @@ func init() {
flags.StringVarP(&restoreOptions.Host, "host", "H", "", `only consider snapshots for this host when the snapshot ID is "latest"`)
flags.Var(&restoreOptions.Tags, "tag", "only consider snapshots which include this `taglist` for snapshot ID \"latest\"")
flags.StringArrayVar(&restoreOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` for snapshot ID \"latest\"")
flags.BoolVar(&restoreOptions.Verify, "verify", false, "verify restored files content")
}
func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
@ -107,7 +104,7 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
}
}
res, err := restorer.NewRestorer(repo, id)
res, err := restic.NewRestorer(repo, id)
if err != nil {
Exitf(2, "creating restorer failed: %v\n", err)
}
@ -156,12 +153,6 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
Verbosef("restoring %s to %s\n", res.Snapshot(), opts.Target)
err = res.RestoreTo(ctx, opts.Target)
if err == nil && opts.Verify {
Verbosef("verifying files in %s\n", opts.Target)
var count int
count, err = res.VerifyFiles(ctx, opts.Target)
Verbosef("finished verifying %d files in %s\n", count, opts.Target)
}
if totalErrors > 0 {
Printf("There were %d errors\n", totalErrors)
}

314
cmd/restic/cmd_stats.go
View file

@ -1,314 +0,0 @@
package main
import (
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"os"
"path/filepath"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/walker"
"github.com/spf13/cobra"
)
var cmdStats = &cobra.Command{
Use: "stats [flags] [snapshot-ID]",
Short: "Scan the repository and show basic statistics",
Long: `
The "stats" command walks one or all snapshots in a repository and
accumulates statistics about the data stored therein. It reports on
the number of unique files and their sizes, according to one of
the counting modes as given by the --mode flag.
If no snapshot is specified, all snapshots will be considered. Some
modes make more sense over just a single snapshot, while others
are useful across all snapshots, depending on what you are trying
to calculate.
The modes are:
restore-size: (default) Counts the size of the restored files.
files-by-contents: Counts total size of files, where a file is
considered unique if it has unique contents.
raw-data: Counts the size of blobs in the repository, regardless
of how many files reference them.
blobs-per-file: A combination of files-by-contents and raw-data.
Refer to the online manual for more details about each mode.
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runStats(globalOptions, args)
},
}
func init() {
cmdRoot.AddCommand(cmdStats)
f := cmdStats.Flags()
f.StringVar(&countMode, "mode", countModeRestoreSize, "counting mode: restore-size (default), files-by-contents, blobs-per-file, or raw-data")
f.StringVar(&snapshotByHost, "host", "", "filter latest snapshot by this hostname")
}
func runStats(gopts GlobalOptions, args []string) error {
err := verifyStatsInput(gopts, args)
if err != nil {
return err
}
ctx, cancel := context.WithCancel(gopts.ctx)
defer cancel()
repo, err := OpenRepository(gopts)
if err != nil {
return err
}
if err = repo.LoadIndex(ctx); err != nil {
return err
}
if !gopts.NoLock {
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
}
// create a container for the stats (and other needed state)
stats := &statsContainer{
uniqueFiles: make(map[fileID]struct{}),
fileBlobs: make(map[string]restic.IDSet),
blobs: restic.NewBlobSet(),
blobsSeen: restic.NewBlobSet(),
}
if snapshotIDString != "" {
// scan just a single snapshot
var sID restic.ID
if snapshotIDString == "latest" {
sID, err = restic.FindLatestSnapshot(ctx, repo, []string{}, []restic.TagList{}, snapshotByHost)
if err != nil {
Exitf(1, "latest snapshot for criteria not found: %v", err)
}
} else {
sID, err = restic.FindSnapshot(repo, snapshotIDString)
if err != nil {
return err
}
}
snapshot, err := restic.LoadSnapshot(ctx, repo, sID)
if err != nil {
return err
}
err = statsWalkSnapshot(ctx, snapshot, repo, stats)
} else {
// iterate every snapshot in the repo
err = repo.List(ctx, restic.SnapshotFile, func(snapshotID restic.ID, size int64) error {
snapshot, err := restic.LoadSnapshot(ctx, repo, snapshotID)
if err != nil {
return fmt.Errorf("Error loading snapshot %s: %v", snapshotID.Str(), err)
}
return statsWalkSnapshot(ctx, snapshot, repo, stats)
})
}
if err != nil {
return err
}
if countMode == countModeRawData {
// the blob handles have been collected, but not yet counted
for blobHandle := range stats.blobs {
blobSize, found := repo.LookupBlobSize(blobHandle.ID, blobHandle.Type)
if !found {
return fmt.Errorf("blob %v not found", blobHandle)
}
stats.TotalSize += uint64(blobSize)
stats.TotalBlobCount++
}
}
if gopts.JSON {
err = json.NewEncoder(os.Stdout).Encode(stats)
if err != nil {
return fmt.Errorf("encoding output: %v", err)
}
return nil
}
if stats.TotalBlobCount > 0 {
Printf(" Total Blob Count: %d\n", stats.TotalBlobCount)
}
if stats.TotalFileCount > 0 {
Printf(" Total File Count: %d\n", stats.TotalFileCount)
}
Printf(" Total Size: %-5s\n", formatBytes(stats.TotalSize))
return nil
}
func statsWalkSnapshot(ctx context.Context, snapshot *restic.Snapshot, repo restic.Repository, stats *statsContainer) error {
if snapshot.Tree == nil {
return fmt.Errorf("snapshot %s has nil tree", snapshot.ID().Str())
}
if countMode == countModeRawData {
// count just the sizes of unique blobs; we don't need to walk the tree
// ourselves in this case, since a nifty function does it for us
return restic.FindUsedBlobs(ctx, repo, *snapshot.Tree, stats.blobs, stats.blobsSeen)
}
err := walker.Walk(ctx, repo, *snapshot.Tree, restic.NewIDSet(), statsWalkTree(repo, stats))
if err != nil {
return fmt.Errorf("walking tree %s: %v", *snapshot.Tree, err)
}
return nil
}
func statsWalkTree(repo restic.Repository, stats *statsContainer) walker.WalkFunc {
return func(npath string, node *restic.Node, nodeErr error) (bool, error) {
if nodeErr != nil {
return true, nodeErr
}
if node == nil {
return true, nil
}
if countMode == countModeUniqueFilesByContents || countMode == countModeBlobsPerFile {
// only count this file if we haven't visited it before
fid := makeFileIDByContents(node)
if _, ok := stats.uniqueFiles[fid]; !ok {
// mark the file as visited
stats.uniqueFiles[fid] = struct{}{}
if countMode == countModeUniqueFilesByContents {
// simply count the size of each unique file (unique by contents only)
stats.TotalSize += node.Size
stats.TotalFileCount++
}
if countMode == countModeBlobsPerFile {
// count the size of each unique blob reference, which is
// by unique file (unique by contents and file path)
for _, blobID := range node.Content {
// ensure we have this file (by path) in our map; in this
// mode, a file is unique by both contents and path
nodePath := filepath.Join(npath, node.Name)
if _, ok := stats.fileBlobs[nodePath]; !ok {
stats.fileBlobs[nodePath] = restic.NewIDSet()
stats.TotalFileCount++
}
if _, ok := stats.fileBlobs[nodePath][blobID]; !ok {
// is always a data blob since we're accessing it via a file's Content array
blobSize, found := repo.LookupBlobSize(blobID, restic.DataBlob)
if !found {
return true, fmt.Errorf("blob %s not found for tree %s", blobID, *node.Subtree)
}
// count the blob's size, then add this blob by this
// file (path) so we don't double-count it
stats.TotalSize += uint64(blobSize)
stats.fileBlobs[nodePath].Insert(blobID)
// this mode also counts total unique blob _references_ per file
stats.TotalBlobCount++
}
}
}
}
}
if countMode == countModeRestoreSize {
// as this is a file in the snapshot, we can simply count its
// size without worrying about uniqueness, since duplicate files
// will still be restored
stats.TotalSize += node.Size
stats.TotalFileCount++
}
return true, nil
}
}
// makeFileIDByContents returns a hash of the blob IDs of the
// node's Content in sequence.
func makeFileIDByContents(node *restic.Node) fileID {
var bb []byte
for _, c := range node.Content {
bb = append(bb, []byte(c[:])...)
}
return sha256.Sum256(bb)
}
func verifyStatsInput(gopts GlobalOptions, args []string) error {
// require a recognized counting mode
switch countMode {
case countModeRestoreSize:
case countModeUniqueFilesByContents:
case countModeBlobsPerFile:
case countModeRawData:
default:
return fmt.Errorf("unknown counting mode: %s (use the -h flag to get a list of supported modes)", countMode)
}
// ensure at most one snapshot was specified
if len(args) > 1 {
return fmt.Errorf("only one snapshot may be specified")
}
// if a snapshot was specified, mark it as the one to scan
if len(args) == 1 {
snapshotIDString = args[0]
}
return nil
}
// statsContainer holds information during a walk of a repository
// to collect information about it, as well as state needed
// for a successful and efficient walk.
type statsContainer struct {
TotalSize uint64 `json:"total_size"`
TotalFileCount uint64 `json:"total_file_count"`
TotalBlobCount uint64 `json:"total_blob_count,omitempty"`
// uniqueFiles marks visited files according to their
// contents (hashed sequence of content blob IDs)
uniqueFiles map[fileID]struct{}
// fileBlobs maps a file name (path) to the set of
// blobs that have been seen as a part of the file
fileBlobs map[string]restic.IDSet
// blobs and blobsSeen are used to count indiviudal
// unique blobs, independent of references to files
blobs, blobsSeen restic.BlobSet
}
// fileID is a 256-bit hash that distinguishes unique files.
type fileID [32]byte
var (
// the mode of counting to perform
countMode string
// the snapshot to scan, as given by the user
snapshotIDString string
// snapshotByHost is the host to filter latest
// snapshot by, if given by user
snapshotByHost string
)
const (
countModeRestoreSize = "restore-size"
countModeUniqueFilesByContents = "files-by-contents"
countModeBlobsPerFile = "blobs-per-file"
countModeRawData = "raw-data"
)

2
cmd/restic/cmd_version.go
View file

@ -16,7 +16,7 @@ and the version of this software.
`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("restic %s compiled with %v on %v/%v\n",
fmt.Printf("restic %s\ncompiled with %v on %v/%v\n",
version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
},
}

12
cmd/restic/exclude.go
View file

@ -185,11 +185,6 @@ func isDirExcludedByFile(dir, tagFilename, header string) bool {
func gatherDevices(items []string) (deviceMap map[string]uint64, err error) {
deviceMap = make(map[string]uint64)
for _, item := range items {
item, err = filepath.Abs(filepath.Clean(item))
if err != nil {
return nil, err
}
fi, err := fs.Lstat(item)
if err != nil {
return nil, err
@ -220,8 +215,6 @@ func rejectByDevice(samples []string) (RejectFunc, error) {
return false
}
item = filepath.Clean(item)
id, err := fs.DeviceID(fi)
if err != nil {
// This should never happen because gatherDevices() would have
@ -229,14 +222,11 @@ func rejectByDevice(samples []string) (RejectFunc, error) {
panic(err)
}
for dir := item; ; dir = filepath.Dir(dir) {
for dir := item; dir != ""; dir = filepath.Dir(dir) {
debug.Log("item %v, test dir %v", item, dir)
allowedID, ok := allowed[dir]
if !ok {
if dir == filepath.Dir(dir) {
break
}
continue
}

31
cmd/restic/excludes Normal file
View file

@ -0,0 +1,31 @@
/boot
/dev
/etc
/home
/lost+found
/mnt
/proc
/root
/run
/sys
/tmp
/usr
/var
/opt/android-sdk
/opt/bullet
/opt/dex2jar
/opt/jameica
/opt/google
/opt/JDownloader
/opt/JDownloaderScripts
/opt/opencascade
/opt/vagrant
/opt/visual-studio-code
/opt/vtk6
/bin
/fonts*
/srv/ftp
/srv/http
/sbin
/lib
/lib64

7
cmd/restic/format.go
View file

@ -3,6 +3,7 @@ package main
import (
"fmt"
"os"
"path/filepath"
"time"
"github.com/restic/restic/internal/restic"
@ -62,9 +63,9 @@ func formatDuration(d time.Duration) string {
return formatSeconds(sec)
}
func formatNode(path string, n *restic.Node, long bool) string {
func formatNode(prefix string, n *restic.Node, long bool) string {
if !long {
return path
return filepath.Join(prefix, n.Name)
}
var mode os.FileMode
@ -90,6 +91,6 @@ func formatNode(path string, n *restic.Node, long bool) string {
return fmt.Sprintf("%s %5d %5d %6d %s %s%s",
mode|n.Mode, n.UID, n.GID, n.Size,
n.ModTime.Format(TimeFormat), path,
n.ModTime.Format(TimeFormat), filepath.Join(prefix, n.Name),
target)
}

64
cmd/restic/global.go
View file

@ -4,6 +4,7 @@ import (
"context"
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"runtime"
@ -17,7 +18,6 @@ import (
"github.com/restic/restic/internal/backend/gs"
"github.com/restic/restic/internal/backend/local"
"github.com/restic/restic/internal/backend/location"
"github.com/restic/restic/internal/backend/rclone"
"github.com/restic/restic/internal/backend/rest"
"github.com/restic/restic/internal/backend/s3"
"github.com/restic/restic/internal/backend/sftp"
@ -29,7 +29,6 @@ import (
"github.com/restic/restic/internal/options"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/textfile"
"github.com/restic/restic/internal/errors"
@ -43,7 +42,6 @@ type GlobalOptions struct {
Repo string
PasswordFile string
Quiet bool
Verbose int
NoLock bool
JSON bool
CacheDir string
@ -60,13 +58,6 @@ type GlobalOptions struct {
stdout io.Writer
stderr io.Writer
// verbosity is set as follows:
// 0 means: don't print any messages except errors, this is used when --quiet is specified
// 1 is the default: print essential messages
// 2 means: print more messages, report minor things, this is used when --verbose is specified
// 3 means: print very detailed debug messages, this is used when --debug is specified
verbosity uint
Options []string
extended options.Options
@ -89,12 +80,11 @@ func init() {
f.StringVarP(&globalOptions.Repo, "repo", "r", os.Getenv("RESTIC_REPOSITORY"), "repository to backup to or restore from (default: $RESTIC_REPOSITORY)")
f.StringVarP(&globalOptions.PasswordFile, "password-file", "p", os.Getenv("RESTIC_PASSWORD_FILE"), "read the repository password from a file (default: $RESTIC_PASSWORD_FILE)")
f.BoolVarP(&globalOptions.Quiet, "quiet", "q", false, "do not output comprehensive progress report")
f.CountVarP(&globalOptions.Verbose, "verbose", "v", "be verbose (specify --verbose multiple times or level `n`)")
f.BoolVar(&globalOptions.NoLock, "no-lock", false, "do not lock the repo, this allows some operations on read-only repos")
f.BoolVarP(&globalOptions.JSON, "json", "", false, "set output mode to JSON for commands that support it")
f.StringVar(&globalOptions.CacheDir, "cache-dir", "", "set the cache directory")
f.BoolVar(&globalOptions.NoCache, "no-cache", false, "do not use a local cache")
f.StringSliceVar(&globalOptions.CACerts, "cacert", nil, "`file` to load root certificates from (default: use system certificates)")
f.StringSliceVar(&globalOptions.CACerts, "cacert", nil, "path to load root certificates from (default: use system certificates)")
f.StringVar(&globalOptions.TLSClientCert, "tls-client-cert", "", "path to a file containing PEM encoded TLS client certificate and private key")
f.BoolVar(&globalOptions.CleanupCache, "cleanup-cache", false, "auto remove old cache directories")
f.IntVar(&globalOptions.LimitUploadKb, "limit-upload", 0, "limits uploads to a maximum rate in KiB/s. (default: unlimited)")
@ -182,9 +172,11 @@ func Printf(format string, args ...interface{}) {
// Verbosef calls Printf to write the message when the verbose flag is set.
func Verbosef(format string, args ...interface{}) {
if globalOptions.verbosity >= 1 {
Printf(format, args...)
if globalOptions.Quiet {
return
}
Printf(format, args...)
}
// PrintProgress wraps fmt.Printf to handle the difference in writing progress
@ -235,8 +227,8 @@ func Exitf(exitcode int, format string, args ...interface{}) {
// resolvePassword determines the password to be used for opening the repository.
func resolvePassword(opts GlobalOptions, env string) (string, error) {
if opts.PasswordFile != "" {
s, err := textfile.Read(opts.PasswordFile)
if os.IsNotExist(errors.Cause(err)) {
s, err := ioutil.ReadFile(opts.PasswordFile)
if os.IsNotExist(err) {
return "", errors.Fatalf("%s does not exist", opts.PasswordFile)
}
return strings.TrimSpace(string(s)), errors.Wrap(err, "Readfile")
@ -293,7 +285,6 @@ func ReadPassword(opts GlobalOptions, prompt string) (string, error) {
password, err = readPasswordTerminal(os.Stdin, os.Stderr, prompt)
} else {
password, err = readPassword(os.Stdin)
Verbosef("read password from stdin\n")
}
if err != nil {
@ -356,11 +347,7 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
}
if stdoutIsTerminal() {
id := s.Config().ID
if len(id) > 8 {
id = id[:8]
}
Verbosef("repository %v opened successfully, password is correct\n", id)
Verbosef("password is correct\n")
}
if opts.NoCache {
@ -391,7 +378,7 @@ func OpenRepository(opts GlobalOptions) (*repository.Repository, error) {
Printf("removing %d old cache dirs from %v\n", len(oldCacheDirs), c.Base)
for _, item := range oldCacheDirs {
dir := filepath.Join(c.Base, item.Name())
dir := filepath.Join(c.Base, item)
err = fs.RemoveAll(dir)
if err != nil {
Warnf("unable to remove %v: %v\n", dir, err)
@ -453,6 +440,18 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
cfg.ProjectID = os.Getenv("GOOGLE_PROJECT_ID")
}
if cfg.JSONKeyPath == "" {
if path := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS"); path != "" {
// Check read access
if _, err := ioutil.ReadFile(path); err != nil {
return nil, errors.Fatalf("Failed to read google credential from file %v: %v", path, err)
}
cfg.JSONKeyPath = path
} else {
return nil, errors.Fatal("No credential file path is set")
}
}
if err := opts.Apply(loc.Scheme, &cfg); err != nil {
return nil, err
}
@ -522,14 +521,6 @@ func parseConfig(loc location.Location, opts options.Options) (interface{}, erro
return nil, err
}
debug.Log("opening rest repository at %#v", cfg)
return cfg, nil
case "rclone":
cfg := loc.Config.(rclone.Config)
if err := opts.Apply(loc.Scheme, &cfg); err != nil {
return nil, err
}
debug.Log("opening rest repository at %#v", cfg)
return cfg, nil
}
@ -562,18 +553,17 @@ func open(s string, gopts GlobalOptions, opts options.Options) (restic.Backend,
}
// wrap the transport so that the throughput via HTTP is limited
lim := limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb)
rt = lim.Transport(rt)
rt = limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb).Transport(rt)
switch loc.Scheme {
case "local":
be, err = local.Open(cfg.(local.Config))
// wrap the backend in a LimitBackend so that the throughput is limited
be = limiter.LimitBackend(be, lim)
be = limiter.LimitBackend(be, limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb))
case "sftp":
be, err = sftp.Open(cfg.(sftp.Config))
// wrap the backend in a LimitBackend so that the throughput is limited
be = limiter.LimitBackend(be, lim)
be = limiter.LimitBackend(be, limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb))
case "s3":
be, err = s3.Open(cfg.(s3.Config), rt)
case "gs":
@ -586,8 +576,6 @@ func open(s string, gopts GlobalOptions, opts options.Options) (restic.Backend,
be, err = b2.Open(globalOptions.ctx, cfg.(b2.Config), rt)
case "rest":
be, err = rest.Open(cfg.(rest.Config), rt)
case "rclone":
be, err = rclone.Open(cfg.(rclone.Config), lim)
default:
return nil, errors.Fatalf("invalid backend: %q", loc.Scheme)
@ -649,8 +637,6 @@ func create(s string, opts options.Options) (restic.Backend, error) {
return b2.Create(globalOptions.ctx, cfg.(b2.Config), rt)
case "rest":
return rest.Create(cfg.(rest.Config), rt)
case "rclone":
return rclone.Open(cfg.(rclone.Config), nil)
}
debug.Log("invalid repository scheme: %v", s)

46
cmd/restic/global_debug.go
View file

@ -1,4 +1,4 @@
// +build debug profile
// +build debug
package main
@ -15,21 +15,17 @@ import (
)
var (
listenProfile string
memProfilePath string
cpuProfilePath string
traceProfilePath string
blockProfilePath string
insecure bool
listenMemoryProfile string
memProfilePath string
cpuProfilePath string
insecure bool
)
func init() {
f := cmdRoot.PersistentFlags()
f.StringVar(&listenProfile, "listen-profile", "", "listen on this `address:port` for memory profiling")
f.StringVar(&listenMemoryProfile, "listen-profile", "", "listen on this `address:port` for memory profiling")
f.StringVar(&memProfilePath, "mem-profile", "", "write memory profile to `dir`")
f.StringVar(&cpuProfilePath, "cpu-profile", "", "write cpu profile to `dir`")
f.StringVar(&traceProfilePath, "trace-profile", "", "write trace to `dir`")
f.StringVar(&blockProfilePath, "block-profile", "", "write block profile to `dir`")
f.BoolVar(&insecure, "insecure-kdf", false, "use insecure KDF settings")
}
@ -40,32 +36,18 @@ func (fakeTestingTB) Logf(msg string, args ...interface{}) {
}
func runDebug() error {
if listenProfile != "" {
fmt.Fprintf(os.Stderr, "running profile HTTP server on %v\n", listenProfile)
if listenMemoryProfile != "" {
fmt.Fprintf(os.Stderr, "running memory profile HTTP server on %v\n", listenMemoryProfile)
go func() {
err := http.ListenAndServe(listenProfile, nil)
err := http.ListenAndServe(listenMemoryProfile, nil)
if err != nil {
fmt.Fprintf(os.Stderr, "profile HTTP server listen failed: %v\n", err)
fmt.Fprintf(os.Stderr, "memory profile listen failed: %v\n", err)
}
}()
}
profilesEnabled := 0
if memProfilePath != "" {
profilesEnabled++
}
if cpuProfilePath != "" {
profilesEnabled++
}
if traceProfilePath != "" {
profilesEnabled++
}
if blockProfilePath != "" {
profilesEnabled++
}
if profilesEnabled > 1 {
return errors.Fatal("only one profile (memory, CPU, trace, or block) may be activated at the same time")
if memProfilePath != "" && cpuProfilePath != "" {
return errors.Fatal("only one profile (memory or CPU) may be activated at the same time")
}
var prof interface {
@ -76,10 +58,6 @@ func runDebug() error {
prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.MemProfile, profile.ProfilePath(memProfilePath))
} else if cpuProfilePath != "" {
prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.CPUProfile, profile.ProfilePath(cpuProfilePath))
} else if traceProfilePath != "" {
prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.TraceProfile, profile.ProfilePath(traceProfilePath))
} else if blockProfilePath != "" {
prof = profile.Start(profile.Quiet, profile.NoShutdownHook, profile.BlockProfile, profile.ProfilePath(blockProfilePath))
}
if prof != nil {

2
cmd/restic/global_release.go
View file

@ -1,4 +1,4 @@
// +build !debug,!profile
// +build !debug
package main

7
cmd/restic/integration_fuse_test.go
View file

@ -1,4 +1,3 @@
// +build !netbsd
// +build !openbsd
// +build !solaris
// +build !windows
@ -172,7 +171,7 @@ func TestMount(t *testing.T) {
rtest.SetupTarTestFixture(t, env.testdata, filepath.Join("testdata", "backup-data.tar.gz"))
// first backup
testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
snapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 1,
"expected one snapshot, got %v", snapshotIDs)
@ -180,7 +179,7 @@ func TestMount(t *testing.T) {
checkSnapshots(t, env.gopts, repo, env.mountpoint, env.repo, snapshotIDs, 2)
// second backup, implicit incremental
testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
snapshotIDs = testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 2,
"expected two snapshots, got %v", snapshotIDs)
@ -189,7 +188,7 @@ func TestMount(t *testing.T) {
// third backup, explicit incremental
bopts := BackupOptions{Parent: snapshotIDs[0].String()}
testRunBackup(t, "", []string{env.testdata}, bopts, env.gopts)
testRunBackup(t, []string{env.testdata}, bopts, env.gopts)
snapshotIDs = testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 3,
"expected three snapshots, got %v", snapshotIDs)

4
cmd/restic/integration_helpers_test.go
View file

@ -11,7 +11,6 @@ import (
"github.com/restic/restic/internal/options"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
)
@ -66,7 +65,7 @@ func isSymlink(fi os.FileInfo) bool {
func sameModTime(fi1, fi2 os.FileInfo) bool {
switch runtime.GOOS {
case "darwin", "freebsd", "openbsd", "netbsd":
case "darwin", "freebsd", "openbsd":
if isSymlink(fi1) && isSymlink(fi2) {
return true
}
@ -190,7 +189,6 @@ func withTestEnvironment(t testing.TB) (env *testEnvironment, cleanup func()) {
}
repository.TestUseLowSecurityKDFParameters(t)
restic.TestDisableCheckPolynomial(t)
tempdir, err := ioutil.TempDir(rtest.TestTempDir, "restic-test-")
rtest.OK(t, err)

349
cmd/restic/integration_test.go
View file

@ -3,7 +3,6 @@ package main
import (
"bufio"
"bytes"
"context"
"crypto/rand"
"encoding/json"
"fmt"
@ -18,14 +17,12 @@ import (
"testing"
"time"
"github.com/restic/restic/internal/debug"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/filter"
"github.com/restic/restic/internal/fs"
"github.com/restic/restic/internal/repository"
"github.com/restic/restic/internal/restic"
rtest "github.com/restic/restic/internal/test"
"github.com/restic/restic/internal/ui/termstatus"
"golang.org/x/sync/errgroup"
)
func parseIDsFromReader(t testing.TB, rd io.Reader) restic.IDs {
@ -47,36 +44,15 @@ func parseIDsFromReader(t testing.TB, rd io.Reader) restic.IDs {
func testRunInit(t testing.TB, opts GlobalOptions) {
repository.TestUseLowSecurityKDFParameters(t)
restic.TestDisableCheckPolynomial(t)
restic.TestSetLockTimeout(t, 0)
rtest.OK(t, runInit(opts, nil))
t.Logf("repository initialized at %v", opts.Repo)
}
func testRunBackup(t testing.TB, dir string, target []string, opts BackupOptions, gopts GlobalOptions) {
ctx, cancel := context.WithCancel(gopts.ctx)
defer cancel()
var wg errgroup.Group
term := termstatus.New(gopts.stdout, gopts.stderr, gopts.Quiet)
wg.Go(func() error { term.Run(ctx); return nil })
gopts.stdout = ioutil.Discard
t.Logf("backing up %v in %v", target, dir)
if dir != "" {
cleanup := fs.TestChdir(t, dir)
defer cleanup()
}
rtest.OK(t, runBackup(opts, gopts, term, target))
cancel()
err := wg.Wait()
if err != nil {
t.Fatal(err)
}
func testRunBackup(t testing.TB, target []string, opts BackupOptions, gopts GlobalOptions) {
t.Logf("backing up %v", target)
rtest.OK(t, runBackup(opts, gopts, target))
}
func testRunList(t testing.TB, tpe string, opts GlobalOptions) restic.IDs {
@ -86,7 +62,7 @@ func testRunList(t testing.TB, tpe string, opts GlobalOptions) restic.IDs {
globalOptions.stdout = os.Stdout
}()
rtest.OK(t, runList(cmdList, opts, []string{tpe}))
rtest.OK(t, runList(opts, []string{tpe}))
return parseIDsFromReader(t, buf)
}
@ -242,7 +218,7 @@ func TestBackup(t *testing.T) {
opts := BackupOptions{}
// first backup
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
snapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 1,
"expected one snapshot, got %v", snapshotIDs)
@ -251,7 +227,7 @@ func TestBackup(t *testing.T) {
stat1 := dirStats(env.repo)
// second backup, implicit incremental
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
snapshotIDs = testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 2,
"expected two snapshots, got %v", snapshotIDs)
@ -265,7 +241,7 @@ func TestBackup(t *testing.T) {
testRunCheck(t, env.gopts)
// third backup, explicit incremental
opts.Parent = snapshotIDs[0].String()
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
snapshotIDs = testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 3,
"expected three snapshots, got %v", snapshotIDs)
@ -309,7 +285,7 @@ func TestBackupNonExistingFile(t *testing.T) {
globalOptions.stderr = os.Stderr
}()
p := filepath.Join(env.testdata, "0", "0", "9")
p := filepath.Join(env.testdata, "0", "0")
dirs := []string{
filepath.Join(p, "0"),
filepath.Join(p, "1"),
@ -319,7 +295,198 @@ func TestBackupNonExistingFile(t *testing.T) {
opts := BackupOptions{}
testRunBackup(t, "", dirs, opts, env.gopts)
testRunBackup(t, dirs, opts, env.gopts)
}
func TestBackupMissingFile1(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(errors.Cause(err)) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
rtest.OK(t, err)
rtest.OK(t, fd.Close())
rtest.SetupTarTestFixture(t, env.testdata, datafile)
testRunInit(t, env.gopts)
globalOptions.stderr = ioutil.Discard
defer func() {
globalOptions.stderr = os.Stderr
}()
ranHook := false
debug.Hook("pipe.walk1", func(context interface{}) {
pathname := context.(string)
if pathname != filepath.Join("testdata", "0", "0", "9") {
return
}
t.Logf("in hook, removing test file testdata/0/0/9/37")
ranHook = true
rtest.OK(t, os.Remove(filepath.Join(env.testdata, "0", "0", "9", "37")))
})
opts := BackupOptions{}
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
rtest.Assert(t, ranHook, "hook did not run")
debug.RemoveHook("pipe.walk1")
}
func TestBackupMissingFile2(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(errors.Cause(err)) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
rtest.OK(t, err)
rtest.OK(t, fd.Close())
rtest.SetupTarTestFixture(t, env.testdata, datafile)
testRunInit(t, env.gopts)
globalOptions.stderr = ioutil.Discard
defer func() {
globalOptions.stderr = os.Stderr
}()
ranHook := false
debug.Hook("pipe.walk2", func(context interface{}) {
pathname := context.(string)
if pathname != filepath.Join("testdata", "0", "0", "9", "37") {
return
}
t.Logf("in hook, removing test file testdata/0/0/9/37")
ranHook = true
rtest.OK(t, os.Remove(filepath.Join(env.testdata, "0", "0", "9", "37")))
})
opts := BackupOptions{}
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
rtest.Assert(t, ranHook, "hook did not run")
debug.RemoveHook("pipe.walk2")
}
func TestBackupChangedFile(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(errors.Cause(err)) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
rtest.OK(t, err)
rtest.OK(t, fd.Close())
rtest.SetupTarTestFixture(t, env.testdata, datafile)
testRunInit(t, env.gopts)
globalOptions.stderr = ioutil.Discard
defer func() {
globalOptions.stderr = os.Stderr
}()
modFile := filepath.Join(env.testdata, "0", "0", "6", "18")
ranHook := false
debug.Hook("archiver.SaveFile", func(context interface{}) {
pathname := context.(string)
if pathname != modFile {
return
}
t.Logf("in hook, modifying test file %v", modFile)
ranHook = true
rtest.OK(t, ioutil.WriteFile(modFile, []byte("modified"), 0600))
})
opts := BackupOptions{}
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
rtest.Assert(t, ranHook, "hook did not run")
debug.RemoveHook("archiver.SaveFile")
}
func TestBackupDirectoryError(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(errors.Cause(err)) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
rtest.OK(t, err)
rtest.OK(t, fd.Close())
rtest.SetupTarTestFixture(t, env.testdata, datafile)
testRunInit(t, env.gopts)
globalOptions.stderr = ioutil.Discard
defer func() {
globalOptions.stderr = os.Stderr
}()
ranHook := false
testdir := filepath.Join(env.testdata, "0", "0", "9")
// install hook that removes the dir right before readdirnames()
debug.Hook("pipe.readdirnames", func(context interface{}) {
path := context.(string)
if path != testdir {
return
}
t.Logf("in hook, removing test file %v", testdir)
ranHook = true
rtest.OK(t, os.RemoveAll(testdir))
})
testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0")}, BackupOptions{}, env.gopts)
testRunCheck(t, env.gopts)
rtest.Assert(t, ranHook, "hook did not run")
debug.RemoveHook("pipe.walk2")
snapshots := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshots) > 0,
"no snapshots found in repo (%v)", datafile)
files := testRunLs(t, env.gopts, snapshots[0].String())
rtest.Assert(t, len(files) > 1, "snapshot is empty")
}
func includes(haystack []string, needle string) bool {
@ -384,33 +551,33 @@ func TestBackupExclude(t *testing.T) {
opts := BackupOptions{}
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
testRunBackup(t, []string{datadir}, opts, env.gopts)
snapshots, snapshotID := lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
files := testRunLs(t, env.gopts, snapshotID)
rtest.Assert(t, includes(files, "/testdata/foo.tar.gz"),
rtest.Assert(t, includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
"expected file %q in first snapshot, but it's not included", "foo.tar.gz")
opts.Excludes = []string{"*.tar.gz"}
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
testRunBackup(t, []string{datadir}, opts, env.gopts)
snapshots, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
files = testRunLs(t, env.gopts, snapshotID)
rtest.Assert(t, !includes(files, "/testdata/foo.tar.gz"),
rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
opts.Excludes = []string{"*.tar.gz", "private/secret"}
testRunBackup(t, filepath.Dir(env.testdata), []string{"testdata"}, opts, env.gopts)
testRunBackup(t, []string{datadir}, opts, env.gopts)
_, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, env.gopts))
files = testRunLs(t, env.gopts, snapshotID)
rtest.Assert(t, !includes(files, "/testdata/foo.tar.gz"),
rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
rtest.Assert(t, !includes(files, "/testdata/private/secret/passwords.txt"),
rtest.Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "private", "secret", "passwords.txt")),
"expected file %q not in first snapshot, but it's included", "passwords.txt")
}
const (
incrementalFirstWrite = 10 * 1042 * 1024
incrementalSecondWrite = 1 * 1042 * 1024
incrementalThirdWrite = 1 * 1042 * 1024
incrementalFirstWrite = 20 * 1042 * 1024
incrementalSecondWrite = 12 * 1042 * 1024
incrementalThirdWrite = 4 * 1042 * 1024
)
func appendRandomData(filename string, bytes uint) error {
@ -448,13 +615,13 @@ func TestIncrementalBackup(t *testing.T) {
opts := BackupOptions{}
testRunBackup(t, "", []string{datadir}, opts, env.gopts)
testRunBackup(t, []string{datadir}, opts, env.gopts)
testRunCheck(t, env.gopts)
stat1 := dirStats(env.repo)
rtest.OK(t, appendRandomData(testfile, incrementalSecondWrite))
testRunBackup(t, "", []string{datadir}, opts, env.gopts)
testRunBackup(t, []string{datadir}, opts, env.gopts)
testRunCheck(t, env.gopts)
stat2 := dirStats(env.repo)
if stat2.size-stat1.size > incrementalFirstWrite {
@ -464,7 +631,7 @@ func TestIncrementalBackup(t *testing.T) {
rtest.OK(t, appendRandomData(testfile, incrementalThirdWrite))
testRunBackup(t, "", []string{datadir}, opts, env.gopts)
testRunBackup(t, []string{datadir}, opts, env.gopts)
testRunCheck(t, env.gopts)
stat3 := dirStats(env.repo)
if stat3.size-stat2.size > incrementalFirstWrite {
@ -483,7 +650,7 @@ func TestBackupTags(t *testing.T) {
opts := BackupOptions{}
testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
newest, _ := testRunSnapshots(t, env.gopts)
rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@ -492,7 +659,7 @@ func TestBackupTags(t *testing.T) {
parent := newest
opts.Tags = []string{"NL"}
testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
newest, _ = testRunSnapshots(t, env.gopts)
rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@ -515,7 +682,7 @@ func TestTag(t *testing.T) {
testRunInit(t, env.gopts)
rtest.SetupTarTestFixture(t, env.testdata, datafile)
testRunBackup(t, "", []string{env.testdata}, BackupOptions{}, env.gopts)
testRunBackup(t, []string{env.testdata}, BackupOptions{}, env.gopts)
testRunCheck(t, env.gopts)
newest, _ := testRunSnapshots(t, env.gopts)
rtest.Assert(t, newest != nil, "expected a new backup, got nil")
@ -691,7 +858,7 @@ func TestRestoreFilter(t *testing.T) {
opts := BackupOptions{}
testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
snapshotID := testRunList(t, "snapshots", env.gopts)[0]
@ -726,12 +893,12 @@ func TestRestore(t *testing.T) {
for i := 0; i < 10; i++ {
p := filepath.Join(env.testdata, fmt.Sprintf("foo/bar/testfile%v", i))
rtest.OK(t, os.MkdirAll(filepath.Dir(p), 0755))
rtest.OK(t, appendRandomData(p, uint(mrand.Intn(2<<21))))
rtest.OK(t, appendRandomData(p, uint(mrand.Intn(5<<21))))
}
opts := BackupOptions{}
testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
// Restore latest without any filters
@ -754,22 +921,12 @@ func TestRestoreLatest(t *testing.T) {
opts := BackupOptions{}
// chdir manually here so we can get the current directory. This is not the
// same as the temp dir returned by ioutil.TempDir() on darwin.
back := fs.TestChdir(t, filepath.Dir(env.testdata))
defer back()
curdir, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
testRunBackup(t, "", []string{filepath.Base(env.testdata)}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
os.Remove(p)
rtest.OK(t, appendRandomData(p, 101))
testRunBackup(t, "", []string{filepath.Base(env.testdata)}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
// Restore latest without any filters
@ -777,18 +934,16 @@ func TestRestoreLatest(t *testing.T) {
rtest.OK(t, testFileSize(filepath.Join(env.base, "restore0", "testdata", "testfile.c"), int64(101)))
// Setup test files in different directories backed up in different snapshots
p1 := filepath.Join(curdir, filepath.FromSlash("p1/testfile.c"))
p1 := filepath.Join(env.testdata, "p1/testfile.c")
rtest.OK(t, os.MkdirAll(filepath.Dir(p1), 0755))
rtest.OK(t, appendRandomData(p1, 102))
testRunBackup(t, "", []string{"p1"}, opts, env.gopts)
testRunBackup(t, []string{filepath.Dir(p1)}, opts, env.gopts)
testRunCheck(t, env.gopts)
p2 := filepath.Join(curdir, filepath.FromSlash("p2/testfile.c"))
p2 := filepath.Join(env.testdata, "p2/testfile.c")
rtest.OK(t, os.MkdirAll(filepath.Dir(p2), 0755))
rtest.OK(t, appendRandomData(p2, 103))
testRunBackup(t, "", []string{"p2"}, opts, env.gopts)
testRunBackup(t, []string{filepath.Dir(p2)}, opts, env.gopts)
testRunCheck(t, env.gopts)
p1rAbs := filepath.Join(env.base, "restore1", "p1/testfile.c")
@ -861,7 +1016,7 @@ func TestRestoreNoMetadataOnIgnoredIntermediateDirs(t *testing.T) {
opts := BackupOptions{}
testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
snapshotID := testRunList(t, "snapshots", env.gopts)[0]
@ -875,6 +1030,9 @@ func TestRestoreNoMetadataOnIgnoredIntermediateDirs(t *testing.T) {
fi, err := os.Stat(f1)
rtest.OK(t, err)
rtest.Assert(t, fi.ModTime() != time.Unix(0, 0),
"meta data of intermediate directory has been restore although it was ignored")
// restore with filter "*", this should restore meta data on everything.
testRunRestoreIncludes(t, env.gopts, filepath.Join(env.base, "restore1"), snapshotID, []string{"*"})
@ -896,7 +1054,7 @@ func TestFind(t *testing.T) {
opts := BackupOptions{}
testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
results := testRunFind(t, false, env.gopts, "unexistingfile")
@ -936,7 +1094,7 @@ func TestFindJSON(t *testing.T) {
opts := BackupOptions{}
testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
testRunCheck(t, env.gopts)
results := testRunFind(t, true, env.gopts, "unexistingfile")
@ -1039,13 +1197,13 @@ func TestPrune(t *testing.T) {
rtest.SetupTarTestFixture(t, env.testdata, datafile)
opts := BackupOptions{}
testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9")}, opts, env.gopts)
testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0")}, opts, env.gopts)
firstSnapshot := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(firstSnapshot) == 1,
"expected one snapshot, got %v", firstSnapshot)
testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9", "2")}, opts, env.gopts)
testRunBackup(t, "", []string{filepath.Join(env.testdata, "0", "0", "9", "3")}, opts, env.gopts)
testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "2")}, opts, env.gopts)
testRunBackup(t, []string{filepath.Join(env.testdata, "0", "0", "3")}, opts, env.gopts)
snapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 3,
@ -1079,7 +1237,7 @@ func TestHardLink(t *testing.T) {
opts := BackupOptions{}
// first backup
testRunBackup(t, filepath.Dir(env.testdata), []string{filepath.Base(env.testdata)}, opts, env.gopts)
testRunBackup(t, []string{env.testdata}, opts, env.gopts)
snapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 1,
"expected one snapshot, got %v", snapshotIDs)
@ -1153,38 +1311,3 @@ func linkEqual(source, dest []string) bool {
return true
}
func TestQuietBackup(t *testing.T) {
env, cleanup := withTestEnvironment(t)
defer cleanup()
datafile := filepath.Join("testdata", "backup-data.tar.gz")
fd, err := os.Open(datafile)
if os.IsNotExist(errors.Cause(err)) {
t.Skipf("unable to find data file %q, skipping", datafile)
return
}
rtest.OK(t, err)
rtest.OK(t, fd.Close())
testRunInit(t, env.gopts)
rtest.SetupTarTestFixture(t, env.testdata, datafile)
opts := BackupOptions{}
env.gopts.Quiet = false
testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
snapshotIDs := testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 1,
"expected one snapshot, got %v", snapshotIDs)
testRunCheck(t, env.gopts)
env.gopts.Quiet = true
testRunBackup(t, "", []string{env.testdata}, opts, env.gopts)
snapshotIDs = testRunList(t, "snapshots", env.gopts)
rtest.Assert(t, len(snapshotIDs) == 2,
"expected two snapshots, got %v", snapshotIDs)
testRunCheck(t, env.gopts)
}

23
cmd/restic/main.go
View file

@ -29,31 +29,14 @@ directories in an encrypted repository stored on different backends.
SilenceUsage: true,
DisableAutoGenTag: true,
PersistentPreRunE: func(c *cobra.Command, args []string) error {
// set verbosity, default is one
globalOptions.verbosity = 1
if globalOptions.Quiet && (globalOptions.Verbose > 1) {
return errors.Fatal("--quiet and --verbose cannot be specified at the same time")
}
switch {
case globalOptions.Verbose >= 2:
globalOptions.verbosity = 3
case globalOptions.Verbose > 0:
globalOptions.verbosity = 2
case globalOptions.Quiet:
globalOptions.verbosity = 0
}
PersistentPreRunE: func(*cobra.Command, []string) error {
// parse extended options
opts, err := options.Parse(globalOptions.Options)
if err != nil {
return err
}
globalOptions.extended = opts
if c.Name() == "version" {
return nil
}
pwd, err := resolvePassword(globalOptions, "RESTIC_PASSWORD")
if err != nil {
fmt.Fprintf(os.Stderr, "Resolving password failed: %v\n", err)
@ -81,7 +64,7 @@ func init() {
func main() {
debug.Log("main %#v", os.Args)
debug.Log("restic %s compiled with %v on %v/%v",
debug.Log("restic %s, compiled with %v on %v/%v",
version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
err := cmdRoot.Execute()

BIN
cmd/restic/testdata/backup-data.tar.gz vendored
View file

Binary file not shown.

3
doc/010_introduction.rst
View file

@ -14,6 +14,3 @@
Introduction
############
Restic is a fast and secure backup program. In the following sections, we will
present typical workflows, starting with installing, preparing a new
repository, and making the first backup.

140
doc/020_installation.rst
View file

@ -17,14 +17,6 @@ Installation
Packages
********
Note that if at any point the package you’re trying to use is outdated, you
always have the option to use an official binary from the restic project.
These are up to date binaries, built in a reproducible and verifiable way, that
you can download and run without having to do additional installation work.
Please see the :ref:`official_binaries` section below for various downloads.
Mac OS X
========
@ -70,110 +62,32 @@ installed from the official repos, e.g. with ``apt-get``:
.. warning:: Please be aware that, at the time of writing, Debian *stable*
has ``restic`` version 0.3.3 which is very old. The *testing* and *unstable*
branches have recent versions of ``restic``.
RHEL & CentOS
=============
restic can be installed via copr repository, for RHEL7/CentOS you can try the following:
Pre-compiled Binary
*******************
.. code-block:: console
$ yum install yum-plugin-copr
$ yum copr enable copart/restic
$ yum install restic
If that doesn't work, you can try adding the repository directly, for CentOS6 use:
.. code-block:: console
$ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-6/copart-restic-epel-6.repo
For CentOS7 use:
.. code-block:: console
$ yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/copart/restic/repo/epel-7/copart-restic-epel-7.repo
Fedora
======
restic can be installed via copr repository.
.. code-block:: console
$ dnf install dnf-plugin-core
$ dnf copr enable copart/restic
$ dnf install restic
Solus
=====
restic can be installed from the official repo of Solus via the ``eopkg`` package manager:
.. code-block:: console
$ eopkg install restic
OpenBSD
=======
On OpenBSD 6.3 and greater, you can install restic using ``pkg_add``:
.. code-block:: console
# pkg_add restic
.. _official_binaries:
Official Binaries
*****************
Stable Releases
===============
You can download the latest stable release versions of restic from the `restic
release page <https://github.com/restic/restic/releases/latest>`__. These builds
are considered stable and releases are made regularly in a controlled manner.
There's both pre-compiled binaries for different platforms as well as the source
code available for download. Just download and run the one matching your system.
Unstable Builds
===============
Another option is to use the latest builds for the master branch, available on
the `restic beta download site
<https://beta.restic.net/?sort=time&order=desc>`__. These too are pre-compiled
and ready to run, and a new version is built every time a push is made to the
master branch.
You can download the latest pre-compiled binary from the `restic release
page <https://github.com/restic/restic/releases/latest>`__.
Windows
=======
On Windows, put the `restic.exe` binary into `%SystemRoot%\\System32` to use restic
On Windows, put the `restic.exe` into `%SystemRoot%\System32` to use restic
in scripts without the need for absolute paths to the binary. This requires
administrator rights.
Admin rights.
Docker Container
****************
We're maintaining a bare docker container with just a few files and the restic
binary, you can get it with `docker pull` like this:
.. code-block:: console
$ docker pull restic/restic
.. note::
| Another docker container which offers more configuration options is
| available as a contribution (Thank you!). You can find it at
| https://github.com/Lobaro/restic-backup-docker
| A docker container is available as a contribution (Thank you!).
| You can find it at https://github.com/Lobaro/restic-backup-docker
From Source
***********
restic is written in the Go programming language and you need at least
Go version 1.9. Building restic may also work with older versions of Go,
Go version 1.8. Building restic may also work with older versions of Go,
but that's not supported. See the `Getting
started <https://golang.org/doc/install>`__ guide of the Go project for
instructions how to install Go.
@ -193,7 +107,7 @@ You can easily cross-compile restic for all supported platforms, just
supply the target OS and platform via the command-line options like this
(for Windows and FreeBSD respectively):
.. code-block:: console
::
$ go run build.go --goos windows --goarch amd64
@ -210,27 +124,35 @@ compiler. Building restic with gccgo may work, but is not supported.
Autocompletion
**************
Restic can write out man pages and bash/zsh compatible autocompletion scripts:
Restic can write out a bash compatible autocompletion script:
.. code-block:: console
$ ./restic generate --help
$ ./restic autocomplete --help
The "autocomplete" command generates a shell autocompletion script.
The "generate" command writes automatically generated files like the man pages
and the auto-completion files for bash and zsh).
NOTE: The current version supports Bash only.
This should work for *nix systems with Bash installed.
By default, the file is written directly to ``/etc/bash_completion.d/``
for convenience, and the command may need superuser rights, e.g.
.. code-block:: console
$ sudo restic autocomplete
Usage:
restic generate [command] [flags]
restic autocomplete [flags]
Flags:
--bash-completion file write bash completion file
-h, --help help for generate
--man directory write man pages to directory
--zsh-completion file write zsh completion file
--completionfile string autocompletion file (default "/etc/bash_completion.d/restic.sh")
Example for using sudo to write a bash completion script directly to the system-wide location:
Global Flags:
--json set output mode to JSON for commands that support it
--no-lock do not lock the repo, this allows some operations on read-only repos
-o, --option key=value set extended option (key=value, can be specified multiple times)
-p, --password-file string read the repository password from a file
-q, --quiet do not output comprehensive progress report
-r, --repo string repository to backup to or restore from (default: $RESTIC_REPOSITORY)
.. code-block:: console
$ sudo ./restic generate --bash-completion /etc/bash_completion.d/restic
writing bash completion file to /etc/bash_completion.d/restic

194
doc/030_preparing_a_new_repo.rst
View file

@ -14,25 +14,21 @@
Preparing a new repository
##########################
The place where your backups will be saved is called a "repository".
This chapter explains how to create ("init") such a repository. The repository
can be stored locally, or on some remote server or service. We'll first cover
using a local repository; the remaining sections of this chapter cover all the
other options. You can skip to the next chapter once you've read the relevant
section here.
The place where your backups will be saved at is called a "repository".
This chapter explains how to create ("init") such a repository.
Local
*****
In order to create a repository at ``/srv/restic-repo``, run the following
In order to create a repository at ``/tmp/backup``, run the following
command and enter the same password twice:
.. code-block:: console
$ restic init --repo /srv/restic-repo
$ restic init --repo /tmp/backup
enter password for new backend:
enter password again:
created restic backend 085b3c76b9 at /srv/restic-repo
created restic backend 085b3c76b9 at /tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
@ -59,10 +55,10 @@ simply be achieved by changing the URL scheme in the ``init`` command:
.. code-block:: console
$ restic -r sftp:user@host:/srv/restic-repo init
$ restic -r sftp:user@host:/tmp/backup init
enter password for new backend:
enter password again:
created restic backend f1c6108821 at sftp:user@host:/srv/restic-repo
created restic backend f1c6108821 at sftp:user@host:/tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
@ -91,7 +87,7 @@ specify the user name in this case):
::
$ restic -r sftp:foo:/srv/restic-repo init
$ restic -r sftp:foo:/tmp/backup init
You can also add an entry with a special host name which does not exist,
just for use with restic, and use the ``Hostname`` option to set the
@ -108,7 +104,7 @@ Then use it in the backend specification:
::
$ restic -r sftp:restic-backup-host:/srv/restic-repo init
$ restic -r sftp:restic-backup-host:/tmp/backup init
Last, if you'd like to use an entirely different program to create the
SFTP connection, you can specify the command to be run with the option
@ -129,8 +125,8 @@ scheme like this:
$ restic -r rest:http://host:8000/
Depending on your REST server setup, you can use HTTPS protocol,
password protection, multiple repositories or any combination of
those features. The TCP/IP port is also configurable. Here
password protection, or multiple repositories. Or any combination of
those features, as you see fit. TCP/IP port is also configurable. Here
are some more examples:
.. code-block:: console
@ -143,10 +139,7 @@ If you use TLS, restic will use the system's CA certificates to verify the
server certificate. When the verification fails, restic refuses to proceed and
exits with an error. If you have your own self-signed certificate, or a custom
CA certificate should be used for verification, you can pass restic the
certificate filename via the ``--cacert`` option. It will then verify that the
server's certificate is contained in the file passed to this option, or signed
by a CA certificate in the file. In this case, the system CA certificates are
not considered at all.
certificate filename via the `--cacert` option.
REST server uses exactly the same directory structure as local backend,
so you should be able to access it both locally and via HTTP, even
@ -167,7 +160,7 @@ while creating the bucket.
$ export AWS_SECRET_ACCESS_KEY=<MY_SECRET_ACCESS_KEY>
You can then easily initialize a repository that uses your Amazon S3 as
a backend. If the bucket does not exist it will be created in the
a backend, if the bucket does not exist yet it will be created in the
default location:
.. code-block:: console
@ -209,7 +202,7 @@ written in Go and compatible with AWS S3 API.
on installation and getting started on Minio Client and Minio Server.
You must first setup the following environment variables with the
credentials of your Minio Server.
credentials of your running Minio Server.
.. code-block:: console
@ -234,7 +227,7 @@ OpenStack Swift
Restic can backup data to an OpenStack Swift container. Because Swift supports
various authentication methods, credentials are passed through environment
variables. In order to help integration with existing OpenStack installations,
the naming convention of those variables follows the official Python Swift client:
the naming convention of those variables follows official python swift client:
.. code-block:: console
@ -265,12 +258,12 @@ the naming convention of those variables follows the official Python Swift clien
$ export OS_AUTH_TOKEN=<MY_AUTH_TOKEN>
Restic should be compatible with an `OpenStack RC file
Restic should be compatible with `OpenStack RC file
<https://docs.openstack.org/user-guide/common/cli-set-environment-variables-using-openstack-rc.html>`__
in most cases.
Once environment variables are set up, a new repository can be created. The
name of the Swift container and optional path can be specified. If
name of swift container and optional path can be specified. If
the container does not exist, it will be created automatically:
.. code-block:: console
@ -282,7 +275,7 @@ the container does not exist, it will be created automatically:
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
The policy of the new container created by restic can be changed using environment variable:
The policy of new container created by restic can be changed using environment variable:
.. code-block:: console
@ -293,24 +286,16 @@ Backblaze B2
************
Restic can backup data to any Backblaze B2 bucket. You need to first setup the
following environment variables with the credentials you can find in the
dashboard in on the "Buckets" page when signed into your B2 account:
following environment variables with the credentials you obtained when signed
into your B2 account:
.. code-block:: console
$ export B2_ACCOUNT_ID=<MY_ACCOUNT_ID>
$ export B2_ACCOUNT_KEY=<MY_SECRET_ACCOUNT_KEY>
You can either specify the so-called "Master Application Key" here (which can
access any bucket at any path) or a dedicated "Application Key" created just
for restic (which may be restricted to a specific bucket and/or path). The
master key consists of a ``B2_ACCOUNT_ID`` and a ``B2_ACCOUNT_KEY``, and each
application key also is a pair of ``B2_ACCOUNT_ID`` and ``B2_ACCOUNT_KEY``. The
ID of an application key is much longer than the ID of the master key.
You can then initialize a repository stored at Backblaze B2. If the
bucket does not exist yet and the credentials you passed to restic have the
privilege to create buckets, it will be created automatically:
You can then easily initialize a repository stored at Backblaze B2. If the
bucket does not exist yet, it will be created:
.. code-block:: console
@ -321,10 +306,8 @@ privilege to create buckets, it will be created automatically:
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
Note that the bucket name must be unique across all of B2.
The number of concurrent connections to the B2 service can be set with the ``-o
b2.connections=10`` switch. By default, at most five parallel connections are
The number of concurrent connections to the B2 service can be set with the `-o
b2.connections=10`. By default, at most five parallel connections are
established.
Microsoft Azure Blob Storage
@ -338,7 +321,7 @@ account name and key as follows:
$ export AZURE_ACCOUNT_NAME=<ACCOUNT_NAME>
$ export AZURE_ACCOUNT_KEY=<SECRET_KEY>
Afterwards you can initialize a repository in a container called ``foo`` in the
Afterwards you can initialize a repository in a container called `foo` in the
root path like this:
.. code-block:: console
@ -351,13 +334,15 @@ root path like this:
[...]
The number of concurrent connections to the Azure Blob Storage service can be set with the
``-o azure.connections=10`` switch. By default, at most five parallel connections are
`-o azure.connections=10`. By default, at most five parallel connections are
established.
Google Cloud Storage
********************
Restic supports Google Cloud Storage as a backend and connects via a `service account`_.
Restic supports Google Cloud Storage as a backend.
Restic connects to Google Cloud Storage via a `service account`_.
For normal restic operation, the service account must have the
``storage.objects.{create,delete,get,list}`` permissions for the bucket. These
@ -377,13 +362,8 @@ key file and the project ID as follows:
$ export GOOGLE_PROJECT_ID=123123123123
$ export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gs-secret-restic-key.json
Restic uses Google's client library to generate `default authentication material`_,
which means if you're running in Google Container Engine or are otherwise
located on an instance with default service accounts then these should work out of
the box.
Once authenticated, you can use the ``gs:`` backend type to create a new
repository in the bucket ``foo`` at the root path:
Then you can use the ``gs:`` backend type to create a new repository in the
bucket `foo` at the root path:
.. code-block:: console
@ -395,117 +375,11 @@ repository in the bucket ``foo`` at the root path:
[...]
The number of concurrent connections to the GCS service can be set with the
``-o gs.connections=10`` switch. By default, at most five parallel connections are
`-o gs.connections=10`. By default, at most five parallel connections are
established.
.. _service account: https://cloud.google.com/storage/docs/authentication#service_accounts
.. _create a service account key: https://cloud.google.com/storage/docs/authentication#generating-a-private-key
.. _default authentication material: https://developers.google.com/identity/protocols/application-default-credentials
Other Services via rclone
*************************
The program `rclone`_ can be used to access many other different services and
store data there. First, you need to install and `configure`_ rclone. The
general backend specification format is ``rclone:<remote>:<path>``, the
``<remote>:<path>`` component will be directly passed to rclone. When you
configure a remote named ``foo``, you can then call restic as follows to
initiate a new repository in the path ``bar`` in the repo:
.. code-block:: console
$ restic -r rclone:foo:bar init
Restic takes care of starting and stopping rclone.
As a more concrete example, suppose you have configured a remote named
``b2prod`` for Backblaze B2 with rclone, with a bucket called ``yggdrasil``.
You can then use rclone to list files in the bucket like this:
.. code-block:: console
$ rclone ls b2prod:yggdrasil
In order to create a new repository in the root directory of the bucket, call
restic like this:
.. code-block:: console
$ restic -r rclone:b2prod:yggdrasil init
If you want to use the path ``foo/bar/baz`` in the bucket instead, pass this to
restic:
.. code-block:: console
$ restic -r rclone:b2prod:yggdrasil/foo/bar/baz init
Listing the files of an empty repository directly with rclone should return a
listing similar to the following:
.. code-block:: console
$ rclone ls b2prod:yggdrasil/foo/bar/baz
155 bar/baz/config
448 bar/baz/keys/4bf9c78049de689d73a56ed0546f83b8416795295cda12ec7fb9465af3900b44
Rclone can be `configured with environment variables`_, so for instance
configuring a bandwidth limit for rclone can be achieved by setting the
``RCLONE_BWLIMIT`` environment variable:
.. code-block:: console
$ export RCLONE_BWLIMIT=1M
For debugging rclone, you can set the environment variable ``RCLONE_VERBOSE=2``.
The rclone backend has two additional options:
* ``-o rclone.program`` specifies the path to rclone, the default value is just ``rclone``
* ``-o rclone.args`` allows setting the arguments passed to rclone, by default this is ``serve restic --stdio --b2-hard-delete --drive-use-trash=false``
The reason for the two last parameters (``--b2-hard-delete`` and
``--drive-use-trash=false``) can be found in the corresponding GitHub `issue #1657`_.
In order to start rclone, restic will build a list of arguments by joining the
following lists (in this order): ``rclone.program``, ``rclone.args`` and as the
last parameter the value that follows the ``rclone:`` prefix of the repository
specification.
So, calling restic like this
.. code-block:: console
$ restic -o rclone.program="/path/to/rclone" \
-o rclone.args="serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose" \
-r rclone:b2:foo/bar
runs rclone as follows:
.. code-block:: console
$ /path/to/rclone serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose b2:foo/bar
Manually setting ``rclone.program`` also allows running a remote instance of
rclone e.g. via SSH on a server, for example:
.. code-block:: console
$ restic -o rclone.program="ssh user@host rclone" -r rclone:b2:foo/bar
The rclone command may also be hard-coded in the SSH configuration or the
user's public key, in this case it may be sufficient to just start the SSH
connection (and it's irrelevant what's passed after ``rclone:`` in the
repository specification):
.. code-block:: console
$ restic -o rclone.program="ssh user@host" -r rclone:x
.. _rclone: https://rclone.org/
.. _configure: https://rclone.org/docs/
.. _configured with environment variables: https://rclone.org/docs/#environment-variables
.. _issue #1657: https://github.com/restic/restic/pull/1657#issuecomment-377707486
Password prompt on Windows
**************************
@ -514,7 +388,7 @@ At the moment, restic only supports the default Windows console
interaction. If you use emulation environments like
`MSYS2 <https://msys2.github.io/>`__ or
`Cygwin <https://www.cygwin.com/>`__, which use terminals like
``Mintty`` or ``rxvt``, you may get a password error.
``Mintty`` or ``rxvt``, you may get a password error:
You can workaround this by using a special tool called ``winpty`` (look
`here <https://sourceforge.net/p/msys2/wiki/Porting/>`__ and
@ -524,5 +398,5 @@ On MSYS2, you can install ``winpty`` as follows:
.. code-block:: console
$ pacman -S winpty
$ winpty restic -r /srv/restic-repo init
$ winpty restic -r /tmp/backup init

174
doc/040_backup.rst
View file

@ -21,87 +21,43 @@ again:
.. code-block:: console
$ restic -r /srv/restic-repo --verbose backup ~/work
open repository
$ restic -r /tmp/backup backup ~/work
enter password for repository:
password is correct
lock repository
load index files
start scan
start backup
scan finished in 1.837s
processed 1.720 GiB in 0:12
Files: 5307 new, 0 changed, 0 unmodified
Dirs: 1867 new, 0 changed, 0 unmodified
Added: 1.700 GiB
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:29] 100.00% 54.732 MiB/s 1.582 GiB / 1.582 GiB 2580 / 2580 items 0 errors ETA 0:00
duration: 0:29, 54.47MiB/s
snapshot 40dc1520 saved
As you can see, restic created a backup of the directory and was pretty
fast! The specific snapshot just created is identified by a sequence of
hexadecimal characters, ``40dc1520`` in this case.
If you don't pass the ``--verbose`` option, restic will print less data. You'll still get a nice live status display. Be aware that the live status shows the processed files and not the transferred data. Transferred volume might be lower (due to deduplication) or higher.
If you run the command again, restic will create another snapshot of
your data, but this time it's even faster. This is de-duplication at
work!
.. code-block:: console
$ restic -r /srv/restic-repo backup --verbose ~/work
open repository
$ restic -r /tmp/backup backup ~/work
enter password for repository:
password is correct
lock repository
load index files
using parent snapshot d875ae93
start scan
start backup
scan finished in 1.881s
processed 1.720 GiB in 0:03
Files: 0 new, 0 changed, 5307 unmodified
Dirs: 0 new, 0 changed, 1867 unmodified
Added: 0 B
using parent snapshot 40dc1520aa6a07b7b3ae561786770a01951245d2367241e71e9485f18ae8228c
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:00] 100.00% 0B/s 1.582 GiB / 1.582 GiB 2580 / 2580 items 0 errors ETA 0:00
duration: 0:00, 6572.38MiB/s
snapshot 79766175 saved
You can even backup individual files in the same repository (not passing
``--verbose`` means less output):
You can even backup individual files in the same repository.
.. code-block:: console
$ restic -r /srv/restic-repo backup ~/work.txt
enter password for repository:
password is correct
snapshot 249d0210 saved
If you're interested in what restic does, pass ``--verbose`` twice (or
``--verbose 2``) to display detailed information about each file and directory
restic encounters:
.. code-block:: console
$ echo 'more data foo bar' >> ~/work.txt
$ restic -r /srv/restic-repo backup --verbose --verbose ~/work.txt
open repository
enter password for repository:
password is correct
lock repository
load index files
using parent snapshot f3f8d56b
start scan
start backup
scan finished in 2.115s
modified /home/user/work.txt, saved in 0.007s (22 B added)
modified /home/user/, saved in 0.008s (0 B added, 378 B metadata)
modified /home/, saved in 0.009s (0 B added, 375 B metadata)
processed 22 B in 0:02
Files: 0 new, 1 changed, 0 unmodified
Dirs: 0 new, 2 changed, 0 unmodified
Data Blobs: 1 new
Tree Blobs: 3 new
Added: 1.116 KiB
snapshot 8dc503fc saved
$ restic -r /tmp/backup backup ~/work.txt
scan [/home/user/work.txt]
scanned 0 directories, 1 files in 0:00
[0:00] 100.00% 0B/s 220B / 220B 1 / 1 items 0 errors ETA 0:00
duration: 0:00, 0.03MiB/s
snapshot 31f7bd63 saved
In fact several hosts may use the same repository to backup directories
and files leading to a greater de-duplication.
@ -119,9 +75,6 @@ Now is a good time to run ``restic check`` to verify that all data
is properly stored in the repository. You should run this command regularly
to make sure the internal structure of the repository is free of errors.
Including and Excluding Files
*****************************
You can exclude folders and files by specifying exclude patterns, currently
the exclude options are:
@ -131,56 +84,33 @@ the exclude options are:
- ``--exclude-if-present`` Specified one or more times to exclude a folders content
if it contains a given file (optionally having a given header)
Let's say we have a file called ``excludes.txt`` with the following content:
Basic example:
::
.. code-block:: console
$ cat exclude
# exclude go-files
*.go
# exclude foo/x/y/z/bar foo/x/bar foo/bar
foo/**/bar
It can be used like this:
.. code-block:: console
$ restic -r /srv/restic-repo backup ~/work --exclude="*.c" --exclude-file=excludes.txt
This instruct restic to exclude files matching the following criteria:
* All files matching ``*.go`` (second line in ``excludes.txt``)
* All files and sub-directories named ``bar`` which reside somewhere below a directory called ``foo`` (fourth line in ``excludes.txt``)
* All files matching ``*.c`` (parameter ``--exclude``)
$ restic -r /tmp/backup backup ~/work --exclude=*.c --exclude-file=exclude
Please see ``restic help backup`` for more specific information about each exclude option.
Patterns use `filepath.Glob <https://golang.org/pkg/path/filepath/#Glob>`__ internally,
see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for
syntax. Patterns are tested against the full path of a file/dir to be saved,
even if restic is passed a relative path to save. Environment-variables in
exclude-files are expanded with `os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__,
so `/home/$USER/foo` will be expanded to `/home/bob/foo` for the user `bob`. To
get a literal dollar sign, write `$$` to the file.
Patterns need to match on complete path components. For example, the pattern ``foo``:
* matches ``/dir1/foo/dir2/file`` and ``/dir/foo``
* does not match ``/dir/foobar`` or ``barfoo``
A trailing ``/`` is ignored, a leading ``/`` anchors the
pattern at the root directory. This means, ``/bin`` matches ``/bin/bash`` but
does not match ``/usr/bin/restic``.
Regular wildcards cannot be used to match over the
directory separator ``/``. For example: ``b*ash`` matches ``/bin/bash`` but does not match
``/bin/ash``.
For this, the special wildcard ``**`` can be used to match arbitrary
sub-directories: The pattern ``foo/**/bar`` matches:
* ``/dir1/foo/dir2/bar/file``
* ``/foo/bar/file``
* ``/tmp/foo/bar``
see `filepath.Match <https://golang.org/pkg/path/filepath/#Match>`__ for syntax.
Patterns are tested against the full path of a file/dir to be saved, not only
against the relative path below the argument given to restic backup.
Patterns need to match on complete path components. (``foo`` matches
``/dir1/foo/dir2/file`` and ``/dir/foo`` but does not match ``/dir/foobar`` or
``barfoo``.) A trailing ``/`` is ignored. A leading ``/`` anchors the
pattern at the root directory. (``/bin`` matches ``/bin/bash`` but does not
match ``/usr/bin/restic``.) Regular wildcards cannot be used to match over the
directory separator ``/``. (``b*ash`` matches ``/bin/bash`` but does not match
``/bin/ash``.) However ``**`` matches arbitrary subdirectories. (``foo/**/bar``
matches ``/dir1/foo/dir2/bar/file``, ``/foo/bar/file`` and ``/tmp/foo/bar``.)
Environment-variables in exclude-files are expanded with
`os.ExpandEnv <https://golang.org/pkg/os/#ExpandEnv>`__.
By specifying the option ``--one-file-system`` you can instruct restic
to only backup files from the file systems the initially specified files
@ -189,15 +119,15 @@ backup ``/sys`` or ``/dev`` on a Linux system:
.. code-block:: console
$ restic -r /srv/restic-repo backup --one-file-system /
$ restic -r /tmp/backup backup --one-file-system /
By using the ``--files-from`` option you can read the files you want to
backup from a file. This is especially useful if a lot of files have to
be backed up that are not in the same folder or are maybe pre-filtered
by other software.
For example maybe you want to backup files which have a name that matches a
certain pattern:
or example maybe you want to backup files that have a certain filename
in them:
.. code-block:: console
@ -207,16 +137,14 @@ You can then use restic to backup the filtered files:
.. code-block:: console
$ restic -r /srv/restic-repo backup --files-from /tmp/files_to_backup
$ restic -r /tmp/backup backup --files-from /tmp/files_to_backup
Incidentally you can also combine ``--files-from`` with the normal files
args:
.. code-block:: console
$ restic -r /srv/restic-repo backup --files-from /tmp/files_to_backup /tmp/some_additional_file
Paths in the listing file can be absolute or relative.
$ restic -r /tmp/backup backup --files-from /tmp/files_to_backup /tmp/some_additional_file
Comparing Snapshots
*******************
@ -226,7 +154,7 @@ and displays a small statistic, just pass the command two snapshot IDs:
.. code-block:: console
$ restic -r /srv/restic-repo diff 5845b002 2ab627a6
$ restic -r /tmp/backup diff 5845b002 2ab627a6
password is correct
comparing snapshot ea657ce5 to 2ab627a6:
@ -273,7 +201,7 @@ this mode of operation, just supply the option ``--stdin`` to the
.. code-block:: console
$ mysqldump [...] | restic -r /srv/restic-repo backup --stdin
$ mysqldump [...] | restic -r /tmp/backup backup --stdin
This creates a new snapshot of the output of ``mysqldump``. You can then
use e.g. the fuse mounting option (see below) to mount the repository
@ -284,7 +212,7 @@ specified with ``--stdin-filename``, e.g. like this:
.. code-block:: console
$ mysqldump [...] | restic -r /srv/restic-repo backup --stdin --stdin-filename production.sql
$ mysqldump [...] | restic -r /tmp/backup backup --stdin --stdin-filename production.sql
Tags for backup
***************
@ -294,23 +222,9 @@ information. Just specify the tags for a snapshot one by one with ``--tag``:
.. code-block:: console
$ restic -r /srv/restic-repo backup --tag projectX --tag foo --tag bar ~/work
$ restic -r /tmp/backup backup --tag projectX --tag foo --tag bar ~/work
[...]
The tags can later be used to keep (or forget) snapshots with the ``forget``
command. The command ``tag`` can be used to modify tags on an existing
snapshot.
Space requirements
******************
Restic currently assumes that your backup repository has sufficient space
for the backup operation you are about to perform. This is a realistic
assumption for many cloud providers, but may not be true when backing up
to local disks.
Should you run out of space during the middle of a backup, there will be
some additional data in the repository, but the snapshot will never be
created as it would only be written at the very (successful) end of
the backup operation. Previous snapshots will still be there and will still
work.

22
doc/045_working_with_repos.rst
View file

@ -22,7 +22,7 @@ Now, you can list all the snapshots stored in the repository:
.. code-block:: console
$ restic -r /srv/restic-repo snapshots
$ restic -r /tmp/backup snapshots
enter password for repository:
ID Date Host Tags Directory
----------------------------------------------------------------------
@ -36,7 +36,7 @@ You can filter the listing by directory path:
.. code-block:: console
$ restic -r /srv/restic-repo snapshots --path="/srv"
$ restic -r /tmp/backup snapshots --path="/srv"
enter password for repository:
ID Date Host Tags Directory
----------------------------------------------------------------------
@ -47,7 +47,7 @@ Or filter by host:
.. code-block:: console
$ restic -r /srv/restic-repo snapshots --host luigi
$ restic -r /tmp/backup snapshots --host luigi
enter password for repository:
ID Date Host Tags Directory
----------------------------------------------------------------------
@ -74,7 +74,7 @@ backup data is consistent and the integrity is unharmed:
.. code-block:: console
$ restic -r /srv/restic-repo check
$ restic -r /tmp/backup check
Load indexes
ciphertext verification failed
@ -83,7 +83,7 @@ yield the same error:
.. code-block:: console
$ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work
$ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work
Load indexes
ciphertext verification failed
@ -93,7 +93,7 @@ data files:
.. code-block:: console
$ restic -r /srv/restic-repo check --read-data
$ restic -r /tmp/backup check --read-data
load indexes
check all packs
check snapshots, trees and blobs
@ -107,9 +107,9 @@ commands check all repository data files over 5 separate invocations:
.. code-block:: console
$ restic -r /srv/restic-repo check --read-data-subset=1/5
$ restic -r /srv/restic-repo check --read-data-subset=2/5
$ restic -r /srv/restic-repo check --read-data-subset=3/5
$ restic -r /srv/restic-repo check --read-data-subset=4/5
$ restic -r /srv/restic-repo check --read-data-subset=5/5
$ restic -r /tmp/backup check --read-data-subset=1/5
$ restic -r /tmp/backup check --read-data-subset=2/5
$ restic -r /tmp/backup check --read-data-subset=3/5
$ restic -r /tmp/backup check --read-data-subset=4/5
$ restic -r /tmp/backup check --read-data-subset=5/5

49
doc/050_restore.rst
View file

@ -23,7 +23,7 @@ command to restore the contents of the latest snapshot to
.. code-block:: console
$ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work
$ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work
enter password for repository:
restoring <Snapshot of [/home/user/work] at 2015-05-08 21:40:19.884408621 +0200 CEST> to /tmp/restore-work
@ -33,7 +33,7 @@ backup for a specific host, path or both.
.. code-block:: console
$ restic -r /srv/restic-repo restore latest --target /tmp/restore-art --path "/home/art" --host luigi
$ restic -r /tmp/backup restore latest --target /tmp/restore-art --path "/home/art" --host luigi
enter password for repository:
restoring <Snapshot of [/home/art] at 2015-05-08 21:45:17.884408621 +0200 CEST> to /tmp/restore-art
@ -42,16 +42,12 @@ files in the snapshot. For example, to restore a single file:
.. code-block:: console
$ restic -r /srv/restic-repo restore 79766175 --target /tmp/restore-work --include /work/foo
$ restic -r /tmp/backup restore 79766175 --target /tmp/restore-work --include /work/foo
enter password for repository:
restoring <Snapshot of [/home/user/work] at 2015-05-08 21:40:19.884408621 +0200 CEST> to /tmp/restore-work
This will restore the file ``foo`` to ``/tmp/restore-work/work/foo``.
You can use the command ``restic ls latest`` or ``restic find foo`` to find the
path to the file within the snapshot. This path you can then pass to
`--include` in verbatim to only restore the single file or directory.
Restore using mount
===================
@ -62,9 +58,9 @@ command to serve the repository with FUSE:
.. code-block:: console
$ mkdir /mnt/restic
$ restic -r /srv/restic-repo mount /mnt/restic
$ restic -r /tmp/backup mount /mnt/restic
enter password for repository:
Now serving /srv/restic-repo at /mnt/restic
Now serving /tmp/backup at /mnt/restic
Don't forget to umount after quitting!
Mounting repositories via FUSE is not possible on OpenBSD, Solaris/illumos
@ -84,37 +80,4 @@ the data directly. This can be achieved by using the `dump` command, like this:
.. code-block:: console
$ restic -r /srv/restic-repo dump latest production.sql | mysql
If you have saved multiple different things into the same repo, the ``latest``
snapshot may not be the right one. For example, consider the following
snapshots in a repo:
.. code-block:: console
$ restic -r /srv/restic-repo snapshots
ID Date Host Tags Directory
----------------------------------------------------------------------
562bfc5e 2018-07-14 20:18:01 mopped /home/user/file1
bbacb625 2018-07-14 20:18:07 mopped /home/other/work
e922c858 2018-07-14 20:18:10 mopped /home/other/work
098db9d5 2018-07-14 20:18:13 mopped /production.sql
b62f46ec 2018-07-14 20:18:16 mopped /home/user/file1
1541acae 2018-07-14 20:18:18 mopped /home/other/work
----------------------------------------------------------------------
Here, restic would resolve ``latest`` to the snapshot ``1541acae``, which does
not contain the file we'd like to print at all (``production.sql``). In this
case, you can pass restic the snapshot ID of the snapshot you like to restore:
.. code-block:: console
$ restic -r /srv/restic-repo dump 098db9d5 production.sql | mysql
Or you can pass restic a path that should be used for selecting the latest
snapshot. The path must match the patch printed in the "Directory" column,
e.g.:
.. code-block:: console
$ restic -r /srv/restic-repo dump --path /production.sql latest production.sql | mysql
$ restic -r /tmp/backup dump latest production.sql | mysql

22
doc/060_forget.rst
View file

@ -23,13 +23,6 @@ data that was referenced by the snapshot from the repository. This can
be automated with the ``--prune`` option of the ``forget`` command,
which runs ``prune`` automatically if snapshots have been removed.
.. Warning::
Pruning snapshots can be a very time-consuming process, taking nearly
as long as backups themselves. During a prune operation, the index is
locked and backups cannot be completed. Performance improvements are
planned for this feature.
It is advisable to run ``restic check`` after pruning, to make sure
you are alerted, should the internal data structures of the repository
be damaged.
@ -42,7 +35,7 @@ repository like this:
.. code-block:: console
$ restic -r /srv/restic-repo snapshots
$ restic -r /tmp/backup snapshots
enter password for repository:
ID Date Host Tags Directory
----------------------------------------------------------------------
@ -57,7 +50,7 @@ command and specify the snapshot ID on the command line:
.. code-block:: console
$ restic -r /srv/restic-repo forget bdbd3439
$ restic -r /tmp/backup forget bdbd3439
enter password for repository:
removed snapshot d3f01f63
@ -65,7 +58,7 @@ Afterwards this snapshot is removed:
.. code-block:: console
$ restic -r /srv/restic-repo snapshots
$ restic -r /tmp/backup snapshots
enter password for repository:
ID Date Host Tags Directory
----------------------------------------------------------------------
@ -80,7 +73,7 @@ command must be run:
.. code-block:: console
$ restic -r /srv/restic-repo prune
$ restic -r /tmp/backup prune
enter password for repository:
counting files in repo
@ -166,13 +159,6 @@ The ``forget`` command accepts the following parameters:
snapshots, only keep the last one for that year.
- ``--keep-tag`` keep all snapshots which have all tags specified by
this option (can be specified multiple times).
- ``--keep-within duration`` keep all snapshots which have been made within
the duration of the latest snapshot. ``duration`` needs to be a number of
years, months, and days, e.g. ``2y5m7d`` will keep all snapshots made in the
two years, five months, and seven days before the latest snapshot.
Multiple policies will be ORed together so as to be as inclusive as possible
for keeping snapshots.
Additionally, you can restrict removing snapshots to those which have a
particular hostname with the ``--hostname`` parameter, or tags with the

10
doc/070_encryption.rst
View file

@ -16,8 +16,8 @@ Encryption
*"The design might not be perfect, but it’s good. Encryption is a first-class feature,
the implementation looks sane and I guess the deduplication trade-off is worth
it. So… I’m going to use restic for my personal backups.*" `Filippo Valsorda`_
the implementation looks sane and I guess the deduplication trade-off is worth it. So… I’m going to use restic for
my personal backups.*" `Filippo Valsorda`_
.. _Filippo Valsorda: https://blog.filippo.io/restic-cryptography/
@ -31,19 +31,19 @@ per repository. In fact, you can use the ``list``, ``add``, ``remove``, and
.. code-block:: console
$ restic -r /srv/restic-repo key list
$ restic -r /tmp/backup key list
enter password for repository:
ID User Host Created
----------------------------------------------------------------------
*eb78040b username kasimir 2015-08-12 13:29:57
$ restic -r /srv/restic-repo key add
$ restic -r /tmp/backup key add
enter password for repository:
enter password for new key:
enter password again:
saved new key as <Key of username@kasimir, created on 2015-08-12 13:35:05.316831933 +0200 CEST>
$ restic -r /srv/restic-repo key list
$ restic -r backup key list
enter password for repository:
ID User Host Created
----------------------------------------------------------------------

39
doc/075_scripting.rst
View file

@ -1,39 +0,0 @@
..
Normally, there are no heading levels assigned to certain characters as the structure is
determined from the succession of headings. However, this convention is used in Python’s
Style Guide for documenting which you may follow:
# with overline, for parts
* for chapters
= for sections
- for subsections
^ for subsubsections
" for paragraphs
#########################
Scripting
#########################
This is a list of how certain tasks may be accomplished when you use
restic via scripts.
Check if a repository is already initialized
********************************************
You may find a need to check if a repository is already initialized,
perhaps to prevent your script from initializing a repository multiple
times. The command ``snapshots`` may be used for this purpose:
.. code-block:: console
$ restic -r /srv/restic-repo snapshots
Fatal: unable to open config file: Stat: stat /srv/restic-repo/config: no such file or directory
Is there a repository at the following location?
/srv/restic-repo
If a repository does not exist, restic will return a non-zero exit code
and print an error message. Note that restic will also return a non-zero
exit code if a different error is encountered (e.g.: incorrect password
to ``snapshots``) and it may print a different error message. If there
are no errors, restic will return a zero exit code and print all the
snapshots.

804
doc/100_references.rst
View file

@ -18,6 +18,804 @@ References
Design
******
.. include:: design.rst
.. include:: cache.rst
.. include:: REST_backend.rst
Terminology
===========
This section introduces terminology used in this document.
*Repository*: All data produced during a backup is sent to and stored in
a repository in a structured form, for example in a file system
hierarchy with several subdirectories. A repository implementation must
be able to fulfill a number of operations, e.g. list the contents.
*Blob*: A Blob combines a number of data bytes with identifying
information like the SHA-256 hash of the data and its length.
*Pack*: A Pack combines one or more Blobs, e.g. in a single file.
*Snapshot*: A Snapshot stands for the state of a file or directory that
has been backed up at some point in time. The state here means the
content and meta data like the name and modification time for the file
or the directory and its contents.
*Storage ID*: A storage ID is the SHA-256 hash of the content stored in
the repository. This ID is required in order to load the file from the
repository.
Repository Format
=================
All data is stored in a restic repository. A repository is able to store
data of several different types, which can later be requested based on
an ID. This so-called "storage ID" is the SHA-256 hash of the content of
a file. All files in a repository are only written once and never
modified afterwards. This allows accessing and even writing to the
repository with multiple clients in parallel. Only the ``prune`` operation
removes data from the repository.
Repositories consist of several directories and a top-level file called
``config``. For all other files stored in the repository, the name for
the file is the lower case hexadecimal representation of the storage ID,
which is the SHA-256 hash of the file's contents. This allows for easy
verification of files for accidental modifications, like disk read
errors, by simply running the program ``sha256sum`` on the file and
comparing its output to the file name. If the prefix of a filename is
unique amongst all the other files in the same directory, the prefix may
be used instead of the complete filename.
Apart from the files stored within the ``keys`` directory, all files are
encrypted with AES-256 in counter mode (CTR). The integrity of the
encrypted data is secured by a Poly1305-AES message authentication code
(sometimes also referred to as a "signature").
In the first 16 bytes of each encrypted file the initialisation vector
(IV) is stored. It is followed by the encrypted data and completed by
the 16 byte MAC. The format is: ``IV || CIPHERTEXT || MAC``. The
complete encryption overhead is 32 bytes. For each file, a new random IV
is selected.
The file ``config`` is encrypted this way and contains a JSON document
like the following:
.. code:: json
{
"version": 1,
"id": "5956a3f67a6230d4a92cefb29529f10196c7d92582ec305fd71ff6d331d6271b",
"chunker_polynomial": "25b468838dcb75"
}
After decryption, restic first checks that the version field contains a
version number that it understands, otherwise it aborts. At the moment,
the version is expected to be 1. The field ``id`` holds a unique ID
which consists of 32 random bytes, encoded in hexadecimal. This uniquely
identifies the repository, regardless if it is accessed via SFTP or
locally. The field ``chunker_polynomial`` contains a parameter that is
used for splitting large files into smaller chunks (see below).
Repository Layout
-----------------
The ``local`` and ``sftp`` backends are implemented using files and
directories stored in a file system. The directory layout is the same
for both backend types.
The basic layout of a repository stored in a ``local`` or ``sftp``
backend is shown here:
::
/tmp/restic-repo
├── config
├── data
│ ├── 21
│ │ └── 2159dd48f8a24f33c307b750592773f8b71ff8d11452132a7b2e2a6a01611be1
│ ├── 32
│ │ └── 32ea976bc30771cebad8285cd99120ac8786f9ffd42141d452458089985043a5
│ ├── 59
│ │ └── 59fe4bcde59bd6222eba87795e35a90d82cd2f138a27b6835032b7b58173a426
│ ├── 73
│ │ └── 73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c
│ [...]
├── index
│ ├── c38f5fb68307c6a3e3aa945d556e325dc38f5fb68307c6a3e3aa945d556e325d
│ └── ca171b1b7394d90d330b265d90f506f9984043b342525f019788f97e745c71fd
├── keys
│ └── b02de829beeb3c01a63e6b25cbd421a98fef144f03b9a02e46eff9e2ca3f0bd7
├── locks
├── snapshots
│ └── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec
└── tmp
A local repository can be initialized with the ``restic init`` command,
e.g.:
.. code-block:: console
$ restic -r /tmp/restic-repo init
The local and sftp backends will auto-detect and accept all layouts described
in the following sections, so that remote repositories mounted locally e.g. via
fuse can be accessed. The layout auto-detection can be overridden by specifying
the option ``-o local.layout=default``, valid values are ``default`` and
``s3legacy``. The option for the sftp backend is named ``sftp.layout``, for the
s3 backend ``s3.layout``.
S3 Legacy Layout
----------------
Unfortunately during development the AWS S3 backend uses slightly different
paths (directory names use singular instead of plural for ``key``,
``lock``, and ``snapshot`` files), and the data files are stored directly below
the ``data`` directory. The S3 Legacy repository layout looks like this:
::
/config
/data
├── 2159dd48f8a24f33c307b750592773f8b71ff8d11452132a7b2e2a6a01611be1
├── 32ea976bc30771cebad8285cd99120ac8786f9ffd42141d452458089985043a5
├── 59fe4bcde59bd6222eba87795e35a90d82cd2f138a27b6835032b7b58173a426
├── 73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c
[...]
/index
├── c38f5fb68307c6a3e3aa945d556e325dc38f5fb68307c6a3e3aa945d556e325d
└── ca171b1b7394d90d330b265d90f506f9984043b342525f019788f97e745c71fd
/key
└── b02de829beeb3c01a63e6b25cbd421a98fef144f03b9a02e46eff9e2ca3f0bd7
/lock
/snapshot
└── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec
The S3 backend understands and accepts both forms, new backends are
always created with the default layout for compatibility reasons.
Pack Format
===========
All files in the repository except Key and Pack files just contain raw
data, stored as ``IV || Ciphertext || MAC``. Pack files may contain one
or more Blobs of data.
A Pack's structure is as follows:
::
EncryptedBlob1 || ... || EncryptedBlobN || EncryptedHeader || Header_Length
At the end of the Pack file is a header, which describes the content.
The header is encrypted and authenticated. ``Header_Length`` is the
length of the encrypted header encoded as a four byte integer in
little-endian encoding. Placing the header at the end of a file allows
writing the blobs in a continuous stream as soon as they are read during
the backup phase. This reduces code complexity and avoids having to
re-write a file once the pack is complete and the content and length of
the header is known.
All the blobs (``EncryptedBlob1``, ``EncryptedBlobN`` etc.) are
authenticated and encrypted independently. This enables repository
reorganisation without having to touch the encrypted Blobs. In addition
it also allows efficient indexing, for only the header needs to be read
in order to find out which Blobs are contained in the Pack. Since the
header is authenticated, authenticity of the header can be checked
without having to read the complete Pack.
After decryption, a Pack's header consists of the following elements:
::
Type_Blob1 || Length(EncryptedBlob1) || Hash(Plaintext_Blob1) ||
[...]
Type_BlobN || Length(EncryptedBlobN) || Hash(Plaintext_Blobn) ||
This is enough to calculate the offsets for all the Blobs in the Pack.
Length is the length of a Blob as a four byte integer in little-endian
format. The type field is a one byte field and labels the content of a
blob according to the following table:
+--------+-----------+
| Type | Meaning |
+========+===========+
| 0 | data |
+--------+-----------+
| 1 | tree |
+--------+-----------+
All other types are invalid, more types may be added in the future.
For reconstructing the index or parsing a pack without an index, first
the last four bytes must be read in order to find the length of the
header. Afterwards, the header can be read and parsed, which yields all
plaintext hashes, types, offsets and lengths of all included blobs.
Indexing
========
Index files contain information about Data and Tree Blobs and the Packs
they are contained in and store this information in the repository. When
the local cached index is not accessible any more, the index files can
be downloaded and used to reconstruct the index. The files are encrypted
and authenticated like Data and Tree Blobs, so the outer structure is
``IV || Ciphertext || MAC`` again. The plaintext consists of a JSON
document like the following:
.. code:: json
{
"supersedes": [
"ed54ae36197f4745ebc4b54d10e0f623eaaaedd03013eb7ae90df881b7781452"
],
"packs": [
{
"id": "73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c",
"blobs": [
{
"id": "3ec79977ef0cf5de7b08cd12b874cd0f62bbaf7f07f3497a5b1bbcc8cb39b1ce",
"type": "data",
"offset": 0,
"length": 25
},{
"id": "9ccb846e60d90d4eb915848add7aa7ea1e4bbabfc60e573db9f7bfb2789afbae",
"type": "tree",
"offset": 38,
"length": 100
},
{
"id": "d3dc577b4ffd38cc4b32122cabf8655a0223ed22edfd93b353dc0c3f2b0fdf66",
"type": "data",
"offset": 150,
"length": 123
}
]
}, [...]
]
}
This JSON document lists Packs and the blobs contained therein. In this
example, the Pack ``73d04e61`` contains two data Blobs and one Tree
blob, the plaintext hashes are listed afterwards.
The field ``supersedes`` lists the storage IDs of index files that have
been replaced with the current index file. This happens when index files
are repacked, for example when old snapshots are removed and Packs are
recombined.
There may be an arbitrary number of index files, containing information
on non-disjoint sets of Packs. The number of packs described in a single
file is chosen so that the file size is kept below 8 MiB.
Keys, Encryption and MAC
========================
All data stored by restic in the repository is encrypted with AES-256 in
counter mode and authenticated using Poly1305-AES. For encrypting new
data first 16 bytes are read from a cryptographically secure
pseudorandom number generator as a random nonce. This is used both as
the IV for counter mode and the nonce for Poly1305. This operation needs
three keys: A 32 byte for AES-256 for encryption, a 16 byte AES key and
a 16 byte key for Poly1305. For details see the original paper `The
Poly1305-AES message-authentication
code <http://cr.yp.to/mac/poly1305-20050329.pdf>`__ by Dan Bernstein.
The data is then encrypted with AES-256 and afterwards a message
authentication code (MAC) is computed over the ciphertext, everything is
then stored as IV \|\| CIPHERTEXT \|\| MAC.
The directory ``keys`` contains key files. These are simple JSON
documents which contain all data that is needed to derive the
repository's master encryption and message authentication keys from a
user's password. The JSON document from the repository can be
pretty-printed for example by using the Python module ``json``
(shortened to increase readability):
::
$ python -mjson.tool /tmp/restic-repo/keys/b02de82*
{
"hostname": "kasimir",
"username": "fd0"
"kdf": "scrypt",
"N": 65536,
"r": 8,
"p": 1,
"created": "2015-01-02T18:10:13.48307196+01:00",
"data": "tGwYeKoM0C4j4/9DFrVEmMGAldvEn/+iKC3te/QE/6ox/V4qz58FUOgMa0Bb1cIJ6asrypCx/Ti/pRXCPHLDkIJbNYd2ybC+fLhFIJVLCvkMS+trdywsUkglUbTbi+7+Ldsul5jpAj9vTZ25ajDc+4FKtWEcCWL5ICAOoTAxnPgT+Lh8ByGQBH6KbdWabqamLzTRWxePFoYuxa7yXgmj9A==",
"salt": "uW4fEI1+IOzj7ED9mVor+yTSJFd68DGlGOeLgJELYsTU5ikhG/83/+jGd4KKAaQdSrsfzrdOhAMftTSih5Ux6w==",
}
When the repository is opened by restic, the user is prompted for the
repository password. This is then used with ``scrypt``, a key derivation
function (KDF), and the supplied parameters (``N``, ``r``, ``p`` and
``salt``) to derive 64 key bytes. The first 32 bytes are used as the
encryption key (for AES-256) and the last 32 bytes are used as the
message authentication key (for Poly1305-AES). These last 32 bytes are
divided into a 16 byte AES key ``k`` followed by 16 bytes of secret key
``r``. The key ``r`` is then masked for use with Poly1305 (see the paper
for details).
Those keys are used to authenticate and decrypt the bytes contained in
the JSON field ``data`` with AES-256 and Poly1305-AES as if they were
any other blob (after removing the Base64 encoding). If the
password is incorrect or the key file has been tampered with, the
computed MAC will not match the last 16 bytes of the data, and restic
exits with an error. Otherwise, the data yields a JSON document
which contains the master encryption and message authentication keys for
this repository (encoded in Base64). The command
``restic cat masterkey`` can be used as follows to decrypt and
pretty-print the master key:
.. code-block:: console
$ restic -r /tmp/restic-repo cat masterkey
{
"mac": {
"k": "evFWd9wWlndL9jc501268g==",
"r": "E9eEDnSJZgqwTOkDtOp+Dw=="
},
"encrypt": "UQCqa0lKZ94PygPxMRqkePTZnHRYh1k1pX2k2lM2v3Q=",
}
All data in the repository is encrypted and authenticated with these
master keys. For encryption, the AES-256 algorithm in Counter mode is
used. For message authentication, Poly1305-AES is used as described
above.
A repository can have several different passwords, with a key file for
each. This way, the password can be changed without having to re-encrypt
all data.
Snapshots
=========
A snapshot represents a directory with all files and sub-directories at
a given point in time. For each backup that is made, a new snapshot is
created. A snapshot is a JSON document that is stored in an encrypted
file below the directory ``snapshots`` in the repository. The filename
is the storage ID. This string is unique and used within restic to
uniquely identify a snapshot.
The command ``restic cat snapshot`` can be used as follows to decrypt
and pretty-print the contents of a snapshot file:
.. code-block:: console
$ restic -r /tmp/restic-repo cat snapshot 251c2e58
enter password for repository:
{
"time": "2015-01-02T18:10:50.895208559+01:00",
"tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf",
"dir": "/tmp/testdata",
"hostname": "kasimir",
"username": "fd0",
"uid": 1000,
"gid": 100,
"tags": [
"NL"
]
}
Here it can be seen that this snapshot represents the contents of the
directory ``/tmp/testdata``. The most important field is ``tree``. When
the meta data (e.g. the tags) of a snapshot change, the snapshot needs
to be re-encrypted and saved. This will change the storage ID, so in
order to relate these seemingly different snapshots, a field
``original`` is introduced which contains the ID of the original
snapshot, e.g. after adding the tag ``DE`` to the snapshot above it
becomes:
.. code-block:: console
$ restic -r /tmp/restic-repo cat snapshot 22a5af1b
enter password for repository:
{
"time": "2015-01-02T18:10:50.895208559+01:00",
"tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf",
"dir": "/tmp/testdata",
"hostname": "kasimir",
"username": "fd0",
"uid": 1000,
"gid": 100,
"tags": [
"NL",
"DE"
],
"original": "251c2e5841355f743f9d4ffd3260bee765acee40a6229857e32b60446991b837"
}
Once introduced, the ``original`` field is not modified when the
snapshot's meta data is changed again.
All content within a restic repository is referenced according to its
SHA-256 hash. Before saving, each file is split into variable sized
Blobs of data. The SHA-256 hashes of all Blobs are saved in an ordered
list which then represents the content of the file.
In order to relate these plaintext hashes to the actual location within
a Pack file , an index is used. If the index is not available, the
header of all data Blobs can be read.
Trees and Data
==============
A snapshot references a tree by the SHA-256 hash of the JSON string
representation of its contents. Trees and data are saved in pack files
in a subdirectory of the directory ``data``.
The command ``restic cat blob`` can be used to inspect the tree
referenced above (piping the output of the command to ``jq .`` so that
the JSON is indented):
.. code-block:: console
$ restic -r /tmp/restic-repo cat blob 2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf | jq .
enter password for repository:
{
"nodes": [
{
"name": "testdata",
"type": "dir",
"mode": 493,
"mtime": "2014-12-22T14:47:59.912418701+01:00",
"atime": "2014-12-06T17:49:21.748468803+01:00",
"ctime": "2014-12-22T14:47:59.912418701+01:00",
"uid": 1000,
"gid": 100,
"user": "fd0",
"inode": 409704562,
"content": null,
"subtree": "b26e315b0988ddcd1cee64c351d13a100fedbc9fdbb144a67d1b765ab280b4dc"
}
]
}
A tree contains a list of entries (in the field ``nodes``) which contain
meta data like a name and timestamps. When the entry references a
directory, the field ``subtree`` contains the plain text ID of another
tree object.
When the command ``restic cat blob`` is used, the plaintext ID is needed
to print a tree. The tree referenced above can be dumped as follows:
.. code-block:: console
$ restic -r /tmp/restic-repo cat blob b26e315b0988ddcd1cee64c351d13a100fedbc9fdbb144a67d1b765ab280b4dc
enter password for repository:
{
"nodes": [
{
"name": "testfile",
"type": "file",
"mode": 420,
"mtime": "2014-12-06T17:50:23.34513538+01:00",
"atime": "2014-12-06T17:50:23.338468713+01:00",
"ctime": "2014-12-06T17:50:23.34513538+01:00",
"uid": 1000,
"gid": 100,
"user": "fd0",
"inode": 416863351,
"size": 1234,
"links": 1,
"content": [
"50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d"
]
},
[...]
]
}
This tree contains a file entry. This time, the ``subtree`` field is not
present and the ``content`` field contains a list with one plain text
SHA-256 hash.
The command ``restic cat blob`` can also be used to extract and decrypt
data given a plaintext ID, e.g. for the data mentioned above:
.. code-block:: console
$ restic -r /tmp/restic-repo cat blob 50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d | sha256sum
enter password for repository:
50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d -
As can be seen from the output of the program ``sha256sum``, the hash
matches the plaintext hash from the map included in the tree above, so
the correct data has been returned.
Locks
=====
The restic repository structure is designed in a way that allows
parallel access of multiple instance of restic and even parallel writes.
However, there are some functions that work more efficient or even
require exclusive access of the repository. In order to implement these
functions, restic processes are required to create a lock on the
repository before doing anything.
Locks come in two types: Exclusive and non-exclusive locks. At most one
process can have an exclusive lock on the repository, and during that
time there must not be any other locks (exclusive and non-exclusive).
There may be multiple non-exclusive locks in parallel.
A lock is a file in the subdir ``locks`` whose filename is the storage
ID of the contents. It is encrypted and authenticated the same way as
other files in the repository and contains the following JSON structure:
.. code:: json
{
"time": "2015-06-27T12:18:51.759239612+02:00",
"exclusive": false,
"hostname": "kasimir",
"username": "fd0",
"pid": 13607,
"uid": 1000,
"gid": 100
}
The field ``exclusive`` defines the type of lock. When a new lock is to
be created, restic checks all locks in the repository. When a lock is
found, it is tested if the lock is stale, which is the case for locks
with timestamps older than 30 minutes. If the lock was created on the
same machine, even for younger locks it is tested whether the process is
still alive by sending a signal to it. If that fails, restic assumes
that the process is dead and considers the lock to be stale.
When a new lock is to be created and no other conflicting locks are
detected, restic creates a new lock, waits, and checks if other locks
appeared in the repository. Depending on the type of the other locks and
the lock to be created, restic either continues or fails.
Backups and Deduplication
=========================
For creating a backup, restic scans the source directory for all files,
sub-directories and other entries. The data from each file is split into
variable length Blobs cut at offsets defined by a sliding window of 64
byte. The implementation uses Rabin Fingerprints for implementing this
Content Defined Chunking (CDC). An irreducible polynomial is selected at
random and saved in the file ``config`` when a repository is
initialized, so that watermark attacks are much harder.
Files smaller than 512 KiB are not split, Blobs are of 512 KiB to 8 MiB
in size. The implementation aims for 1 MiB Blob size on average.
For modified files, only modified Blobs have to be saved in a subsequent
backup. This even works if bytes are inserted or removed at arbitrary
positions within the file.
Threat Model
============
The design goals for restic include being able to securely store backups
in a location that is not completely trusted, e.g. a shared system where
others can potentially access the files or (in the case of the system
administrator) even modify or delete them.
General assumptions:
- The host system a backup is created on is trusted. This is the most
basic requirement, and essential for creating trustworthy backups.
The restic backup program guarantees the following:
- Accessing the unencrypted content of stored files and metadata should
not be possible without a password for the repository. Everything
except the metadata included for informational purposes in the key
files is encrypted and authenticated.
- Modifications (intentional or unintentional) can be detected
automatically on several layers:
1. For all accesses of data stored in the repository it is checked
whether the cryptographic hash of the contents matches the storage
ID (the file's name). This way, modifications (bad RAM, broken
harddisk) can be detected easily.
2. Before decrypting any data, the MAC on the encrypted data is
checked. If there has been a modification, the MAC check will
fail. This step happens even before the data is decrypted, so data
that has been tampered with is not decrypted at all.
However, the restic backup program is not designed to protect against
attackers deleting files at the storage location. There is nothing that
can be done about this. If this needs to be guaranteed, get a secure
location without any access from third parties. If you assume that
attackers have write access to your files at the storage location,
attackers are able to figure out (e.g. based on the timestamps of the
stored files) which files belong to what snapshot. When only these files
are deleted, the particular snapshot vanished and all snapshots
depending on data that has been added in the snapshot cannot be restored
completely. Restic is not designed to detect this attack.
Local Cache
===========
In order to speed up certain operations, restic manages a local cache of data.
This document describes the data structures for the local cache with version 1.
Versions
--------
The cache directory is selected according to the `XDG base dir specification
<http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html>`__.
Each repository has its own cache sub-directory, consting of the repository ID
which is chosen at ``init``. All cache directories for different repos are
independent of each other.
The cache dir for a repo contains a file named ``version``, which contains a
single ASCII integer line that stands for the current version of the cache. If
a lower version number is found the cache is recreated with the current
version. If a higher version number is found the cache is ignored and left as
is.
Snapshots and Indexes
---------------------
Snapshot, Data and Index files are cached in the sub-directories ``snapshots``,
``data`` and ``index``, as read from the repository.
************
REST Backend
************
Restic can interact with HTTP Backend that respects the following REST
API.
The following values are valid for ``{type}``:
* ``data``
* ``keys``
* ``locks``
* ``snapshots``
* ``index``
* ``config``
The API version is selected via the ``Accept`` HTTP header in the request. The
following values are defined:
* ``application/vnd.x.restic.rest.v1+json`` or empty: Select API version 1
* ``application/vnd.x.restic.rest.v2+json``: Select API version 2
The server will respond with the value of the highest version it supports in
the ``Content-Type`` HTTP response header for the HTTP requests which should
return JSON. Any different value for this header means API version 1.
The placeholder ``{path}`` in this document is a path to the repository, so
that multiple different repositories can be accessed. The default path is
``/``.
POST {path}?create=true
=======================
This request is used to initially create a new repository. The server
responds with "200 OK" if the repository structure was created
successfully or already exists, otherwise an error is returned.
DELETE {path}
=============
Deletes the repository on the server side. The server responds with "200
OK" if the repository was successfully removed. If this function is not
implemented the server returns "501 Not Implemented", if this it is
denied by the server it returns "403 Forbidden".
HEAD {path}/config
==================
Returns "200 OK" if the repository has a configuration, an HTTP error
otherwise.
GET {path}/config
=================
Returns the content of the configuration file if the repository has a
configuration, an HTTP error otherwise.
Response format: binary/octet-stream
POST {path}/config
==================
Returns "200 OK" if the configuration of the request body has been
saved, an HTTP error otherwise.
GET {path}/{type}/
==================
API version 1
-------------
Returns a JSON array containing the names of all the blobs stored for a given
type, example:
.. code:: json
[
"245bc4c430d393f74fbe7b13325e30dbde9fb0745e50caad57c446c93d20096b",
"85b420239efa1132c41cea0065452a40ebc20c6f8e0b132a5b2f5848360973ec",
"8e2006bb5931a520f3c7009fe278d1ebb87eb72c3ff92a50c30e90f1b8cf3e60",
"e75c8c407ea31ba399ab4109f28dd18c4c68303d8d86cc275432820c42ce3649"
]
API version 2
-------------
Returns a JSON array containing an object for each file of the given type. The
objects have two keys: ``name`` for the file name, and ``size`` for the size in
bytes.
.. code:: json
[
{
"name": "245bc4c430d393f74fbe7b13325e30dbde9fb0745e50caad57c446c93d20096b",
"size": 2341058
},
{
"name": "85b420239efa1132c41cea0065452a40ebc20c6f8e0b132a5b2f5848360973ec",
"size": 2908900
},
{
"name": "8e2006bb5931a520f3c7009fe278d1ebb87eb72c3ff92a50c30e90f1b8cf3e60",
"size": 3030712
},
{
"name": "e75c8c407ea31ba399ab4109f28dd18c4c68303d8d86cc275432820c42ce3649",
"size": 2804
}
]
HEAD {path}/{type}/{name}
=========================
Returns "200 OK" if the blob with the given name and type is stored in
the repository, "404 not found" otherwise. If the blob exists, the HTTP
header ``Content-Length`` is set to the file size.
GET {path}/{type}/{name}
========================
Returns the content of the blob with the given name and type if it is
stored in the repository, "404 not found" otherwise.
If the request specifies a partial read with a Range header field, then
the status code of the response is 206 instead of 200 and the response
only contains the specified range.
Response format: binary/octet-stream
POST {path}/{type}/{name}
=========================
Saves the content of the request body as a blob with the given name and
type, an HTTP error otherwise.
Request format: binary/octet-stream
DELETE {path}/{type}/{name}
===========================
Returns "200 OK" if the blob with the given name and type has been
deleted from the repository, an HTTP error otherwise.
*****
Talks
*****
The following talks will be or have been given about restic:
- 2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016,
Brussels, Belgium
- 2016-01-29: `restic - Backups mal
richtig <https://media.ccc.de/v/c4.openchaos.2016.01.restic>`__:
Public lecture in German at `CCC Cologne
e.V. <https://koeln.ccc.de>`__ in Cologne, Germany
- 2015-08-23: `A Solution to the Backup
Inconvenience <https://programm.froscon.de/2015/events/1515.html>`__:
Lecture at `FROSCON 2015 <https://www.froscon.de>`__ in Bonn, Germany
- 2015-02-01: `Lightning Talk at FOSDEM
2015 <https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s>`__: A
short introduction (with slightly outdated command line)
- 2015-01-27: `Talk about restic at CCC
Aachen <https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content>`__
(in German)

34
doc/110_talks.rst
View file

@ -1,34 +0,0 @@
..
Normally, there are no heading levels assigned to certain characters as the structure is
determined from the succession of headings. However, this convention is used in Python’s
Style Guide for documenting which you may follow:
# with overline, for parts
* for chapters
= for sections
- for subsections
^ for subsubsections
" for paragraphs
#####
Talks
#####
The following talks will be or have been given about restic:
- 2016-01-31: Lightning Talk at the Go Devroom at FOSDEM 2016,
Brussels, Belgium
- 2016-01-29: `restic - Backups mal
richtig <https://media.ccc.de/v/c4.openchaos.2016.01.restic>`__:
Public lecture in German at `CCC Cologne
e.V. <https://koeln.ccc.de>`__ in Cologne, Germany
- 2015-08-23: `A Solution to the Backup
Inconvenience <https://programm.froscon.de/2015/events/1515.html>`__:
Lecture at `FROSCON 2015 <https://www.froscon.de>`__ in Bonn, Germany
- 2015-02-01: `Lightning Talk at FOSDEM
2015 <https://www.youtube.com/watch?v=oM-MfeflUZ8&t=11m40s>`__: A
short introduction (with slightly outdated command line)
- 2015-01-27: `Talk about restic at CCC
Aachen <https://videoag.fsmpi.rwth-aachen.de/?view=player&lectureid=4442#content>`__
(in German)

145
doc/REST_backend.rst
View file

@ -1,145 +0,0 @@
************
REST Backend
************
Restic can interact with HTTP Backend that respects the following REST
API.
The following values are valid for ``{type}``:
* ``data``
* ``keys``
* ``locks``
* ``snapshots``
* ``index``
* ``config``
The API version is selected via the ``Accept`` HTTP header in the request. The
following values are defined:
* ``application/vnd.x.restic.rest.v1`` or empty: Select API version 1
* ``application/vnd.x.restic.rest.v2``: Select API version 2
The server will respond with the value of the highest version it supports in
the ``Content-Type`` HTTP response header for the HTTP requests which should
return JSON. Any different value for this header means API version 1.
The placeholder ``{path}`` in this document is a path to the repository, so
that multiple different repositories can be accessed. The default path is
``/``. The path must end with a slash.
POST {path}?create=true
=======================
This request is used to initially create a new repository. The server
responds with "200 OK" if the repository structure was created
successfully or already exists, otherwise an error is returned.
DELETE {path}
=============
Deletes the repository on the server side. The server responds with "200
OK" if the repository was successfully removed. If this function is not
implemented the server returns "501 Not Implemented", if this it is
denied by the server it returns "403 Forbidden".
HEAD {path}/config
==================
Returns "200 OK" if the repository has a configuration, an HTTP error
otherwise.
GET {path}/config
=================
Returns the content of the configuration file if the repository has a
configuration, an HTTP error otherwise.
Response format: binary/octet-stream
POST {path}/config
==================
Returns "200 OK" if the configuration of the request body has been
saved, an HTTP error otherwise.
GET {path}/{type}/
==================
API version 1
-------------
Returns a JSON array containing the names of all the blobs stored for a given
type, example:
.. code:: json
[
"245bc4c430d393f74fbe7b13325e30dbde9fb0745e50caad57c446c93d20096b",
"85b420239efa1132c41cea0065452a40ebc20c6f8e0b132a5b2f5848360973ec",
"8e2006bb5931a520f3c7009fe278d1ebb87eb72c3ff92a50c30e90f1b8cf3e60",
"e75c8c407ea31ba399ab4109f28dd18c4c68303d8d86cc275432820c42ce3649"
]
API version 2
-------------
Returns a JSON array containing an object for each file of the given type. The
objects have two keys: ``name`` for the file name, and ``size`` for the size in
bytes.
.. code:: json
[
{
"name": "245bc4c430d393f74fbe7b13325e30dbde9fb0745e50caad57c446c93d20096b",
"size": 2341058
},
{
"name": "85b420239efa1132c41cea0065452a40ebc20c6f8e0b132a5b2f5848360973ec",
"size": 2908900
},
{
"name": "8e2006bb5931a520f3c7009fe278d1ebb87eb72c3ff92a50c30e90f1b8cf3e60",
"size": 3030712
},
{
"name": "e75c8c407ea31ba399ab4109f28dd18c4c68303d8d86cc275432820c42ce3649",
"size": 2804
}
]
HEAD {path}/{type}/{name}
=========================
Returns "200 OK" if the blob with the given name and type is stored in
the repository, "404 not found" otherwise. If the blob exists, the HTTP
header ``Content-Length`` is set to the file size.
GET {path}/{type}/{name}
========================
Returns the content of the blob with the given name and type if it is
stored in the repository, "404 not found" otherwise.
If the request specifies a partial read with a Range header field, then
the status code of the response is 206 instead of 200 and the response
only contains the specified range.
Response format: binary/octet-stream
POST {path}/{type}/{name}
=========================
Saves the content of the request body as a blob with the given name and
type, an HTTP error otherwise.
Request format: binary/octet-stream
DELETE {path}/{type}/{name}
===========================
Returns "200 OK" if the blob with the given name and type has been
deleted from the repository, an HTTP error otherwise.

192
doc/bash-completion.sh
View file

@ -1,6 +1,6 @@
# bash completion for restic -*- shell-script -*-
__restic_debug()
__debug()
{
if [[ -n ${BASH_COMP_DEBUG_FILE} ]]; then
echo "$*" >> "${BASH_COMP_DEBUG_FILE}"
@ -9,13 +9,13 @@ __restic_debug()
# Homebrew on Macs have version 1.3 of bash-completion which doesn't include
# _init_completion. This is a very minimal version of that function.
__restic_init_completion()
__my_init_completion()
{
COMPREPLY=()
_get_comp_words_by_ref "$@" cur prev words cword
}
__restic_index_of_word()
__index_of_word()
{
local w word=$1
shift
@ -27,7 +27,7 @@ __restic_index_of_word()
index=-1
}
__restic_contains_word()
__contains_word()
{
local w word=$1; shift
for w in "$@"; do
@ -36,9 +36,9 @@ __restic_contains_word()
return 1
}
__restic_handle_reply()
__handle_reply()
{
__restic_debug "${FUNCNAME[0]}"
__debug "${FUNCNAME[0]}"
case $cur in
-*)
if [[ $(type -t compopt) = "builtin" ]]; then
@ -62,8 +62,8 @@ __restic_handle_reply()
fi
local index flag
flag="${cur%=*}"
__restic_index_of_word "${flag}" "${flags_with_completion[@]}"
flag="${cur%%=*}"
__index_of_word "${flag}" "${flags_with_completion[@]}"
COMPREPLY=()
if [[ ${index} -ge 0 ]]; then
PREFIX=""
@ -81,7 +81,7 @@ __restic_handle_reply()
# check if we are handling a flag with special work handling
local index
__restic_index_of_word "${prev}" "${flags_with_completion[@]}"
__index_of_word "${prev}" "${flags_with_completion[@]}"
if [[ ${index} -ge 0 ]]; then
${flags_completion[${index}]}
return
@ -114,30 +114,24 @@ __restic_handle_reply()
if declare -F __ltrim_colon_completions >/dev/null; then
__ltrim_colon_completions "$cur"
fi
# If there is only 1 completion and it is a flag with an = it will be completed
# but we don't want a space after the =
if [[ "${#COMPREPLY[@]}" -eq "1" ]] && [[ $(type -t compopt) = "builtin" ]] && [[ "${COMPREPLY[0]}" == --*= ]]; then
compopt -o nospace
fi
}
# The arguments should be in the form "ext1|ext2|extn"
__restic_handle_filename_extension_flag()
__handle_filename_extension_flag()
{
local ext="$1"
_filedir "@(${ext})"
}
__restic_handle_subdirs_in_dir_flag()
__handle_subdirs_in_dir_flag()
{
local dir="$1"
pushd "${dir}" >/dev/null 2>&1 && _filedir -d && popd >/dev/null 2>&1
}
__restic_handle_flag()
__handle_flag()
{
__restic_debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
__debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
# if a command required a flag, and we found it, unset must_have_one_flag()
local flagname=${words[c]}
@ -145,33 +139,30 @@ __restic_handle_flag()
# if the word contained an =
if [[ ${words[c]} == *"="* ]]; then
flagvalue=${flagname#*=} # take in as flagvalue after the =
flagname=${flagname%=*} # strip everything after the =
flagname=${flagname%%=*} # strip everything after the =
flagname="${flagname}=" # but put the = back
fi
__restic_debug "${FUNCNAME[0]}: looking for ${flagname}"
if __restic_contains_word "${flagname}" "${must_have_one_flag[@]}"; then
__debug "${FUNCNAME[0]}: looking for ${flagname}"
if __contains_word "${flagname}" "${must_have_one_flag[@]}"; then
must_have_one_flag=()
fi
# if you set a flag which only applies to this command, don't show subcommands
if __restic_contains_word "${flagname}" "${local_nonpersistent_flags[@]}"; then
if __contains_word "${flagname}" "${local_nonpersistent_flags[@]}"; then
commands=()
fi
# keep flag value with flagname as flaghash
# flaghash variable is an associative array which is only supported in bash > 3.
if [[ -z "${BASH_VERSION}" || "${BASH_VERSINFO[0]}" -gt 3 ]]; then
if [ -n "${flagvalue}" ] ; then
flaghash[${flagname}]=${flagvalue}
elif [ -n "${words[ $((c+1)) ]}" ] ; then
flaghash[${flagname}]=${words[ $((c+1)) ]}
else
flaghash[${flagname}]="true" # pad "true" for bool flag
fi
if [ -n "${flagvalue}" ] ; then
flaghash[${flagname}]=${flagvalue}
elif [ -n "${words[ $((c+1)) ]}" ] ; then
flaghash[${flagname}]=${words[ $((c+1)) ]}
else
flaghash[${flagname}]="true" # pad "true" for bool flag
fi
# skip the argument to a two word flag
if __restic_contains_word "${words[c]}" "${two_word_flags[@]}"; then
if __contains_word "${words[c]}" "${two_word_flags[@]}"; then
c=$((c+1))
# if we are looking for a flags value, don't show commands
if [[ $c -eq $cword ]]; then
@ -183,13 +174,13 @@ __restic_handle_flag()
}
__restic_handle_noun()
__handle_noun()
{
__restic_debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
__debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
if __restic_contains_word "${words[c]}" "${must_have_one_noun[@]}"; then
if __contains_word "${words[c]}" "${must_have_one_noun[@]}"; then
must_have_one_noun=()
elif __restic_contains_word "${words[c]}" "${noun_aliases[@]}"; then
elif __contains_word "${words[c]}" "${noun_aliases[@]}"; then
must_have_one_noun=()
fi
@ -197,42 +188,42 @@ __restic_handle_noun()
c=$((c+1))
}
__restic_handle_command()
__handle_command()
{
__restic_debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
__debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
local next_command
if [[ -n ${last_command} ]]; then
next_command="_${last_command}_${words[c]//:/__}"
else
if [[ $c -eq 0 ]]; then
next_command="_restic_root_command"
next_command="_$(basename "${words[c]//:/__}")"
else
next_command="_${words[c]//:/__}"
fi
fi
c=$((c+1))
__restic_debug "${FUNCNAME[0]}: looking for ${next_command}"
__debug "${FUNCNAME[0]}: looking for ${next_command}"
declare -F "$next_command" >/dev/null && $next_command
}
__restic_handle_word()
__handle_word()
{
if [[ $c -ge $cword ]]; then
__restic_handle_reply
__handle_reply
return
fi
__restic_debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
__debug "${FUNCNAME[0]}: c is $c words[c] is ${words[c]}"
if [[ "${words[c]}" == -* ]]; then
__restic_handle_flag
elif __restic_contains_word "${words[c]}" "${commands[@]}"; then
__restic_handle_command
elif [[ $c -eq 0 ]]; then
__restic_handle_command
__handle_flag
elif __contains_word "${words[c]}" "${commands[@]}"; then
__handle_command
elif [[ $c -eq 0 ]] && __contains_word "$(basename "${words[c]}")" "${commands[@]}"; then
__handle_command
else
__restic_handle_noun
__handle_noun
fi
__restic_handle_word
__handle_word
}
_restic_backup()
@ -297,51 +288,6 @@ _restic_backup()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
noun_aliases=()
}
_restic_cache()
{
last_command="restic_cache"
commands=()
flags=()
two_word_flags=()
local_nonpersistent_flags=()
flags_with_completion=()
flags_completion=()
flags+=("--cleanup")
local_nonpersistent_flags+=("--cleanup")
flags+=("--help")
flags+=("-h")
local_nonpersistent_flags+=("--help")
flags+=("--max-age=")
local_nonpersistent_flags+=("--max-age=")
flags+=("--cacert=")
flags+=("--cache-dir=")
flags+=("--cleanup-cache")
flags+=("--json")
flags+=("--limit-download=")
flags+=("--limit-upload=")
flags+=("--no-cache")
flags+=("--no-lock")
flags+=("--option=")
two_word_flags+=("-o")
flags+=("--password-file=")
two_word_flags+=("-p")
flags+=("--quiet")
flags+=("-q")
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -379,8 +325,6 @@ _restic_cat()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -426,8 +370,6 @@ _restic_check()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -467,8 +409,6 @@ _restic_diff()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -513,8 +453,6 @@ _restic_dump()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -574,8 +512,6 @@ _restic_find()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -611,8 +547,6 @@ _restic_forget()
flags+=("--keep-yearly=")
two_word_flags+=("-y")
local_nonpersistent_flags+=("--keep-yearly=")
flags+=("--keep-within=")
local_nonpersistent_flags+=("--keep-within=")
flags+=("--keep-tag=")
local_nonpersistent_flags+=("--keep-tag=")
flags+=("--host=")
@ -654,8 +588,6 @@ _restic_forget()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -699,8 +631,6 @@ _restic_generate()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -738,8 +668,6 @@ _restic_init()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -760,8 +688,6 @@ _restic_key()
flags+=("--help")
flags+=("-h")
local_nonpersistent_flags+=("--help")
flags+=("--new-password-file=")
local_nonpersistent_flags+=("--new-password-file=")
flags+=("--cacert=")
flags+=("--cache-dir=")
flags+=("--cleanup-cache")
@ -779,8 +705,6 @@ _restic_key()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -818,8 +742,6 @@ _restic_list()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -867,8 +789,6 @@ _restic_ls()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -909,8 +829,6 @@ _restic_migrate()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -963,8 +881,6 @@ _restic_mount()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -1002,8 +918,6 @@ _restic_prune()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -1041,8 +955,6 @@ _restic_rebuild-index()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -1096,8 +1008,6 @@ _restic_restore()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -1147,8 +1057,6 @@ _restic_snapshots()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -1199,8 +1107,6 @@ _restic_tag()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -1240,8 +1146,6 @@ _restic_unlock()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -1279,20 +1183,17 @@ _restic_version()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
noun_aliases=()
}
_restic_root_command()
_restic()
{
last_command="restic"
commands=()
commands+=("backup")
commands+=("cache")
commands+=("cat")
commands+=("check")
commands+=("diff")
@ -1310,7 +1211,6 @@ _restic_root_command()
commands+=("rebuild-index")
commands+=("restore")
commands+=("snapshots")
commands+=("stats")
commands+=("tag")
commands+=("unlock")
commands+=("version")
@ -1341,8 +1241,6 @@ _restic_root_command()
flags+=("--repo=")
two_word_flags+=("-r")
flags+=("--tls-client-cert=")
flags+=("--verbose")
flags+=("-v")
must_have_one_flag=()
must_have_one_noun=()
@ -1356,7 +1254,7 @@ __start_restic()
if declare -F _init_completion >/dev/null 2>&1; then
_init_completion -s || return
else
__restic_init_completion -n "=" || return
__my_init_completion -n "=" || return
fi
local c=0
@ -1371,7 +1269,7 @@ __start_restic()
local last_command
local nouns=()
__restic_handle_word
__handle_word
}
if [[ $(type -t compopt) = "builtin" ]]; then

10
doc/cache.rst
View file

@ -1,12 +1,11 @@
***********
Local Cache
***********
===========
In order to speed up certain operations, restic manages a local cache of data.
This document describes the data structures for the local cache with version 1.
Versions
========
--------
The cache directory is selected according to the `XDG base dir specification
<http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html>`__.
@ -21,16 +20,17 @@ version. If a higher version number is found the cache is ignored and left as
is.
Snapshots, Data and Indexes
===========================
---------------------------
Snapshot, Data and Index files are cached in the sub-directories ``snapshots``,
``data`` and ``index``, as read from the repository.
Expiry
======
------
Whenever a cache directory for a repo is used, that directory's modification
timestamp is updated to the current time. By looking at the modification
timestamps of the repo cache directories it is easy to decide which directories
are old and haven't been used in a long time. Those are probably stale and can
be removed.

2
doc/conf.py
View file

@ -35,7 +35,7 @@ master_doc = 'index'
# General information about the project.
project = 'restic'
copyright = '2018, restic authors'
copyright = '2017, restic authors'
author = 'fd0'
# The version info for the project you're documenting, acts as replacement for

608
doc/design.rst
View file

@ -1,608 +0,0 @@
Terminology
===========
This section introduces terminology used in this document.
*Repository*: All data produced during a backup is sent to and stored in
a repository in a structured form, for example in a file system
hierarchy with several subdirectories. A repository implementation must
be able to fulfill a number of operations, e.g. list the contents.
*Blob*: A Blob combines a number of data bytes with identifying
information like the SHA-256 hash of the data and its length.
*Pack*: A Pack combines one or more Blobs, e.g. in a single file.
*Snapshot*: A Snapshot stands for the state of a file or directory that
has been backed up at some point in time. The state here means the
content and meta data like the name and modification time for the file
or the directory and its contents.
*Storage ID*: A storage ID is the SHA-256 hash of the content stored in
the repository. This ID is required in order to load the file from the
repository.
Repository Format
=================
All data is stored in a restic repository. A repository is able to store
data of several different types, which can later be requested based on
an ID. This so-called "storage ID" is the SHA-256 hash of the content of
a file. All files in a repository are only written once and never
modified afterwards. This allows accessing and even writing to the
repository with multiple clients in parallel. Only the ``prune`` operation
removes data from the repository.
Repositories consist of several directories and a top-level file called
``config``. For all other files stored in the repository, the name for
the file is the lower case hexadecimal representation of the storage ID,
which is the SHA-256 hash of the file's contents. This allows for easy
verification of files for accidental modifications, like disk read
errors, by simply running the program ``sha256sum`` on the file and
comparing its output to the file name. If the prefix of a filename is
unique amongst all the other files in the same directory, the prefix may
be used instead of the complete filename.
Apart from the files stored within the ``keys`` directory, all files are
encrypted with AES-256 in counter mode (CTR). The integrity of the
encrypted data is secured by a Poly1305-AES message authentication code
(sometimes also referred to as a "signature").
In the first 16 bytes of each encrypted file the initialisation vector
(IV) is stored. It is followed by the encrypted data and completed by
the 16 byte MAC. The format is: ``IV || CIPHERTEXT || MAC``. The
complete encryption overhead is 32 bytes. For each file, a new random IV
is selected.
The file ``config`` is encrypted this way and contains a JSON document
like the following:
.. code:: json
{
"version": 1,
"id": "5956a3f67a6230d4a92cefb29529f10196c7d92582ec305fd71ff6d331d6271b",
"chunker_polynomial": "25b468838dcb75"
}
After decryption, restic first checks that the version field contains a
version number that it understands, otherwise it aborts. At the moment,
the version is expected to be 1. The field ``id`` holds a unique ID
which consists of 32 random bytes, encoded in hexadecimal. This uniquely
identifies the repository, regardless if it is accessed via SFTP or
locally. The field ``chunker_polynomial`` contains a parameter that is
used for splitting large files into smaller chunks (see below).
Repository Layout
-----------------
The ``local`` and ``sftp`` backends are implemented using files and
directories stored in a file system. The directory layout is the same
for both backend types.
The basic layout of a repository stored in a ``local`` or ``sftp``
backend is shown here:
::
/tmp/restic-repo
├── config
├── data
│ ├── 21
│ │ └── 2159dd48f8a24f33c307b750592773f8b71ff8d11452132a7b2e2a6a01611be1
│ ├── 32
│ │ └── 32ea976bc30771cebad8285cd99120ac8786f9ffd42141d452458089985043a5
│ ├── 59
│ │ └── 59fe4bcde59bd6222eba87795e35a90d82cd2f138a27b6835032b7b58173a426
│ ├── 73
│ │ └── 73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c
│ [...]
├── index
│ ├── c38f5fb68307c6a3e3aa945d556e325dc38f5fb68307c6a3e3aa945d556e325d
│ └── ca171b1b7394d90d330b265d90f506f9984043b342525f019788f97e745c71fd
├── keys
│ └── b02de829beeb3c01a63e6b25cbd421a98fef144f03b9a02e46eff9e2ca3f0bd7
├── locks
├── snapshots
│ └── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec
└── tmp
A local repository can be initialized with the ``restic init`` command,
e.g.:
.. code-block:: console
$ restic -r /tmp/restic-repo init
The local and sftp backends will auto-detect and accept all layouts described
in the following sections, so that remote repositories mounted locally e.g. via
fuse can be accessed. The layout auto-detection can be overridden by specifying
the option ``-o local.layout=default``, valid values are ``default`` and
``s3legacy``. The option for the sftp backend is named ``sftp.layout``, for the
s3 backend ``s3.layout``.
S3 Legacy Layout
----------------
Unfortunately during development the AWS S3 backend uses slightly different
paths (directory names use singular instead of plural for ``key``,
``lock``, and ``snapshot`` files), and the data files are stored directly below
the ``data`` directory. The S3 Legacy repository layout looks like this:
::
/config
/data
├── 2159dd48f8a24f33c307b750592773f8b71ff8d11452132a7b2e2a6a01611be1
├── 32ea976bc30771cebad8285cd99120ac8786f9ffd42141d452458089985043a5
├── 59fe4bcde59bd6222eba87795e35a90d82cd2f138a27b6835032b7b58173a426
├── 73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c
[...]
/index
├── c38f5fb68307c6a3e3aa945d556e325dc38f5fb68307c6a3e3aa945d556e325d
└── ca171b1b7394d90d330b265d90f506f9984043b342525f019788f97e745c71fd
/key
└── b02de829beeb3c01a63e6b25cbd421a98fef144f03b9a02e46eff9e2ca3f0bd7
/lock
/snapshot
└── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec
The S3 backend understands and accepts both forms, new backends are
always created with the default layout for compatibility reasons.
Pack Format
===========
All files in the repository except Key and Pack files just contain raw
data, stored as ``IV || Ciphertext || MAC``. Pack files may contain one
or more Blobs of data.
A Pack's structure is as follows:
::
EncryptedBlob1 || ... || EncryptedBlobN || EncryptedHeader || Header_Length
At the end of the Pack file is a header, which describes the content.
The header is encrypted and authenticated. ``Header_Length`` is the
length of the encrypted header encoded as a four byte integer in
little-endian encoding. Placing the header at the end of a file allows
writing the blobs in a continuous stream as soon as they are read during
the backup phase. This reduces code complexity and avoids having to
re-write a file once the pack is complete and the content and length of
the header is known.
All the blobs (``EncryptedBlob1``, ``EncryptedBlobN`` etc.) are
authenticated and encrypted independently. This enables repository
reorganisation without having to touch the encrypted Blobs. In addition
it also allows efficient indexing, for only the header needs to be read
in order to find out which Blobs are contained in the Pack. Since the
header is authenticated, authenticity of the header can be checked
without having to read the complete Pack.
After decryption, a Pack's header consists of the following elements:
::
Type_Blob1 || Length(EncryptedBlob1) || Hash(Plaintext_Blob1) ||
[...]
Type_BlobN || Length(EncryptedBlobN) || Hash(Plaintext_Blobn) ||
This is enough to calculate the offsets for all the Blobs in the Pack.
Length is the length of a Blob as a four byte integer in little-endian
format. The type field is a one byte field and labels the content of a
blob according to the following table:
+--------+-----------+
| Type | Meaning |
+========+===========+
| 0 | data |
+--------+-----------+
| 1 | tree |
+--------+-----------+
All other types are invalid, more types may be added in the future.
For reconstructing the index or parsing a pack without an index, first
the last four bytes must be read in order to find the length of the
header. Afterwards, the header can be read and parsed, which yields all
plaintext hashes, types, offsets and lengths of all included blobs.
Indexing
========
Index files contain information about Data and Tree Blobs and the Packs
they are contained in and store this information in the repository. When
the local cached index is not accessible any more, the index files can
be downloaded and used to reconstruct the index. The files are encrypted
and authenticated like Data and Tree Blobs, so the outer structure is
``IV || Ciphertext || MAC`` again. The plaintext consists of a JSON
document like the following:
.. code:: json
{
"supersedes": [
"ed54ae36197f4745ebc4b54d10e0f623eaaaedd03013eb7ae90df881b7781452"
],
"packs": [
{
"id": "73d04e6125cf3c28a299cc2f3cca3b78ceac396e4fcf9575e34536b26782413c",
"blobs": [
{
"id": "3ec79977ef0cf5de7b08cd12b874cd0f62bbaf7f07f3497a5b1bbcc8cb39b1ce",
"type": "data",
"offset": 0,
"length": 25
},{
"id": "9ccb846e60d90d4eb915848add7aa7ea1e4bbabfc60e573db9f7bfb2789afbae",
"type": "tree",
"offset": 38,
"length": 100
},
{
"id": "d3dc577b4ffd38cc4b32122cabf8655a0223ed22edfd93b353dc0c3f2b0fdf66",
"type": "data",
"offset": 150,
"length": 123
}
]
}, [...]
]
}
This JSON document lists Packs and the blobs contained therein. In this
example, the Pack ``73d04e61`` contains two data Blobs and one Tree
blob, the plaintext hashes are listed afterwards.
The field ``supersedes`` lists the storage IDs of index files that have
been replaced with the current index file. This happens when index files
are repacked, for example when old snapshots are removed and Packs are
recombined.
There may be an arbitrary number of index files, containing information
on non-disjoint sets of Packs. The number of packs described in a single
file is chosen so that the file size is kept below 8 MiB.
Keys, Encryption and MAC
========================
All data stored by restic in the repository is encrypted with AES-256 in
counter mode and authenticated using Poly1305-AES. For encrypting new
data first 16 bytes are read from a cryptographically secure
pseudorandom number generator as a random nonce. This is used both as
the IV for counter mode and the nonce for Poly1305. This operation needs
three keys: A 32 byte for AES-256 for encryption, a 16 byte AES key and
a 16 byte key for Poly1305. For details see the original paper `The
Poly1305-AES message-authentication
code <http://cr.yp.to/mac/poly1305-20050329.pdf>`__ by Dan Bernstein.
The data is then encrypted with AES-256 and afterwards a message
authentication code (MAC) is computed over the ciphertext, everything is
then stored as IV \|\| CIPHERTEXT \|\| MAC.
The directory ``keys`` contains key files. These are simple JSON
documents which contain all data that is needed to derive the
repository's master encryption and message authentication keys from a
user's password. The JSON document from the repository can be
pretty-printed for example by using the Python module ``json``
(shortened to increase readability):
::
$ python -mjson.tool /tmp/restic-repo/keys/b02de82*
{
"hostname": "kasimir",
"username": "fd0"
"kdf": "scrypt",
"N": 65536,
"r": 8,
"p": 1,
"created": "2015-01-02T18:10:13.48307196+01:00",
"data": "tGwYeKoM0C4j4/9DFrVEmMGAldvEn/+iKC3te/QE/6ox/V4qz58FUOgMa0Bb1cIJ6asrypCx/Ti/pRXCPHLDkIJbNYd2ybC+fLhFIJVLCvkMS+trdywsUkglUbTbi+7+Ldsul5jpAj9vTZ25ajDc+4FKtWEcCWL5ICAOoTAxnPgT+Lh8ByGQBH6KbdWabqamLzTRWxePFoYuxa7yXgmj9A==",
"salt": "uW4fEI1+IOzj7ED9mVor+yTSJFd68DGlGOeLgJELYsTU5ikhG/83/+jGd4KKAaQdSrsfzrdOhAMftTSih5Ux6w==",
}
When the repository is opened by restic, the user is prompted for the
repository password. This is then used with ``scrypt``, a key derivation
function (KDF), and the supplied parameters (``N``, ``r``, ``p`` and
``salt``) to derive 64 key bytes. The first 32 bytes are used as the
encryption key (for AES-256) and the last 32 bytes are used as the
message authentication key (for Poly1305-AES). These last 32 bytes are
divided into a 16 byte AES key ``k`` followed by 16 bytes of secret key
``r``. The key ``r`` is then masked for use with Poly1305 (see the paper
for details).
Those keys are used to authenticate and decrypt the bytes contained in
the JSON field ``data`` with AES-256 and Poly1305-AES as if they were
any other blob (after removing the Base64 encoding). If the
password is incorrect or the key file has been tampered with, the
computed MAC will not match the last 16 bytes of the data, and restic
exits with an error. Otherwise, the data yields a JSON document
which contains the master encryption and message authentication keys for
this repository (encoded in Base64). The command
``restic cat masterkey`` can be used as follows to decrypt and
pretty-print the master key:
.. code-block:: console
$ restic -r /tmp/restic-repo cat masterkey
{
"mac": {
"k": "evFWd9wWlndL9jc501268g==",
"r": "E9eEDnSJZgqwTOkDtOp+Dw=="
},
"encrypt": "UQCqa0lKZ94PygPxMRqkePTZnHRYh1k1pX2k2lM2v3Q=",
}
All data in the repository is encrypted and authenticated with these
master keys. For encryption, the AES-256 algorithm in Counter mode is
used. For message authentication, Poly1305-AES is used as described
above.
A repository can have several different passwords, with a key file for
each. This way, the password can be changed without having to re-encrypt
all data.
Snapshots
=========
A snapshot represents a directory with all files and sub-directories at
a given point in time. For each backup that is made, a new snapshot is
created. A snapshot is a JSON document that is stored in an encrypted
file below the directory ``snapshots`` in the repository. The filename
is the storage ID. This string is unique and used within restic to
uniquely identify a snapshot.
The command ``restic cat snapshot`` can be used as follows to decrypt
and pretty-print the contents of a snapshot file:
.. code-block:: console
$ restic -r /tmp/restic-repo cat snapshot 251c2e58
enter password for repository:
{
"time": "2015-01-02T18:10:50.895208559+01:00",
"tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf",
"dir": "/tmp/testdata",
"hostname": "kasimir",
"username": "fd0",
"uid": 1000,
"gid": 100,
"tags": [
"NL"
]
}
Here it can be seen that this snapshot represents the contents of the
directory ``/tmp/testdata``. The most important field is ``tree``. When
the meta data (e.g. the tags) of a snapshot change, the snapshot needs
to be re-encrypted and saved. This will change the storage ID, so in
order to relate these seemingly different snapshots, a field
``original`` is introduced which contains the ID of the original
snapshot, e.g. after adding the tag ``DE`` to the snapshot above it
becomes:
.. code-block:: console
$ restic -r /tmp/restic-repo cat snapshot 22a5af1b
enter password for repository:
{
"time": "2015-01-02T18:10:50.895208559+01:00",
"tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf",
"dir": "/tmp/testdata",
"hostname": "kasimir",
"username": "fd0",
"uid": 1000,
"gid": 100,
"tags": [
"NL",
"DE"
],
"original": "251c2e5841355f743f9d4ffd3260bee765acee40a6229857e32b60446991b837"
}
Once introduced, the ``original`` field is not modified when the
snapshot's meta data is changed again.
All content within a restic repository is referenced according to its
SHA-256 hash. Before saving, each file is split into variable sized
Blobs of data. The SHA-256 hashes of all Blobs are saved in an ordered
list which then represents the content of the file.
In order to relate these plaintext hashes to the actual location within
a Pack file , an index is used. If the index is not available, the
header of all data Blobs can be read.
Trees and Data
==============
A snapshot references a tree by the SHA-256 hash of the JSON string
representation of its contents. Trees and data are saved in pack files
in a subdirectory of the directory ``data``.
The command ``restic cat blob`` can be used to inspect the tree
referenced above (piping the output of the command to ``jq .`` so that
the JSON is indented):
.. code-block:: console
$ restic -r /tmp/restic-repo cat blob 2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf | jq .
enter password for repository:
{
"nodes": [
{
"name": "testdata",
"type": "dir",
"mode": 493,
"mtime": "2014-12-22T14:47:59.912418701+01:00",
"atime": "2014-12-06T17:49:21.748468803+01:00",
"ctime": "2014-12-22T14:47:59.912418701+01:00",
"uid": 1000,
"gid": 100,
"user": "fd0",
"inode": 409704562,
"content": null,
"subtree": "b26e315b0988ddcd1cee64c351d13a100fedbc9fdbb144a67d1b765ab280b4dc"
}
]
}
A tree contains a list of entries (in the field ``nodes``) which contain
meta data like a name and timestamps. When the entry references a
directory, the field ``subtree`` contains the plain text ID of another
tree object.
When the command ``restic cat blob`` is used, the plaintext ID is needed
to print a tree. The tree referenced above can be dumped as follows:
.. code-block:: console
$ restic -r /tmp/restic-repo cat blob b26e315b0988ddcd1cee64c351d13a100fedbc9fdbb144a67d1b765ab280b4dc
enter password for repository:
{
"nodes": [
{
"name": "testfile",
"type": "file",
"mode": 420,
"mtime": "2014-12-06T17:50:23.34513538+01:00",
"atime": "2014-12-06T17:50:23.338468713+01:00",
"ctime": "2014-12-06T17:50:23.34513538+01:00",
"uid": 1000,
"gid": 100,
"user": "fd0",
"inode": 416863351,
"size": 1234,
"links": 1,
"content": [
"50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d"
]
},
[...]
]
}
This tree contains a file entry. This time, the ``subtree`` field is not
present and the ``content`` field contains a list with one plain text
SHA-256 hash.
The command ``restic cat blob`` can also be used to extract and decrypt
data given a plaintext ID, e.g. for the data mentioned above:
.. code-block:: console
$ restic -r /tmp/restic-repo cat blob 50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d | sha256sum
enter password for repository:
50f77b3b4291e8411a027b9f9b9e64658181cc676ce6ba9958b95f268cb1109d -
As can be seen from the output of the program ``sha256sum``, the hash
matches the plaintext hash from the map included in the tree above, so
the correct data has been returned.
Locks
=====
The restic repository structure is designed in a way that allows
parallel access of multiple instance of restic and even parallel writes.
However, there are some functions that work more efficient or even
require exclusive access of the repository. In order to implement these
functions, restic processes are required to create a lock on the
repository before doing anything.
Locks come in two types: Exclusive and non-exclusive locks. At most one
process can have an exclusive lock on the repository, and during that
time there must not be any other locks (exclusive and non-exclusive).
There may be multiple non-exclusive locks in parallel.
A lock is a file in the subdir ``locks`` whose filename is the storage
ID of the contents. It is encrypted and authenticated the same way as
other files in the repository and contains the following JSON structure:
.. code:: json
{
"time": "2015-06-27T12:18:51.759239612+02:00",
"exclusive": false,
"hostname": "kasimir",
"username": "fd0",
"pid": 13607,
"uid": 1000,
"gid": 100
}
The field ``exclusive`` defines the type of lock. When a new lock is to
be created, restic checks all locks in the repository. When a lock is
found, it is tested if the lock is stale, which is the case for locks
with timestamps older than 30 minutes. If the lock was created on the
same machine, even for younger locks it is tested whether the process is
still alive by sending a signal to it. If that fails, restic assumes
that the process is dead and considers the lock to be stale.
When a new lock is to be created and no other conflicting locks are
detected, restic creates a new lock, waits, and checks if other locks
appeared in the repository. Depending on the type of the other locks and
the lock to be created, restic either continues or fails.
Backups and Deduplication
=========================
For creating a backup, restic scans the source directory for all files,
sub-directories and other entries. The data from each file is split into
variable length Blobs cut at offsets defined by a sliding window of 64
byte. The implementation uses Rabin Fingerprints for implementing this
Content Defined Chunking (CDC). An irreducible polynomial is selected at
random and saved in the file ``config`` when a repository is
initialized, so that watermark attacks are much harder.
Files smaller than 512 KiB are not split, Blobs are of 512 KiB to 8 MiB
in size. The implementation aims for 1 MiB Blob size on average.
For modified files, only modified Blobs have to be saved in a subsequent
backup. This even works if bytes are inserted or removed at arbitrary
positions within the file.
Threat Model
============
The design goals for restic include being able to securely store backups
in a location that is not completely trusted, e.g. a shared system where
others can potentially access the files or (in the case of the system
administrator) even modify or delete them.
General assumptions:
- The host system a backup is created on is trusted. This is the most
basic requirement, and essential for creating trustworthy backups.
The restic backup program guarantees the following:
- Accessing the unencrypted content of stored files and metadata should
not be possible without a password for the repository. Everything
except the metadata included for informational purposes in the key
files is encrypted and authenticated.
- Modifications (intentional or unintentional) can be detected
automatically on several layers:
1. For all accesses of data stored in the repository it is checked
whether the cryptographic hash of the contents matches the storage
ID (the file's name). This way, modifications (bad RAM, broken
harddisk) can be detected easily.
2. Before decrypting any data, the MAC on the encrypted data is
checked. If there has been a modification, the MAC check will
fail. This step happens even before the data is decrypted, so data
that has been tampered with is not decrypted at all.
However, the restic backup program is not designed to protect against
attackers deleting files at the storage location. There is nothing that
can be done about this. If this needs to be guaranteed, get a secure
location without any access from third parties. If you assume that
attackers have write access to your files at the storage location,
attackers are able to figure out (e.g. based on the timestamps of the
stored files) which files belong to what snapshot. When only these files
are deleted, the particular snapshot vanished and all snapshots
depending on data that has been added in the snapshot cannot be restored
completely. Restic is not designed to detect this attack.

9
doc/faq.rst
View file

@ -111,17 +111,8 @@ directory structure via sftp, so the path that needs to be specified is
different than the directory structure on the device and maybe even as exposed
via other protocols.
Try removing the /volume1 prefix in your paths. If this does not work, use sftp
and ls to explore the SFTP file system hierarchy on your NAS.
The following may work:
::
$ restic init -r sftp:user@nas:/restic-repo init
Why does restic perform so poorly on Windows?
---------------------------------------------
In some cases the realtime protection of antivirus software can interfere with restic's operations. If you are experiencing bad performace you can try to temporarily disable your antivirus software to find out if it is the cause for your performance problems.

2
doc/index.rst
View file

@ -12,10 +12,8 @@ Restic Documentation
050_restore
060_forget
070_encryption
075_scripting
080_examples
090_participating
100_references
110_talks
faq
manual_rest

Some files were not shown because too many files have changed in this diff Show more