Implemented Google Cloud Storage support using the existing minio libraries.
GCS repositories get specified using gs://bucketname/prefix syntax.
Cloned the existing S3 support and tests to use them for GCS as well.
Added tests for the various cases to config_test and location_test.
Remove trailing slashes in repository specification.
Prepends the object path prefix to all s3 paths and allows to have multiple independent
restic backup repositories in a single s3 bucket.
Removed the hardcoded "restic" prefix from s3 paths.
Use "restic" as the default object path prefix for s3 if no other prefix gets specified.
This will retain backward compatibility with existing s3 repository configurations.
Simplified the parse flow to have a single point where we parse the bucket name and the prefix within the bucket.
Added tests for s3 object path prefix and the new default prefix to config_test and location_test.
Prepends the object path prefix to all s3 paths and allows to have multiple independent
restic backup repositories in a single s3 bucket.
Removed the hardcoded "restic" prefix from s3 paths.
Use "restic" as the default object path prefix for s3 if no other prefix gets specified.
This will retain backward compatibility with existing s3 repository configurations.
Simplified the parse flow to have a single point where we parse the bucket name and the prefix within the bucket.
Added tests for s3 object path prefix and the new default prefix to config_test and location_test.
The current code returns io.ErrUnexpectedEOF, but it is the normal,
expected behaviour of the function LoadAll() to load until the item is
completely loaded. Therefore, the io.ErrUnexpectedEOF is not returned to
the caller.
When saving `/`, it was replaced with the contents, but without the
proper path. So `/` was replaced by [`boot`, `bin`, `home`, ...], but
without prefixing the entry name with the proper path.
Errors returned from backend.LoadAll() were not handled, leading to
these fatal errors from the unpacker trying to read the size from the end of
an empty buffer:
`seeking to read header length failed: bytes.Reader.Seek: negative position`
This change takes care of returning on error, as well as showing which pack
failed to load and validating pack integrity.
It was discovered that when restic is instructed to save `/`, the tree
structures in the repository contain an invalid node.
When saving the dir `/home/user`, the following structure is created:
snapshot
-> tree
nodes: ["user"]
[...]
When the root directory `/` is saved, the structure is as follows:
snapshot
-> tree
nodes: ["/"]
[...]
This behavior is caused by the walker in pipe.go sending a node with the
name "." to the archiver, so this commit adds a test for invalid node
names.
This removes the list of in-flight blobs from the master index and
instead keeps a list of "known" blobs in the Archiver. "known" here
means: either already processed, or included in an index. This property
is tested atomically, when the blob is not in the list of "known" blobs,
it is added to the list and the caller is responsible to make this
happen (i.e. save the blob).
