1
0
Fork 0
mirror of https://github.com/restic/restic.git synced 2025-03-16 00:00:05 +01:00

Merge branch 'master' into progress_interval_cli_option

* master: (28 commits)
  Fix layout issue in cmd_snapshot "ascii art"
  Fix SamePaths() and make it into a receiver function
  Documentation fixes
  Add design and user documentation for the `restic tag` command
  Snapshot: Add Original ID
  Fix type of ID field in `cmd_snapshots` type Snapshot
  Snapshot: Add `AddTags()` and `RemoveTags()`
  Add `tag`: Manipulate tags on existing snapshots
  Add hint for other backend URI formats
  Make columns for host and tags size width dynamicly on their content.
  Add integration test to make sure cmd_backup adds tags when required.
  Changed cmd_snapshots to be testable (no more using os.Stdout)
  English typo: rewriten > rewritten.
  Add progressbar to repack and blob remove phases of prune cmd.
  Display the proper amount of bytes we will be pruning from the repo.
  Update github.com/pkg/xattr
  Fix Minio Server URL
  Really use absolute pathnames, not all systems use /.
  Fix unit test, we need to check for absolute paths now.
  Display absolute paths when displaying the output of ls and find.
  ...
This commit is contained in:
trbs 2017-03-06 21:09:47 +01:00
commit 7e6c38e335
24 changed files with 703 additions and 204 deletions

View file

@ -19,13 +19,15 @@ matrix:
exclude:
- os: osx
go: 1.6.4
- os: osx
go: 1.7.5
- os: osx
go: tip
- os: linux
go: 1.7.5
go: 1.8
include:
- os: linux
go: 1.7.5
go: 1.8
sudo: true
env:
RESTIC_TEST_FUSE=1

View file

@ -285,7 +285,7 @@ This way, the password can be changed without having to re-encrypt all data.
Snapshots
---------
A snapshots represents a directory with all files and sub-directories at a
A snapshot represents a directory with all files and sub-directories at a
given point in time. For each backup that is made, a new snapshot is created. A
snapshot is a JSON document that is stored in an encrypted file below the
directory `snapshots` in the repository. The filename is the storage ID. This
@ -294,6 +294,31 @@ string is unique and used within restic to uniquely identify a snapshot.
The command `restic cat snapshot` can be used as follows to decrypt and
pretty-print the contents of a snapshot file:
```console
$ restic -r /tmp/restic-repo cat snapshot 251c2e58
enter password for repository:
{
"time": "2015-01-02T18:10:50.895208559+01:00",
"tree": "2da81727b6585232894cfbb8f8bdab8d1eccd3d8f7c92bc934d62e62e618ffdf",
"dir": "/tmp/testdata",
"hostname": "kasimir",
"username": "fd0",
"uid": 1000,
"gid": 100,
"tags": [
"NL"
]
}
```
Here it can be seen that this snapshot represents the contents of the directory
`/tmp/testdata`. The most important field is `tree`. When the meta data (e.g.
the tags) of a snapshot change, the snapshot needs to be re-encrypted and saved.
This will change the storage ID, so in order to relate these seemingly
different snapshots, a field `original` is introduced which contains the ID of
the original snapshot, e.g. after adding the tag `DE` to the snapshot above it
becomes:
```console
$ restic -r /tmp/restic-repo cat snapshot 22a5af1b
enter password for repository:
@ -304,12 +329,17 @@ enter password for repository:
"hostname": "kasimir",
"username": "fd0",
"uid": 1000,
"gid": 100
"gid": 100,
"tags": [
"NL",
"DE"
],
"original": "251c2e5841355f743f9d4ffd3260bee765acee40a6229857e32b60446991b837"
}
```
Here it can be seen that this snapshot represents the contents of the directory
`/tmp/testdata`. The most important field is `tree`.
Once introduced, the `original` field is not modified when the snapshot's meta
data is changed again.
All content within a restic repository is referenced according to its SHA-256
hash. Before saving, each file is split into variable sized Blobs of data. The

View file

@ -73,6 +73,7 @@ Available Commands:
rebuild-index build a new index file
restore extract the data from a snapshot
snapshots list all snapshots
tag modifies tags on snapshots
unlock remove locks other processes created
version Print version information
@ -145,6 +146,8 @@ Please note that knowledge of your password is required to access the repository
Losing your password means that your data is irrecoverably lost.
```
Other backends like sftp and s3 are [described in a later section](#create-an-sftp-repository) of this document.
Remembering your password is important! If you lose it, you won't be able to
access data stored in the repository.
@ -394,6 +397,45 @@ enter password for repository:
*eb78040b username kasimir 2015-08-12 13:29:57
```
# Manage tags
Managing tags on snapshots is done with the `tag` command. The existing set of
tags can be replaced completely, tags can be added to removed. The result is
directly visible in the `snapshots` command.
Let's say we want to tag snapshot `590c8fc8` with the tags `NL` and `CH` and
remove all other tags that may be present, the following command does that:
```console
$ restic -r /tmp/backup tag --set NL,CH 590c8fc8
Create exclusive lock for repository
Modified tags on 1 snapshots
```
Note the snapshot ID has changed, so between each change we need to look up
the new ID of the snapshot. But there is an even better way, the `tag` command
accepts `--tag` for a filter, so we can filter snapshots based on the tag we
just added.
So we can add and remove tags incrementally like this:
```console
$ restic -r /tmp/backup tag --tag NL --remove CH
Create exclusive lock for repository
Modified tags on 1 snapshots
$ restic -r /tmp/backup tag --tag NL --add UK
Create exclusive lock for repository
Modified tags on 1 snapshots
$ restic -r /tmp/backup tag --tag NL --remove NL
Create exclusive lock for repository
Modified tags on 1 snapshots
$ restic -r /tmp/backup tag --tag NL --add SOMETHING
No snapshots were modified
```
# Check integrity and consistency
Imagine your repository is saved on a server that has a faulty hard drive, or
@ -528,7 +570,7 @@ only available via HTTP, you can specify the URL to the server like this:
### Pre-Requisites
* Download and Install [Minio Server](https://minio.io/download/).
* Download and Install [Minio Server](https://minio.io/downloads/#minio-server).
* You can also refer to [https://docs.minio.io](https://docs.minio.io) for step by step guidance on installation and getting started on Minio Client and Minio Server.
You must first setup the following environment variables with the credentials of your running Minio Server.

View file

@ -272,7 +272,13 @@ func readBackupFromStdin(opts BackupOptions, gopts GlobalOptions, args []string)
return err
}
_, id, err := archiver.ArchiveReader(repo, newArchiveStdinProgress(gopts), os.Stdin, opts.StdinFilename, opts.Tags, opts.Hostname)
r := &archiver.Reader{
Repository: repo,
Tags: opts.Tags,
Hostname: opts.Hostname,
}
_, id, err := r.Archive(opts.StdinFilename, os.Stdin, newArchiveStdinProgress(gopts))
if err != nil {
return err
}
@ -304,7 +310,11 @@ func readLinesFromFile(filename string) ([]string, error) {
scanner := bufio.NewScanner(r)
for scanner.Scan() {
lines = append(lines, scanner.Text())
line := scanner.Text()
if line == "" {
continue
}
lines = append(lines, line)
}
if err := scanner.Err(); err != nil {

View file

@ -129,7 +129,7 @@ func findInSnapshot(repo *repository.Repository, pat findPattern, id restic.ID)
return err
}
results, err := findInTree(repo, pat, *sn.Tree, "")
results, err := findInTree(repo, pat, *sn.Tree, string(filepath.Separator))
if err != nil {
return err
}

View file

@ -121,5 +121,5 @@ func runLs(gopts GlobalOptions, args []string) error {
Verbosef("snapshot of %v at %s:\n", sn.Paths, sn.Time)
return printTree("", repo, *sn.Tree)
return printTree(string(filepath.Separator), repo, *sn.Tree)
}

View file

@ -184,7 +184,7 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
}
}
removeBytes := 0
removeBytes := duplicateBytes
// find packs that are unneeded
removePacks := restic.NewIDSet()
@ -217,17 +217,28 @@ func pruneRepository(gopts GlobalOptions, repo restic.Repository) error {
Verbosef("will delete %d packs and rewrite %d packs, this frees %s\n",
len(removePacks), len(rewritePacks), formatBytes(uint64(removeBytes)))
err = repository.Repack(repo, rewritePacks, usedBlobs)
if err != nil {
return err
if len(rewritePacks) != 0 {
bar = newProgressMax(!gopts.Quiet, uint64(len(rewritePacks)), "packs rewritten")
bar.Start()
err = repository.Repack(repo, rewritePacks, usedBlobs, bar)
if err != nil {
return err
}
bar.Done()
}
for packID := range removePacks {
h := restic.Handle{Type: restic.DataFile, Name: packID.String()}
err = repo.Backend().Remove(h)
if err != nil {
Warnf("unable to remove file %v from the repository\n", packID.Str())
if len(removePacks) != 0 {
bar = newProgressMax(!gopts.Quiet, uint64(len(removePacks)), "packs deleted")
bar.Start()
for packID := range removePacks {
h := restic.Handle{Type: restic.DataFile, Name: packID.String()}
err = repo.Backend().Remove(h)
if err != nil {
Warnf("unable to remove file %v from the repository\n", packID.Str())
}
bar.Report(restic.Stat{Blobs: 1})
}
bar.Done()
}
Verbosef("creating new index\n")

View file

@ -1,7 +1,8 @@
package main
import (
"restic/repository"
"restic"
"restic/index"
"github.com/spf13/cobra"
)
@ -34,5 +35,47 @@ func runRebuildIndex(gopts GlobalOptions) error {
return err
}
return repository.RebuildIndex(repo)
done := make(chan struct{})
defer close(done)
Verbosef("counting files in repo\n")
var packs uint64
for _ = range repo.List(restic.DataFile, done) {
packs++
}
bar := newProgressMax(!gopts.Quiet, packs, "packs")
idx, err := index.New(repo, bar)
if err != nil {
return err
}
Verbosef("listing old index files\n")
var supersedes restic.IDs
for id := range repo.List(restic.IndexFile, done) {
supersedes = append(supersedes, id)
}
id, err := idx.Save(repo, supersedes)
if err != nil {
return err
}
Verbosef("saved new index as %v\n", id.Str())
Verbosef("remove %d old index files\n", len(supersedes))
for _, id := range supersedes {
err := repo.Backend().Remove(restic.Handle{
Type: restic.IndexFile,
Name: id.String(),
})
if err != nil {
Warnf("error deleting old index %v: %v\n", id.Str(), err)
}
}
return nil
}

View file

@ -2,7 +2,7 @@ package main
import (
"fmt"
"os"
"io"
"restic/errors"
"sort"
@ -64,11 +64,11 @@ func runSnapshots(opts SnapshotOptions, gopts GlobalOptions, args []string) erro
for id := range repo.List(restic.SnapshotFile, done) {
sn, err := restic.LoadSnapshot(repo, id)
if err != nil {
fmt.Fprintf(os.Stderr, "error loading snapshot %s: %v\n", id, err)
Warnf("error loading snapshot %s: %v\n", id, err)
continue
}
if restic.SamePaths(sn.Paths, opts.Paths) && (opts.Host == "" || opts.Host == sn.Hostname) {
if (opts.Host == "" || opts.Host == sn.Hostname) && sn.HasPaths(opts.Paths) {
pos := sort.Search(len(list), func(i int) bool {
return list[i].Time.After(sn.Time)
})
@ -85,23 +85,36 @@ func runSnapshots(opts SnapshotOptions, gopts GlobalOptions, args []string) erro
}
if gopts.JSON {
err := printSnapshotsJSON(list)
err := printSnapshotsJSON(gopts.stdout, list)
if err != nil {
fmt.Fprintf(os.Stderr, "error printing snapshot: %v\n", err)
Warnf("error printing snapshot: %v\n", err)
}
return nil
}
printSnapshotsReadable(list)
printSnapshotsReadable(gopts.stdout, list)
return nil
}
// printSnapshotsReadable prints a text table of the snapshots in list to stdout.
func printSnapshotsReadable(list []*restic.Snapshot) {
func printSnapshotsReadable(stdout io.Writer, list []*restic.Snapshot) {
// Determine the max widths for host and tag.
maxHost, maxTag := 10, 6
for _, sn := range list {
if len(sn.Hostname) > maxHost {
maxHost = len(sn.Hostname)
}
for _, tag := range sn.Tags {
if len(tag) > maxTag {
maxTag = len(tag)
}
}
}
tab := NewTable()
tab.Header = fmt.Sprintf("%-8s %-19s %-10s %-10s %-3s %s", "ID", "Date", "Host", "Tags", "", "Directory")
tab.RowFormat = "%-8s %-19s %-10s %-10s %-3s %s"
tab.Header = fmt.Sprintf("%-8s %-19s %-*s %-*s %-3s %s", "ID", "Date", -maxHost, "Host", -maxTag, "Tags", "", "Directory")
tab.RowFormat = fmt.Sprintf("%%-8s %%-19s %%%ds %%%ds %%-3s %%s", -maxHost, -maxTag)
for _, sn := range list {
if len(sn.Paths) == 0 {
@ -114,6 +127,9 @@ func printSnapshotsReadable(list []*restic.Snapshot) {
}
rows := len(sn.Paths)
if rows < len(sn.Tags) {
rows = len(sn.Tags)
}
treeElement := " "
if rows != 1 {
@ -146,20 +162,18 @@ func printSnapshotsReadable(list []*restic.Snapshot) {
}
}
tab.Write(os.Stdout)
return
tab.Write(stdout)
}
// Snapshot helps to print Snaphots as JSON
type Snapshot struct {
*restic.Snapshot
ID string `json:"id"`
ID *restic.ID `json:"id"`
}
// printSnapshotsJSON writes the JSON representation of list to stdout.
func printSnapshotsJSON(list []*restic.Snapshot) error {
func printSnapshotsJSON(stdout io.Writer, list []*restic.Snapshot) error {
var snapshots []Snapshot
@ -167,11 +181,11 @@ func printSnapshotsJSON(list []*restic.Snapshot) error {
k := Snapshot{
Snapshot: sn,
ID: sn.ID().String(),
ID: sn.ID(),
}
snapshots = append(snapshots, k)
}
return json.NewEncoder(os.Stdout).Encode(snapshots)
return json.NewEncoder(stdout).Encode(snapshots)
}

172
src/cmds/restic/cmd_tag.go Normal file
View file

@ -0,0 +1,172 @@
package main
import (
"github.com/spf13/cobra"
"restic"
"restic/debug"
"restic/errors"
"restic/repository"
)
var cmdTag = &cobra.Command{
Use: "tag [flags] [snapshot-ID ...]",
Short: "modifies tags on snapshots",
Long: `
The "tag" command allows you to modify tags on exiting snapshots.
You can either set/replace the entire set of tags on a snapshot, or
add tags to/remove tags from the existing set.
When no snapshot-ID is given, all snapshots matching the host, tag and path filter criteria are modified.
`,
RunE: func(cmd *cobra.Command, args []string) error {
return runTag(tagOptions, globalOptions, args)
},
}
// TagOptions bundles all options for the 'tag' command.
type TagOptions struct {
Host string
Paths []string
Tags []string
SetTags []string
AddTags []string
RemoveTags []string
}
var tagOptions TagOptions
func init() {
cmdRoot.AddCommand(cmdTag)
tagFlags := cmdTag.Flags()
tagFlags.StringSliceVar(&tagOptions.SetTags, "set", nil, "`tag` which will replace the existing tags (can be given multiple times)")
tagFlags.StringSliceVar(&tagOptions.AddTags, "add", nil, "`tag` which will be added to the existing tags (can be given multiple times)")
tagFlags.StringSliceVar(&tagOptions.RemoveTags, "remove", nil, "`tag` which will be removed from the existing tags (can be given multiple times)")
tagFlags.StringVarP(&tagOptions.Host, "host", "H", "", `only consider snapshots for this host, when no snapshot ID is given`)
tagFlags.StringSliceVar(&tagOptions.Tags, "tag", nil, "only consider snapshots which include this `tag`, when no snapshot-ID is given")
tagFlags.StringSliceVar(&tagOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path`, when no snapshot-ID is given")
}
func changeTags(repo *repository.Repository, snapshotID restic.ID, setTags, addTags, removeTags, tags, paths []string, host string) (bool, error) {
var changed bool
sn, err := restic.LoadSnapshot(repo, snapshotID)
if err != nil {
return false, err
}
if (host != "" && host != sn.Hostname) || !sn.HasTags(tags) || !sn.HasPaths(paths) {
return false, nil
}
if len(setTags) != 0 {
// Setting the tag to an empty string really means no tags.
if len(setTags) == 1 && setTags[0] == "" {
setTags = nil
}
sn.Tags = setTags
changed = true
} else {
changed = sn.AddTags(addTags)
if sn.RemoveTags(removeTags) {
changed = true
}
}
if changed {
// Retain the original snapshot id over all tag changes.
if sn.Original == nil {
sn.Original = sn.ID()
}
// Save the new snapshot.
id, err := repo.SaveJSONUnpacked(restic.SnapshotFile, sn)
if err != nil {
return false, err
}
debug.Log("new snapshot saved as %v", id.Str())
if err = repo.Flush(); err != nil {
return false, err
}
// Remove the old snapshot.
h := restic.Handle{Type: restic.SnapshotFile, Name: sn.ID().String()}
if err = repo.Backend().Remove(h); err != nil {
return false, err
}
debug.Log("old snapshot %v removed", sn.ID())
}
return changed, nil
}
func runTag(opts TagOptions, gopts GlobalOptions, args []string) error {
if len(opts.SetTags) == 0 && len(opts.AddTags) == 0 && len(opts.RemoveTags) == 0 {
return errors.Fatal("nothing to do!")
}
if len(opts.SetTags) != 0 && (len(opts.AddTags) != 0 || len(opts.RemoveTags) != 0) {
return errors.Fatal("--set and --add/--remove cannot be given at the same time")
}
repo, err := OpenRepository(gopts)
if err != nil {
return err
}
if !gopts.NoLock {
Verbosef("Create exclusive lock for repository\n")
lock, err := lockRepoExclusive(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
}
var ids restic.IDs
if len(args) != 0 {
// When explit snapshot-IDs are given, the filtering does not matter anymore.
opts.Host = ""
opts.Tags = nil
opts.Paths = nil
// Process all snapshot IDs given as arguments.
for _, s := range args {
snapshotID, err := restic.FindSnapshot(repo, s)
if err != nil {
Warnf("could not find a snapshot for ID %q, ignoring: %v\n", s, err)
continue
}
ids = append(ids, snapshotID)
}
ids = ids.Uniq()
} else {
// If there were no arguments, just get all snapshots.
done := make(chan struct{})
defer close(done)
for snapshotID := range repo.List(restic.SnapshotFile, done) {
ids = append(ids, snapshotID)
}
}
changeCnt := 0
for _, id := range ids {
changed, err := changeTags(repo, id, opts.SetTags, opts.AddTags, opts.RemoveTags, opts.Tags, opts.Paths, opts.Host)
if err != nil {
Warnf("unable to modify the tags for snapshot ID %q, ignoring: %v\n", id, err)
continue
}
if changed {
changeCnt++
}
}
if changeCnt == 0 {
Verbosef("No snapshots were modified\n")
} else {
Verbosef("Modified tags on %v snapshots\n", changeCnt)
}
return nil
}

View file

@ -4,6 +4,7 @@ import (
"bufio"
"bytes"
"crypto/rand"
"encoding/json"
"fmt"
"io"
"io/ioutil"
@ -160,6 +161,32 @@ func testRunFind(t testing.TB, gopts GlobalOptions, pattern string) []string {
return strings.Split(string(buf.Bytes()), "\n")
}
func testRunSnapshots(t testing.TB, gopts GlobalOptions) (newest *Snapshot, snapmap map[restic.ID]Snapshot) {
buf := bytes.NewBuffer(nil)
globalOptions.stdout = buf
globalOptions.JSON = true
defer func() {
globalOptions.stdout = os.Stdout
globalOptions.JSON = gopts.JSON
}()
opts := SnapshotOptions{}
OK(t, runSnapshots(opts, globalOptions, []string{}))
snapshots := []Snapshot{}
OK(t, json.Unmarshal(buf.Bytes(), &snapshots))
snapmap = make(map[restic.ID]Snapshot, len(snapshots))
for _, sn := range snapshots {
snapmap[*sn.ID] = sn
if newest == nil || sn.Time.After(newest.Time) {
newest = &sn
}
}
return
}
func testRunForget(t testing.TB, gopts GlobalOptions, args ...string) {
opts := ForgetOptions{}
OK(t, runForget(opts, gopts, args))
@ -516,23 +543,23 @@ func TestBackupExclude(t *testing.T) {
testRunBackup(t, []string{datadir}, opts, gopts)
snapshots, snapshotID := lastSnapshot(snapshots, loadSnapshotMap(t, gopts))
files := testRunLs(t, gopts, snapshotID)
Assert(t, includes(files, filepath.Join("testdata", "foo.tar.gz")),
Assert(t, includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
"expected file %q in first snapshot, but it's not included", "foo.tar.gz")
opts.Excludes = []string{"*.tar.gz"}
testRunBackup(t, []string{datadir}, opts, gopts)
snapshots, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, gopts))
files = testRunLs(t, gopts, snapshotID)
Assert(t, !includes(files, filepath.Join("testdata", "foo.tar.gz")),
Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
opts.Excludes = []string{"*.tar.gz", "private/secret"}
testRunBackup(t, []string{datadir}, opts, gopts)
snapshots, snapshotID = lastSnapshot(snapshots, loadSnapshotMap(t, gopts))
files = testRunLs(t, gopts, snapshotID)
Assert(t, !includes(files, filepath.Join("testdata", "foo.tar.gz")),
Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "foo.tar.gz")),
"expected file %q not in first snapshot, but it's included", "foo.tar.gz")
Assert(t, !includes(files, filepath.Join("testdata", "private", "secret", "passwords.txt")),
Assert(t, !includes(files, filepath.Join(string(filepath.Separator), "testdata", "private", "secret", "passwords.txt")),
"expected file %q not in first snapshot, but it's included", "passwords.txt")
})
}
@ -602,6 +629,105 @@ func TestIncrementalBackup(t *testing.T) {
})
}
func TestBackupTags(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, gopts GlobalOptions) {
datafile := filepath.Join("testdata", "backup-data.tar.gz")
testRunInit(t, gopts)
SetupTarTestFixture(t, env.testdata, datafile)
opts := BackupOptions{}
testRunBackup(t, []string{env.testdata}, opts, gopts)
testRunCheck(t, gopts)
newest, _ := testRunSnapshots(t, gopts)
Assert(t, newest != nil, "expected a new backup, got nil")
Assert(t, len(newest.Tags) == 0,
"expected no tags, got %v", newest.Tags)
opts.Tags = []string{"NL"}
testRunBackup(t, []string{env.testdata}, opts, gopts)
testRunCheck(t, gopts)
newest, _ = testRunSnapshots(t, gopts)
Assert(t, newest != nil, "expected a new backup, got nil")
Assert(t, len(newest.Tags) == 1 && newest.Tags[0] == "NL",
"expected one NL tag, got %v", newest.Tags)
})
}
func testRunTag(t testing.TB, opts TagOptions, gopts GlobalOptions) {
OK(t, runTag(opts, gopts, []string{}))
}
func TestTag(t *testing.T) {
withTestEnvironment(t, func(env *testEnvironment, gopts GlobalOptions) {
datafile := filepath.Join("testdata", "backup-data.tar.gz")
testRunInit(t, gopts)
SetupTarTestFixture(t, env.testdata, datafile)
testRunBackup(t, []string{env.testdata}, BackupOptions{}, gopts)
testRunCheck(t, gopts)
newest, _ := testRunSnapshots(t, gopts)
Assert(t, newest != nil, "expected a new backup, got nil")
Assert(t, len(newest.Tags) == 0,
"expected no tags, got %v", newest.Tags)
Assert(t, newest.Original == nil,
"expected original ID to be nil, got %v", newest.Original)
originalID := *newest.ID
testRunTag(t, TagOptions{SetTags: []string{"NL"}}, gopts)
testRunCheck(t, gopts)
newest, _ = testRunSnapshots(t, gopts)
Assert(t, newest != nil, "expected a new backup, got nil")
Assert(t, len(newest.Tags) == 1 && newest.Tags[0] == "NL",
"set failed, expected one NL tag, got %v", newest.Tags)
Assert(t, newest.Original != nil, "expected original snapshot id, got nil")
Assert(t, *newest.Original == originalID,
"expected original ID to be set to the first snapshot id")
testRunTag(t, TagOptions{AddTags: []string{"CH"}}, gopts)
testRunCheck(t, gopts)
newest, _ = testRunSnapshots(t, gopts)
Assert(t, newest != nil, "expected a new backup, got nil")
Assert(t, len(newest.Tags) == 2 && newest.Tags[0] == "NL" && newest.Tags[1] == "CH",
"add failed, expected CH,NL tags, got %v", newest.Tags)
Assert(t, newest.Original != nil, "expected original snapshot id, got nil")
Assert(t, *newest.Original == originalID,
"expected original ID to be set to the first snapshot id")
testRunTag(t, TagOptions{RemoveTags: []string{"NL"}}, gopts)
testRunCheck(t, gopts)
newest, _ = testRunSnapshots(t, gopts)
Assert(t, newest != nil, "expected a new backup, got nil")
Assert(t, len(newest.Tags) == 1 && newest.Tags[0] == "CH",
"remove failed, expected one CH tag, got %v", newest.Tags)
Assert(t, newest.Original != nil, "expected original snapshot id, got nil")
Assert(t, *newest.Original == originalID,
"expected original ID to be set to the first snapshot id")
testRunTag(t, TagOptions{AddTags: []string{"US", "RU"}}, gopts)
testRunTag(t, TagOptions{RemoveTags: []string{"CH", "US", "RU"}}, gopts)
testRunCheck(t, gopts)
newest, _ = testRunSnapshots(t, gopts)
Assert(t, newest != nil, "expected a new backup, got nil")
Assert(t, len(newest.Tags) == 0,
"expected no tags, got %v", newest.Tags)
Assert(t, newest.Original != nil, "expected original snapshot id, got nil")
Assert(t, *newest.Original == originalID,
"expected original ID to be set to the first snapshot id")
// Check special case of removing all tags.
testRunTag(t, TagOptions{SetTags: []string{""}}, gopts)
testRunCheck(t, gopts)
newest, _ = testRunSnapshots(t, gopts)
Assert(t, newest != nil, "expected a new backup, got nil")
Assert(t, len(newest.Tags) == 0,
"expected no tags, got %v", newest.Tags)
Assert(t, newest.Original != nil, "expected original snapshot id, got nil")
Assert(t, *newest.Original == originalID,
"expected original ID to be set to the first snapshot id")
})
}
func testRunKeyListOtherIDs(t testing.TB, gopts GlobalOptions) []string {
buf := bytes.NewBuffer(nil)

View file

@ -11,15 +11,22 @@ import (
"github.com/restic/chunker"
)
// ArchiveReader reads from the reader and archives the data. Returned is the
// resulting snapshot and its ID.
func ArchiveReader(repo restic.Repository, p *restic.Progress, rd io.Reader, name string, tags []string, hostname string) (*restic.Snapshot, restic.ID, error) {
// Reader allows saving a stream of data to the repository.
type Reader struct {
restic.Repository
Tags []string
Hostname string
}
// Archive reads data from the reader and saves it to the repo.
func (r *Reader) Archive(name string, rd io.Reader, p *restic.Progress) (*restic.Snapshot, restic.ID, error) {
if name == "" {
return nil, restic.ID{}, errors.New("no filename given")
}
debug.Log("start archiving %s", name)
sn, err := restic.NewSnapshot([]string{name}, tags, hostname)
sn, err := restic.NewSnapshot([]string{name}, r.Tags, r.Hostname)
if err != nil {
return nil, restic.ID{}, err
}
@ -27,6 +34,7 @@ func ArchiveReader(repo restic.Repository, p *restic.Progress, rd io.Reader, nam
p.Start()
defer p.Done()
repo := r.Repository
chnker := chunker.New(rd, repo.Config().ChunkerPolynomial)
ids := restic.IDs{}

View file

@ -79,7 +79,13 @@ func TestArchiveReader(t *testing.T) {
f := fakeFile(t, seed, size)
sn, id, err := ArchiveReader(repo, nil, f, "fakefile", []string{"test"}, "localhost")
r := &Reader{
Repository: repo,
Hostname: "localhost",
Tags: []string{"test"},
}
sn, id, err := r.Archive("fakefile", f, nil)
if err != nil {
t.Fatalf("ArchiveReader() returned error %v", err)
}
@ -99,7 +105,13 @@ func TestArchiveReaderNull(t *testing.T) {
repo, cleanup := repository.TestRepository(t)
defer cleanup()
sn, id, err := ArchiveReader(repo, nil, bytes.NewReader(nil), "fakefile", nil, "localhost")
r := &Reader{
Repository: repo,
Hostname: "localhost",
Tags: []string{"test"},
}
sn, id, err := r.Archive("fakefile", bytes.NewReader(nil), nil)
if err != nil {
t.Fatalf("ArchiveReader() returned error %v", err)
}
@ -134,7 +146,13 @@ func TestArchiveReaderError(t *testing.T) {
repo, cleanup := repository.TestRepository(t)
defer cleanup()
sn, id, err := ArchiveReader(repo, nil, errReader("error returned by reading stdin"), "fakefile", nil, "localhost")
r := &Reader{
Repository: repo,
Hostname: "localhost",
Tags: []string{"test"},
}
sn, id, err := r.Archive("fakefile", errReader("error returned by reading stdin"), nil)
if err == nil {
t.Errorf("expected error not returned")
}
@ -167,11 +185,17 @@ func BenchmarkArchiveReader(t *testing.B) {
t.Fatal(err)
}
r := &Reader{
Repository: repo,
Hostname: "localhost",
Tags: []string{"test"},
}
t.SetBytes(size)
t.ResetTimer()
for i := 0; i < t.N; i++ {
_, _, err := ArchiveReader(repo, nil, bytes.NewReader(buf), "fakefile", []string{"test"}, "localhost")
_, _, err := r.Archive("fakefile", bytes.NewReader(buf), nil)
if err != nil {
t.Fatal(err)
}

View file

@ -1,66 +0,0 @@
package repository
import (
"fmt"
"os"
"restic"
"restic/debug"
"restic/list"
"restic/worker"
)
// RebuildIndex lists all packs in the repo, writes a new index and removes all
// old indexes. This operation should only be done with an exclusive lock in
// place.
func RebuildIndex(repo restic.Repository) error {
debug.Log("start rebuilding index")
done := make(chan struct{})
defer close(done)
ch := make(chan worker.Job)
go list.AllPacks(repo, ch, done)
idx := NewIndex()
for job := range ch {
id := job.Data.(restic.ID)
if job.Error != nil {
fmt.Fprintf(os.Stderr, "error for pack %v: %v\n", id, job.Error)
continue
}
res := job.Result.(list.Result)
for _, entry := range res.Entries() {
pb := restic.PackedBlob{
Blob: entry,
PackID: res.PackID(),
}
idx.Store(pb)
}
}
oldIndexes := restic.NewIDSet()
for id := range repo.List(restic.IndexFile, done) {
idx.AddToSupersedes(id)
oldIndexes.Insert(id)
}
id, err := SaveIndex(repo, idx)
if err != nil {
debug.Log("error saving index: %v", err)
return err
}
debug.Log("new index saved as %v", id.Str())
for indexID := range oldIndexes {
h := restic.Handle{Type: restic.IndexFile, Name: indexID.String()}
err := repo.Backend().Remove(h)
if err != nil {
fmt.Fprintf(os.Stderr, "unable to remove index %v: %v\n", indexID.Str(), err)
}
}
return nil
}

View file

@ -18,7 +18,7 @@ import (
// these packs. Each pack is loaded and the blobs listed in keepBlobs is saved
// into a new pack. Afterwards, the packs are removed. This operation requires
// an exclusive lock on the repo.
func Repack(repo restic.Repository, packs restic.IDSet, keepBlobs restic.BlobSet) (err error) {
func Repack(repo restic.Repository, packs restic.IDSet, keepBlobs restic.BlobSet, p *restic.Progress) (err error) {
debug.Log("repacking %d packs while keeping %d blobs", len(packs), len(keepBlobs))
for packID := range packs {
@ -118,6 +118,9 @@ func Repack(repo restic.Repository, packs restic.IDSet, keepBlobs restic.BlobSet
if err = os.Remove(tempfile.Name()); err != nil {
return errors.Wrap(err, "Remove")
}
if p != nil {
p.Report(restic.Stat{Blobs: 1})
}
}
if err := repo.Flush(); err != nil {

View file

@ -4,6 +4,7 @@ import (
"io"
"math/rand"
"restic"
"restic/index"
"restic/repository"
"testing"
)
@ -131,7 +132,7 @@ func findPacksForBlobs(t *testing.T, repo restic.Repository, blobs restic.BlobSe
}
func repack(t *testing.T, repo restic.Repository, packs restic.IDSet, blobs restic.BlobSet) {
err := repository.Repack(repo, packs, blobs)
err := repository.Repack(repo, packs, blobs, nil)
if err != nil {
t.Fatal(err)
}
@ -144,8 +145,24 @@ func saveIndex(t *testing.T, repo restic.Repository) {
}
func rebuildIndex(t *testing.T, repo restic.Repository) {
if err := repository.RebuildIndex(repo); err != nil {
t.Fatalf("error rebuilding index: %v", err)
idx, err := index.New(repo, nil)
if err != nil {
t.Fatal(err)
}
for id := range repo.List(restic.IndexFile, nil) {
err = repo.Backend().Remove(restic.Handle{
Type: restic.IndexFile,
Name: id.String(),
})
if err != nil {
t.Fatal(err)
}
}
_, err = idx.Save(repo, nil)
if err != nil {
t.Fatal(err)
}
}

View file

@ -116,11 +116,11 @@ func (res *Restorer) restoreNodeTo(node *Node, dir string, dst string, idx *Hard
return nil
}
// RestoreTo creates the directories and files in the snapshot below dir.
// RestoreTo creates the directories and files in the snapshot below dst.
// Before an item is created, res.Filter is called.
func (res *Restorer) RestoreTo(dir string) error {
func (res *Restorer) RestoreTo(dst string) error {
idx := NewHardlinkIndex()
return res.restoreTo(dir, "", *res.sn.Tree, idx)
return res.restoreTo(dst, string(filepath.Separator), *res.sn.Tree, idx)
}
// Snapshot returns the snapshot this restorer is configured to use.

View file

@ -21,6 +21,7 @@ type Snapshot struct {
GID uint32 `json:"gid,omitempty"`
Excludes []string `json:"excludes,omitempty"`
Tags []string `json:"tags,omitempty"`
Original *ID `json:"original,omitempty"`
id *ID // plaintext ID, used during restore
}
@ -73,8 +74,7 @@ func LoadAllSnapshots(repo Repository) (snapshots []*Snapshot, err error) {
snapshots = append(snapshots, sn)
}
return snapshots, nil
return
}
func (sn Snapshot) String() string {
@ -99,7 +99,42 @@ func (sn *Snapshot) fillUserInfo() error {
return err
}
// HasTags returns true if the snapshot has all the tags.
// AddTags adds the given tags to the snapshots tags, preventing duplicates.
// It returns true if any changes were made.
func (sn *Snapshot) AddTags(addTags []string) (changed bool) {
nextTag:
for _, add := range addTags {
for _, tag := range sn.Tags {
if tag == add {
continue nextTag
}
}
sn.Tags = append(sn.Tags, add)
changed = true
}
return
}
// RemoveTags removes the given tags from the snapshots tags and
// returns true if any changes were made.
func (sn *Snapshot) RemoveTags(removeTags []string) (changed bool) {
for _, remove := range removeTags {
for i, tag := range sn.Tags {
if tag == remove {
// https://github.com/golang/go/wiki/SliceTricks
sn.Tags[i] = sn.Tags[len(sn.Tags)-1]
sn.Tags[len(sn.Tags)-1] = ""
sn.Tags = sn.Tags[:len(sn.Tags)-1]
changed = true
break
}
}
}
return
}
// HasTags returns true if the snapshot has at least all of tags.
func (sn *Snapshot) HasTags(tags []string) bool {
nextTag:
for _, tag := range tags {
@ -115,28 +150,30 @@ nextTag:
return true
}
// SamePaths compares the Snapshot's paths and provided paths are exactly the same
func SamePaths(expected, actual []string) bool {
if len(expected) == 0 || len(actual) == 0 {
return true
}
for i := range expected {
found := false
for j := range actual {
if expected[i] == actual[j] {
found = true
break
// HasPaths returns true if the snapshot has at least all of paths.
func (sn *Snapshot) HasPaths(paths []string) bool {
nextPath:
for _, path := range paths {
for _, snPath := range sn.Paths {
if path == snPath {
continue nextPath
}
}
if !found {
return false
}
return false
}
return true
}
// SamePaths returns true if the snapshot matches the entire paths set
func (sn *Snapshot) SamePaths(paths []string) bool {
if len(sn.Paths) != len(paths) {
return false
}
return sn.HasPaths(paths)
}
// ErrNoSnapshotFound is returned when no snapshot for the given criteria could be found.
var ErrNoSnapshotFound = errors.New("no snapshot found")
@ -153,7 +190,7 @@ func FindLatestSnapshot(repo Repository, targets []string, hostname string) (ID,
if err != nil {
return ID{}, errors.Errorf("Error listing snapshot: %v", err)
}
if snapshot.Time.After(latest) && SamePaths(snapshot.Paths, targets) && (hostname == "" || hostname == snapshot.Hostname) {
if snapshot.Time.After(latest) && snapshot.HasPaths(targets) && (hostname == "" || hostname == snapshot.Hostname) {
latest = snapshot.Time
latestID = snapshotID
found = true

2
vendor/manifest vendored
View file

@ -52,7 +52,7 @@
{
"importpath": "github.com/pkg/xattr",
"repository": "https://github.com/pkg/xattr",
"revision": "1d40b70a947cd8e8457e4715e1123f8e99f5f241",
"revision": "b867675798fa7708a444945602b452ca493f2272",
"branch": "master"
},
{

View file

@ -1,3 +1,7 @@
[![GoDoc](https://godoc.org/github.com/pkg/xattr?status.svg)](http://godoc.org/github.com/pkg/xattr)
[![Go Report Card](https://goreportcard.com/badge/github.com/pkg/xattr)](https://goreportcard.com/report/github.com/pkg/xattr)
[![Build Status](https://travis-ci.org/pkg/xattr.svg?branch=master)](https://travis-ci.org/pkg/xattr)
xattr
=====
Extended attribute support for Go (linux + darwin + freebsd).
@ -10,12 +14,12 @@ Extended attribute support for Go (linux + darwin + freebsd).
const path = "/tmp/myfile"
const prefix = "user."
if err = Setxattr(path, prefix+"test", []byte("test-attr-value")); err != nil {
t.Fatal(err)
if err := xattr.Setxattr(path, prefix+"test", []byte("test-attr-value")); err != nil {
log.Fatal(err)
}
var data []byte
data, err = Getxattr(path, prefix+"test"); err != nil {
t.Fatal(err)
data, err = xattr.Getxattr(path, prefix+"test"); err != nil {
log.Fatal(err)
}
```

View file

@ -1,3 +1,10 @@
/*
Package xattr provides support for extended attributes on linux, darwin and freebsd.
Extended attributes are name:value pairs associated permanently with files and directories,
similar to the environment strings associated with a process.
An attribute may be defined or undefined. If it is defined, its value may be empty or non-empty.
More details you can find here: https://en.wikipedia.org/wiki/Extended_file_attributes
*/
package xattr
// XAttrError records an error and the operation, file path and attribute that caused it.
@ -12,8 +19,7 @@ func (e *XAttrError) Error() string {
return e.Op + " " + e.Path + " " + e.Name + ": " + e.Err.Error()
}
// Convert an array of NULL terminated UTF-8 strings
// to a []string.
// nullTermToStrings converts an array of NULL terminated UTF-8 strings to a []string.
func nullTermToStrings(buf []byte) (result []string) {
offset := 0
for index, b := range buf {

View file

@ -2,44 +2,47 @@
package xattr
// Retrieve extended attribute data associated with path.
// Getxattr retrieves extended attribute data associated with path.
func Getxattr(path, name string) ([]byte, error) {
// find size.
size, err := getxattr(path, name, nil, 0, 0, 0)
if err != nil {
return nil, &XAttrError{"getxattr", path, name, err}
}
buf := make([]byte, size)
// Read into buffer of that size.
read, err := getxattr(path, name, &buf[0], size, 0, 0)
if err != nil {
return nil, &XAttrError{"getxattr", path, name, err}
if size > 0 {
buf := make([]byte, size)
// Read into buffer of that size.
read, err := getxattr(path, name, &buf[0], size, 0, 0)
if err != nil {
return nil, &XAttrError{"getxattr", path, name, err}
}
return buf[:read], nil
}
return buf[:read], nil
return []byte{}, nil
}
// Retrieves a list of names of extended attributes associated with the
// given path in the file system.
// Listxattr retrieves a list of names of extended attributes associated
// with the given path in the file system.
func Listxattr(path string) ([]string, error) {
// find size.
size, err := listxattr(path, nil, 0, 0)
if err != nil {
return nil, &XAttrError{"listxattr", path, "", err}
}
if size == 0 {
return []string{}, nil
}
if size > 0 {
buf := make([]byte, size)
// Read into buffer of that size.
read, err := listxattr(path, &buf[0], size, 0)
if err != nil {
return nil, &XAttrError{"listxattr", path, "", err}
buf := make([]byte, size)
// Read into buffer of that size.
read, err := listxattr(path, &buf[0], size, 0)
if err != nil {
return nil, &XAttrError{"listxattr", path, "", err}
}
return nullTermToStrings(buf[:read]), nil
}
return nullTermToStrings(buf[:read]), nil
return []string{}, nil
}
// Associates name and data together as an attribute of path.
// Setxattr associates name and data together as an attribute of path.
func Setxattr(path, name string, data []byte) error {
if err := setxattr(path, name, &data[0], len(data), 0, 0); err != nil {
return &XAttrError{"setxattr", path, name, err}
@ -47,7 +50,7 @@ func Setxattr(path, name string, data []byte) error {
return nil
}
// Remove the attribute.
// Removexattr removes the attribute associated with the given path.
func Removexattr(path, name string) error {
if err := removexattr(path, name, 0); err != nil {
return &XAttrError{"removexattr", path, name, err}

View file

@ -10,40 +10,46 @@ const (
EXTATTR_NAMESPACE_USER = 1
)
// Retrieve extended attribute data associated with path.
// Getxattr retrieves extended attribute data associated with path.
func Getxattr(path, name string) ([]byte, error) {
// find size.
size, err := extattr_get_file(path, EXTATTR_NAMESPACE_USER, name, nil, 0)
if err != nil {
return nil, &XAttrError{"extattr_get_file", path, name, err}
}
buf := make([]byte, size)
// Read into buffer of that size.
read, err := extattr_get_file(path, EXTATTR_NAMESPACE_USER, name, &buf[0], size)
if err != nil {
return nil, &XAttrError{"extattr_get_file", path, name, err}
if size > 0 {
buf := make([]byte, size)
// Read into buffer of that size.
read, err := extattr_get_file(path, EXTATTR_NAMESPACE_USER, name, &buf[0], size)
if err != nil {
return nil, &XAttrError{"extattr_get_file", path, name, err}
}
return buf[:read], nil
}
return buf[:read], nil
return []byte{}, nil
}
// Retrieves a list of names of extended attributes associated with the
// given path in the file system.
// Listxattr retrieves a list of names of extended attributes associated
// with the given path in the file system.
func Listxattr(path string) ([]string, error) {
// find size.
size, err := extattr_list_file(path, EXTATTR_NAMESPACE_USER, nil, 0)
if err != nil {
return nil, &XAttrError{"extattr_list_file", path, "", err}
}
buf := make([]byte, size)
// Read into buffer of that size.
read, err := extattr_list_file(path, EXTATTR_NAMESPACE_USER, &buf[0], size)
if err != nil {
return nil, &XAttrError{"extattr_list_file", path, "", err}
if size > 0 {
buf := make([]byte, size)
// Read into buffer of that size.
read, err := extattr_list_file(path, EXTATTR_NAMESPACE_USER, &buf[0], size)
if err != nil {
return nil, &XAttrError{"extattr_list_file", path, "", err}
}
return attrListToStrings(buf[:read]), nil
}
return attrListToStrings(buf[:read]), nil
return []string{}, nil
}
// Associates name and data together as an attribute of path.
// Setxattr associates name and data together as an attribute of path.
func Setxattr(path, name string, data []byte) error {
written, err := extattr_set_file(path, EXTATTR_NAMESPACE_USER, name, &data[0], len(data))
if err != nil {
@ -55,7 +61,7 @@ func Setxattr(path, name string, data []byte) error {
return nil
}
// Remove the attribute.
// Removexattr removes the attribute associated with the given path.
func Removexattr(path, name string) error {
if err := extattr_delete_file(path, EXTATTR_NAMESPACE_USER, name); err != nil {
return &XAttrError{"extattr_delete_file", path, name, err}
@ -63,7 +69,7 @@ func Removexattr(path, name string) error {
return nil
}
// Convert a sequnce of attribute name entries to a []string.
// attrListToStrings converts a sequnce of attribute name entries to a []string.
// Each entry consists of a single byte containing the length
// of the attribute name, followed by the attribute name.
// The name is _not_ terminated by NUL.

View file

@ -4,40 +4,46 @@ package xattr
import "syscall"
// Retrieve extended attribute data associated with path.
// Getxattr retrieves extended attribute data associated with path.
func Getxattr(path, name string) ([]byte, error) {
// find size.
size, err := syscall.Getxattr(path, name, nil)
if err != nil {
return nil, &XAttrError{"getxattr", path, name, err}
}
data := make([]byte, size)
// Read into buffer of that size.
read, err := syscall.Getxattr(path, name, data)
if err != nil {
return nil, &XAttrError{"getxattr", path, name, err}
if size > 0 {
data := make([]byte, size)
// Read into buffer of that size.
read, err := syscall.Getxattr(path, name, data)
if err != nil {
return nil, &XAttrError{"getxattr", path, name, err}
}
return data[:read], nil
}
return data[:read], nil
return []byte{}, nil
}
// Retrieves a list of names of extended attributes associated with the
// given path in the file system.
// Listxattr retrieves a list of names of extended attributes associated
// with the given path in the file system.
func Listxattr(path string) ([]string, error) {
// find size.
size, err := syscall.Listxattr(path, nil)
if err != nil {
return nil, &XAttrError{"listxattr", path, "", err}
}
buf := make([]byte, size)
// Read into buffer of that size.
read, err := syscall.Listxattr(path, buf)
if err != nil {
return nil, &XAttrError{"listxattr", path, "", err}
if size > 0 {
buf := make([]byte, size)
// Read into buffer of that size.
read, err := syscall.Listxattr(path, buf)
if err != nil {
return nil, &XAttrError{"listxattr", path, "", err}
}
return nullTermToStrings(buf[:read]), nil
}
return nullTermToStrings(buf[:read]), nil
return []string{}, nil
}
// Associates name and data together as an attribute of path.
// Setxattr associates name and data together as an attribute of path.
func Setxattr(path, name string, data []byte) error {
if err := syscall.Setxattr(path, name, data, 0); err != nil {
return &XAttrError{"setxattr", path, name, err}
@ -45,7 +51,8 @@ func Setxattr(path, name string, data []byte) error {
return nil
}
// Remove the attribute.
// Removexattr removes the attribute associated
// with the given path.
func Removexattr(path, name string) error {
if err := syscall.Removexattr(path, name); err != nil {
return &XAttrError{"removexattr", path, name, err}