• Add an example configuration
• Check/write README usage docs
• Double-check all API docs
• Double-check license compliance
• cargo publish

Allow an environment to require other environments to be true.

The simplest example is unix = ["linux", "freebsd", "macos", ...] where each one is an environment matching the appropriate os.

A more useful example is personal_machine = ["my_laptop", "my_desktop", ...] so that personal files don't accidentally get scattered around a work machine that I use programming dotfiles on.

Will definitely add some complexity to how environments are processed.

• Switch from chrono to time causes deserialization errors with operation logs
  • 0.3.1 release needs to be 0.4.0 instead

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• fix(deps): update rust crate petgraph to v0.6.5
• chore(deps): update knope-dev/action action to v2.1.0
• chore(deps): update rust crate nix to 0.29
• chore(deps): update rust crate tokio to v1.38.0
• fix(deps): update rust crate clap to v4.5.7
• fix(deps): update rust crate hostname to 0.4
• fix(deps): update rust crate itertools to 0.13.0
• fix(deps): update rust crate once_cell to v1.19.0
• fix(deps): update rust crate regex to v1.10.5
• fix(deps): update rust crate similar to v2.5.0
• fix(deps): update rust crate tempfile to v3.10.1
• fix(deps): update rust crate uuid to v1.8.0
• fix(deps): update rust crate windows to 0.57
• chore(deps): update actions/checkout action to v4
• chore(deps): update codecov/codecov-action action to v4
• chore(deps): update dorny/paths-filter action to v3
• chore(deps): update fossa-contrib/fossa-action action to v3
• chore(deps): update nwtgck/actions-netlify action to v3
• chore(deps): update rust crate serial_test to v3
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update rust crate futures to v0.3.30
• fix(deps): update rust crate async-trait to v0.1.80
• fix(deps): update rust crate serde to v1.0.203
• fix(deps): update rust crate serde_json to v1.0.117
• fix(deps): update rust crate serde_yaml to v0.9.34
• fix(deps): update rust crate thiserror to v1.0.61
• fix(deps): update rust crate time to v0.3.36
• fix(deps): update rust crate tokio-stream to v0.1.15
• fix(deps): update rust crate toml to v0.8.14
• fix(deps): update tokio-tracing monorepo (tracing, tracing-subscriber)
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Needed for #5 and blocked by #8.

The current basic design assumes either a single file or a single directory of which everything will be stored.

This needs a way to specify multiple files/folders for a target (easy) and a way to define where those files/folders can be found in different environments (harder).

As a simple example: Vim/Neovim on Windows/Linux.

• Vim:
  • Linux: $HOME/.vimrc and ~/.vim
  • Windows: $HOME/_vimrc and $HOME/.vim
• Neovim:
  • Linux: $HOME/.config/nvim/init.vim and $HOME/.config/nvim
  • Windows: $HOME/AppData/Local/nvim/init.vim and $HOME/AppData/Local/nvim

Since one configuration can be shared between all four setups (with minor conditionals), I want to have one "hoarded" version of the configuration but correctly apply it based on environment.

The application is easy enough: copy from the hoard onto the filesystem. But how do we store these in the first place? Naively using the top-level name doesn't work if there are any conflicts. So some sort of unique id seems necessary.

How would we derive an id?

• A hash of the file path doesn't work, since there are multiple paths based on environment
• A hash of the first or of all paths together doesn't work, because order can change and a new path can be introduced at any time, both of which could change the hash.
• Can't use the file contents, because those will change.

Perhaps how we set up configuration can make a difference. So what would configuration look like?

[environments]
[environments.winvim]
# ...
[environments.linvim]
# ...
[environments.winneo]
# ...
[environments.linneo]
# ...

[hoard]
[hoard.vim]
    [[hoard.vim.files]]
        id = "rcfile"
        envs = {
            winvim = "$HOME/_vimrc",
            linneo = "$HOME/.config/nvim/init.vim",
            # ...
        }

... Maybe something else? Maybe using JSON instead?

• list [hoard(s)]: list all configured hoards
• list pile(s) <hoard>: list all piles in hoard
  • Optionally list the paths that match for each one on this machine?
• diff: list all files that differ between the hoard and this machine
• edit: opens the Hoard configuration file in the default editor

First part of applying #5.

I'd like to make the following variables available for defining an environment.

• Hostname
• Operating System
• Environment Variable values
• Existence of Program
• Existence of File/Folder

Finally, I'd like to have a configuration item/command-line flag for overriding the detected environments, and another for determining whether to detect any when such a list is given.

If there is a large directory that I would like to backup that contains some folders within it that I'd like to ignore, it would be nice to be able to provide some sort of regex to ignore that pattern. The specific case I am referring to is that I am backing up my vimwiki directory and it contains a .git folder. On the first go-around it works perfectly fine; however, whenever I try to back it up a second time once something has changed, I get this error:

ERRORfailed to back up vimwiki: failed to copy /Users/lucasburns/vimwiki/music/.git/objects/0e/b565c87706e00
ec8efd3834f074db5175210e7 to /Users/lucasburns/.config/hoard/root/vimwiki/dir/music/.git/objects/0e/b565c87706
e00ec8efd3834f074db5175210e7: Permission denied (os error 13)
    at src/main.rs:16

If there is something I can do already to fix this, I can try and see if that would work.

I do not think I have implemented this yet, and I definitely want to make sure there are tests to cover this. If I delete a configuration file (e.g. under a conf.d directory), that deletion should propagate to the hoard on next backup.

I'm thinking something like using gitoxide to turn the saves folder into a git repository. Then we can do something like chezmoi init and download the git repo on new machines, make snapshots of the saves at different points in time, etc.

This includes listing hoards and piles within hoards

Should probably happen after #13.

Encryption using public/private key pairs. Probably PGP.

I'll need to do some investigation into the best ways to implement this in Rust.

I think these are related enough to put into an issue together.

The idea here is to detect differences between the hoard and the filesystem. hoard diff then lists all of the files that differ and hoard status lists just the hoards and whether the current machine was the last to touch them (hints at whether to back up or restore).

There may be an argument for showing a literal diff of textual files as well. This would require finding a library to find diffs and finding the best way to invoke a pager for scrolling (or just let the user pipe to it?)

config is a reserved name for hoard/pile-level configuration. The error from trying to parse a pile named config is unhelpful in communicating as much. I do not have the message in front of me, but it is along the lines of not being able to parse a Hoard struct.

There should be a special case that, if parsing fails, checks for something named config that doesn't look like a configuration.

Let's start with just attaching release builds to a tagged release. Things like package manager integrations can come later.

I believe both log and tracing have a way to indicate the context of a logged message. I already cannot easily tell what hoard/pile a message relates to, and the program is currently synchronous. Adding some context to the logs will make things much clearer.

This is partially needed for #75.

• Actually use Checksum enum
• Use a files iterator
• Record created, modified, and deleted files
  • Is there any way to track permissions safely?
• Provide backwards compatibility path

The logs created by the operation history checker will take up space over time. hoard should provide a way to clean up old logs that are not useful to the checker.

This command could also be run automatically, but we should start with only manually.

It seems to be a common request, based on anecdotal evidence from announcing the last couple of releases on Reddit.

I initially chose TOML as the configuration file syntax because it is unambiguous, commonly used among and outside of Rust projects, and more readable than JSON.

From my understanding, YAML has ambiguity (different parsers can parse the same YAML differently) and can be frustrating to use with certain configuration structures, Still, I am willing to consider the feature if I feel it should add something.

For this issue, I am going to take my current configuration (which should be sufficiently long and complex to make the comparison) and translate it to YAML. If I feel there is a significant improvement in readability and/or writability, I may consider also supporting YAML config files.

I need to confirm this, but I was having some issues with files not being matched that should have been. Will need to update the relevant integration test too.

• Code coverage
• Tests
• Automatic builds/artifacts on release

The README is getting rather long. Better to make a proper docs site for it.

Imported from Gitea

Both config.toml and games.toml are going to be in the system-specific configuration directory, which is different from where the Git repository will be stored. This means, though, that configurations will not be synced with the saves.

A good solution is to add a sync subcommand to config and game, and have their add and remove subcommands automatically call sync.

Just tried setting up a new system, got the following output:

   INFO loading configuration...
    at src/config/mod.rs:114

   INFO loaded configuration.
    at src/config/mod.rs:118

   WARN path is neither file nor directory, skipping, path: /home/shadow53/.config/hoard/config.toml
    at src/checkers/history/operation.rs:331

  ERROR error while saving uuid to file, error: No such file or directory (os error 2)
    at src/checkers/history/mod.rs:102
    in running_checks

  ERROR error while saving uuid to file, error: No such file or directory (os error 2)
    at src/checkers/history/mod.rs:102
    in running_checks

  ERROR error while checking against recent remote operations: an I/O error occurred: No such file or directory (os error 2)
    at src/main.rs:16

Some users may need step-by-step instructions for setting up their system(s) with Hoard,

Version: 0.4.0
Platform: x86_64-unknown-linux-gnu

I have a hoard named bash that contains two piles:

# Bash rcfile
[hoards.bash.rcfile]
"linux" = "${HOME}/.bashrc"

# Bash aliases directory
[hoards.bash.aliases]
"linux" = "${HOME}/.bash_aliases.d/"

The file structure at the hoards root looks like this:

bash/:
drwxrwxr-x  3 dev dev 4.0K Dec 31 14:33 aliases/
drwxrwxr-x  2 dev dev 4.0K Dec 31 14:33 rcfile/

bash/aliases:
drwxrwxr-x 2 dev dev 4.0K Dec 31 01:05 .bash_aliases.d/

bash/aliases/.bash_aliases.d:
-rw-rw-r-- 1 dev dev  349 Dec 31 01:04 00-primary-aliases.sh

bash/rcfile:
-rw-r--r-- 1 dev dev 1.9K Dec 31 01:34 .bashrc

However, when I run hoard restore bash I end up with the following directory structure under $HOME:

❯ hoard restore bash
ignoring ``: invalid filter directive
WARN hoard::checkers::history::operation: path is neither file nor directory, skipping: path=/home/<user>/.bash_aliases.d
WARN hoard::checkers::history::operation: path is neither file nor directory, skipping: path=/home/<user>/.bashrc
restoring hoard: hoard=bash

 ❯ la ~/ | grep bash
drwxrwxr-x  3 dev  dev  4.0K Jan  1 00:11 .bash_aliases.d/
drwxrwxr-x  2 dev  dev  4.0K Jan  1 00:11 .bashrc/

 ❯ la ~/.bashrc
-rw-rw-r--  1 dev dev 1.9K Jan  1 00:11 .bashrc

 ❯ la -R ~/.bash_aliases.d/
/home/dev/.bash_aliases.d/:
drwxrwxr-x  2 dev dev 4.0K Jan  1 00:11 .bash_aliases.d/

/home/dev/.bash_aliases.d/.bash_aliases.d:
-rw-rw-r-- 1 dev dev  349 Jan  1 00:11 00-primary-aliases.sh

In other words, hoard creates an extra level of directories instead of placing my bashrc in ${HOME}/.bashrc and my aliases in ${HOME}/.bash_aliases.d/. Am I configuring hoard incorrectly or is this a bug?

Imported from Gitea

Integration tests would be useful to have, to verify that things are working correctly during actual use.

The only thing I am not sure about is once Git support is added, since that adds extra complexity to every operation.

As part of this, I am also making sure all possible unit tests are implemented.

Current status:

• backup.rs
  • copy_path
  • copy_game
  • backup
  • restore
• config
  • builder.rs
  • command.rs
    • Command::default == Help
    • Command::run
      • Help -- how to test?
      • Backup
      • Restore
      • ConfigCmd
      • GameCmd
  • config.rs
    • ConfigField FromStr
    • ConfigField Display
    • Command::init
    • Command::run
      • Init
      • Reset
      • Set
      • Unset
  • game.rs
    • AddGame::add_game
    • RemoveGame::remove_game
    • Command::run
  • mod.rs
    • Config::default == build empty builder
    • Config::builder returns new builder
    • Config::games reads games file
    • Config::get_config_file_path() returns config path
    • Config::get_saves_root_path() returns saves root path
    • Config::get_games_file_path() returns games file path
• games.rs
• lib.rs
• main.rs

I have filled in the remaining unit tests. This leaves the following items for integration testing:

• backup.rs
  • copy_path
  • copy_game
  • backup
  • restore
• config
  • command.rs
    • Help
    • Backup
    • Restore
    • ConfigCmd
    • GameCmd
  • config.rs
    • Init
    • Run
      • Init
      • Reset
      • Set
      • Unset
  • game.rs
    • add_game
    • remove_game
    • run
  • mod.rs
    • Config::get_games

Since integration tests are meant to test the public API of a program/library, and because the library is meant to only be consumed by the application (will make an issue for that), that leaves the following integration test(s):

• Create configuration file with config commands (also removing, reset, etc)
• Create games file with game commands (also removing)
• Backup saves
• Restore saves

As a side note, I'm not sure how to test the help command.

Currently we hoard all or nothing. We should include a way to specify individual targets.

Include in release notes:

• No logging config option. Using HOARD_LOG env variable instead.
• Documentation is now available on https://hoard.rs.
• Consistency checks have been added and may prevent certain operations from succeeding. They also create files in hidden folders.

The purpose of hoard is to make it easy to synchronize random scattered files across machines, so it would be useful to have a way to detect the presence of unsynced changes.

Current Thoughts

• One or more files in hoards_root that define the last time a hoard was backed up/a log of X recent backups
• A similar file in a cache location, e.g. $HOME/.cache/hoard on *nix systems
• If there is a difference in the two files, inform the user
  • Four obvious cases:
    1. Local is more recent: warn on restore
    2. Remote/backup is more recent: warn on backup
    3. Both local and remote have divergent changes: inform user and let them handle conflicts
    4. Files are in sync: no changes or warnings

In the cases of warnings, the operation would abort until it is resolved. Users can force resolution by using a --force flag.

If files are being restored for the first time, the record file would get copied to the cache location.

If done right, this should have an added benefit of allowing users to incorporate hoard in shell rc files and scripts, where they will do nothing if there might be problems.

Command should open the configuration file in $EDITOR or the system's default text editor.

System:

Darwin void.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64 i386 MacBookPro14,1 Darwin

I have gotten this error on a couple of the hoards I have set:

 $ hoard -l trace validate                                                           【28s  ~/.conf/hoard  job】─╯
[2021-06-04T23:45:24Z INFO  hoard::config] Loading configuration...
[2021-06-04T23:45:24Z ERROR hoard] error while building the configuration: failed to parse configuration file: data did not match any variant of untagged enum Hoard for key `hoards.gituip` at line 195 column 5

This was one of the sections it got hungup on:

[hoards.gitui]
    [hoards.gitui.conf]
        "macos" = "/Users/lucasburns/.config/gitui/key_config.ron"
    [hoards.gituip.theme]
        "macos" = "/Users/lucasburns/.config/gitui/theme.ron"

It also got hungup on another one where the hoard was titled [hoard.bat] with several sections underneath. On this one (bat) I changed the name to batp on the parent hoard and changed the name on all of the children hoards and it worked.

On the gitui one, I tried the same thing but still got the same error. I took out the parent and left only one child, still got the error. Then i changed the name to something random like asdfasd and it worked. I then changed it back to [hoard.gitui] and then changed all the children back to gitui and it works now. There was nothing else that I changed.

Steps to reproduce:

Error is thrown here:

[hoards.lazygit]
    [hoards.lazygit.config]
        "macos" = "/Users/lucasburns/.config/lazygit/config.yml"
    [hoards.lazygit.state]
        "macos" = "/Users/lucasburns/.config/lazygit/state.yml"

Change to this and still get error:

[hoards.lazygit.config]
    "macos" = "/Users/lucasburns/.config/lazygit/config.yml"

Change to this and no error:

[hoards.lgasdf]
    "macos" = "/Users/lucasburns/.config/lazygit/config.yml"

Add on test to make sure I can use ., still no error here:

[hoards.lgasdf.test]
    "macos" = "/Users/lucasburns/.config/lazygit/config.yml"

Change back to lazygit and it still works:

[hoards.lazygit.test]
    "macos" = "/Users/lucasburns/.config/lazygit/config.yml"

Version: 0.4.0
Platform: x86_64-unknown-linux-gnu

Observed Behavior

> hoard --help
ignoring ``: invalid filter directive
hoard 0.4.0
The possible subcommands for `hoard`

USAGE:
    hoard [FLAGS] [OPTIONS] [SUBCOMMAND]

FLAGS:
    -f, --force      
        --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -c, --config-file <config-file>    
    -h, --hoards-root <hoards-root>    

SUBCOMMANDS:
    backup      Back up the given hoard(s)
    cleanup     Cleans up the operation logs for all known systems
    help        Prints this message or the help of the given subcommand(s)
    list        List configured hoards
    restore     Restore the files from the given hoard to the filesystem
    validate    Loads all configuration for validation. If the configuration loads and builds, this command succeeds

Notice the ignoring ``: invalid filter directive message at the top (printed to stderr). This happens on every invocation of hoard.

Expected Behavior

Running hoard --help should only produce the help message.

Steps to Reproduce

1. Download archive from GitHub and unzip it.
2. Run hoard --help

SInce multiple hoards can be processed at once and each one may have multiple files, it may make sense to have multiple operations perform at once.

Not sure if this is best done with an async runtime like Tokio or making some background worker threads myself. A worker queue of some sort makes sense. I'll have to look into what is simplest to implement.

This serves as a meta-issue for both #12 and #21.

Track:

1. Files updated by remote system
2. Files updated by local system
3. Local files that differ from hoard
4. Paths previously used for each pile
5. File checksums?

Usage:

• (4) Prevent unintentional file deletions on backup because a different path got matched. (#21)
• (3 + 5) Future implementation of a hoard diff command. (#12)
• (3 + 5) Prevent overwriting local changes with hoard restore (use last local checksum) (#21)
• (1 + 2) Prevent overwriting remote changes with hoard backup (#21)

TODO:

• Implement previous-path file and related check (see #21 for possible format)
• Generate per-device UUIDs for log files
• Implement log file format that tracks files that have changed
  • may have to use checksums to avoid marking unchanged files as changed
  • Also track file creation/deletion
  • Include timestamp
• Implement parsing and combining log files from multiple devices to determine where a file was last modified
• Implement checks based on where a file was last modified

The new integration test in 26-prevent-file-footguns (#26) is failing because a Windows-specific environment variable is not set on Linux. Delaying the processing of the path will fix this.

1. Expand both in all paths in piles and in path_exists conditions.
2. Parsing rules:
   • ${ENV_VARIABLE}
   • \${NOT_EXPANDED}
   • $NOT_EXPANDED (error/warning?)
   • ${VAR_CONTAINS{_BRACE}
3. How to handle non-existent variables? (Different from exists but equals "")
   • Return an error
   • The alternative is defaulting to an empty string or known backup values (e.g. for HOME, XDG_*_HOME, etc.)
     • Already using directories which depends on dirs, and those can be used to detect values for XDG directories and such
     • There would need to be some mapping between common variables and ways to get fallback values if not set

#40 just got fixed, which greatly improved calculated code coverage by including the Python-based integration tests. I'd like to further improve code coverage, with a goal of >90% coverage, 95% if possible.

Right now, no concern is given to when files are deleted.

• If an entire pile is deleted from the filesystem and user runs hoard backup, the copying process will fail with a warning
  • Same with a pile being deleted from the hoards folder and user runs hoard restore.
• If a single file from a directory pile is deleted from the filesystem, it is not copied on backup (because it does not exist) but will be restored with hoard restore.
  • Similar for the opposite situation.

This brings up some questions:

(How) do we detect a deleted file?

This question is related to #12 and should primarily be discussed there. As far as this issue is concerned, there is the possibility of knowing that a file was deleted from a pile (e.g. by a different machine), a separate possibility of knowing a file was deleted from the filesystem, and not knowing the previous state at all.

What do we do if we discover a file has been deleted/is missing?

Depends on implementation details.

We have some record that the file existed and now doesn't

• Do we know the user intentionally deleted the file? The answer is probably no. The safest option is to warn the user and abort, and provide option flags that prevent the abort in favor of one of the following:
  • Ignore missing files: Basically perform the operation, but skip any files marked as missing. This means no deleting the version that still exists (e.g., the backup when backing up) and no replacing the file that was deleted.
  • Keep deleted files: Do not delete any files, but replace missing ones as applicable.
  • Force deletion: Do not replace any files, but delete "orphans".
• This means extra cognitive effort on the part of the user to remember to use the appropriate flag or always run the command twice.
• Alternatively, there is the possibility of adding an interactive piece to the program where the user can tell hoard whether a file was intentionally deleted.

We don't have such a record

This is similar to the above, but we no longer know the difference between a newly created file and a newly deleted one.

• Warn/etc. whenever the wrong file exists, i.e. only the backup exists while backing up or the original exists while restoring. Provide a --force flag to make the operation happen anyways.
• Don't care and propagate possible deletions, i.e. delete the backup if backing up while the original file is missing. Assume the user has good backups/is using git.

Missing files and checking if a clean backup/restore can happen

Also related to #12 and the possibility of having commands to check if a clean backup or restore can happen (i.e. no changes will be clobbered). I think the following applies:

• We keep records of file existence:
  • If the file did not previously exist, copying existing file to other location is considered "clean"
  • If it did previously exist, this is a potential conflict and needs to be manually resolved.
• We do not keep records:
  • Retaining the existing file is considered "clean", deleting the file is not.
  • Not caring is always considered "clean". At that point, though, a check for a clean application does not make sense.

There may be circumstances where merging configs may be difficult to figure out, but for right now it is simple enough and provides the least surprise to users.

There isn't much of anything specific to game saves for this program. Generalizing a bit, the feature set looks like this:

• Define files/folders for a target/environment combination
• (Future item) Define a custom environment
  • Various rules for the environment
  • An option for the environment to apply whenever it matches or only when it is the most specific one
    • Implies some level of filesystem layering
• Bidirectional sync of files
• (Optional future item) Encryption

Definitions:

• Environment: the computer the program is running on
• Target: A specific folder/game/etc.

Integration tests provide most of the testing coverage now. If possible, I'd like to run the Python test scripts and generate coverage that can be uploaded to Codecov to more accurately reflect testing coverage.

There is a little bit of configuration support for this already, but no actual support.

Essentially, backed up files will get encrypted using a given password, and restored files will be decrypted by the same.

There will be two sources of password: a raw password password = "" and a password command to retrieve it from somewhere else.

I'll need to do research into the best encryption to use for both speed and security. AES, probably.