When I run install.sh, the directory gets declared as ., resulting in a symlink path of ./asimov inside /usr/local/bin. This causes the script to fail as it can't find the correct path of the executable inside the repo.

Changing to $(pwd) inside the installer fixes this, so the symlink points to the full path and not the relative one.

It's very common for people to create a venv or .venv folder, using virtualenv or python -m venv, alongside the requirements.txt file for Python

    'venv requirements.txt'       # virtualenv (python)
    '.venv requirements.txt'      # virtualenv (python)

...
...
- /Users/vadorequest/.cache/act/[email protected]/node_modules/@babel/core/node_modules has been excluded from Time Machine backups (872K).
- /Users/vadorequest/.cache/act/[email protected]/node_modules/@babel/generator/node_modules has been excluded from Time Machine backups (792K).
- /Users/vadorequest/.cache/act/jwalton-gh-find-current-pr@v1/node_modules has been excluded from Time Machine backups (3.2M).

Displaying a summary of the total size that's been ignored would be great! It's the best indicator about how useful this utility is, as it would show how much space will be gained in the future and how much time will be saved by not copying those useless files 😉

Shouldn't the .plist file be copied to ~/Library/LaunchAgents and then be loaded from there?

I know that it technically works when it's loaded from within the project folder, but doing so goes against the grain -- and may be silently disallowed by Apple at some point in the future.

Hi there! Cool project, it worked great.

Is there a way to erase the changes made by this program and back up all (not otherwise excluded folders) from the backup?

It would be really nice if the script could also add the folders to the exclusion list of Spotlight…

I was wondering if this script would cause issue when restoring the back for apps that have one of the excluded patterns in their folder. For example, Electron-based apps such as Atom have a node_module folder:
~/Applications/Atom.app/Contents/Resources/app/apm/node_modules

If this is getting excluded, then upon restoring the backup, won't it malfunction?

This looks very interesting...can it be setup to exclude .git directories as well?

Apologies for asking, but I don't understand how any of this works, wandering in a black hole ...

1. When I launch Time Machine all my projects, vendor and node_modules are still there.
2. When I run sudo mdfind "com_apple_backup_excludeItem = 'com.apple.backupd' none of the vendor/node_modules, dirs are listed.

I launched the daemon a few days ago and thought that the newer backups wouldn't include those folders?

• I followed the installation instructions.
  • Got "Asimov daemon has been loaded!" in terminal
  • In the list generated from the process, I could see my projects in the "Sites" dir being added for exclusion.

My "web" projects are either in the "Sites" folder or in my "Github" folder on my mac.
Each of them have node_modules and vendor folders ...

Lot of people have their .gitignore setup which already removes files

Maybe it could solve something like https://stackoverflow.com/questions/50788796/how-to-exclude-or-use-global-gitignore-file-and-directories-from-timemachine-ba

Just ran into this, might even PR

For example, ~/.atom/packages/* should probably be included in the backup since it will otherwise break atom upon restoring and won't have that many files anyway (no babel/webpack/etc. necessary, packages normally reasonably small/change rarely).

Alternatively, a configurable entry-point would be great. (In my case, that's ~/Code).

Why don‘t you add the -p flag, so it will not be sticky?

Thanks for that tool, exactly something I was looking for.

One thing that may make it even better would be skipping some common folders while searching for development dependencies like:

• ~/Library
• ~/Applications
• ~/Music
  etc.

The /Users/{user}/.Trash directory is already (ideally) excluded from Time Machine backups, but we should make sure the Asimov script doesn't waste its time checking those directories.

Not really sure what this means, but I ran into it at: /Users/username/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/underscore-rails-1.8.3/vendor

The project should have a formal change log, keeping with the Keep a Changelog standard.

Thanks for this script - I found it while waiting days for my first time machine backup to process!

It seems to work well but is hardcoded to look in ~/Sites. My code lives in ~/Code... I edited the script but it would be nice if this was in some way configurable...

Asimov has received a lot of attention, but I haven't had much time as-of-late to dedicate to the project.

If there's another developer out there (preferably with experience writing performant bash scripts) that would like to volunteer to join me in maintaining this project, please get in touch!

Package managers will typically create a cache directory in order to prevent the same packages from being downloaded over and over again. For example, Composer will store these in ~/.composer/cache.

Since these too are development dependencies that can easily be restored, they should be omitted from Time Machine backups.

It's worth noting that these are often configurable, so if the given tools are available we may want to try to query their configurations.

What other cache directories are hiding out in our home directories?

hello - nice tool. However I installed it without realising there is no documentation for removing excludes and uninstalling.

How would I go about this?

thanks, D.

The project should contain a CONTRIBUTING.md file, describing how people can get involved (particularly with regards to contributing other dependency patterns).

B/c I always forget these things:

I cd'd into my home directory, cloned this repo, then ran the install script. For anyone who isn't 100% confident in doing all that, it looks like this:

1. Open Terminal, run cd. This will take you to your home directory.
2. Git clone the repo as described in the instructions
3. cd asimov to enter that directory
4. Run the install script by typing: ./install.sh

That's it!

Thanks Steve!

Hi. I've made some little app for list all excluded folders. May be it can be helpful.
https://github.com/cnstntn-kndrtv/TimeMachine-Exclusions

Hi, I wonder if this is still current on Monterey, as Homebrew installs now on /opt/homebrew rather than /usr/bin.

It would be great if it did, because I don't want to send to my NAS all node_modules and virtual environments!

Thanks!

Since Asimov is a macOS-specific utility for developers, it would be great if it could easily be installed via Homebrew.

I've worked through the Homebrew Formula Cookbook and put together a formula that's syntactically correct:

class Asimov < Formula
  desc "Automatically exclude development dependencies from Time Machine backups"
  homepage "https://github.com/stevegrunwell/asimov"
  url "https://github.com/stevegrunwell/asimov/archive/v0.1.0.tar.gz"
  sha256 "0fc0bf66f16ecb94ab711f92ac9107f301f50ec98a28e141d7b14f006d1f13b8"
  head "https://github.com/stevegrunwell/asimov.git", :branch => "develop"

  def install
    bin.install "asimov"
  end

  test do
    system "#{bin}/asimov"
  end
end

Unfortunately (for me), Homebrew seems to put some requirements on new formulas:

$ brew audit --new-formula asimov
asimov:
  * GitHub repository not notable enough (<20 forks, <20 watchers and <50 stars)
  * GitHub repository too new (<30 days old)
Error: 2 problems in 1 formula

I'll hold onto the formula for now and hope a blog post helps get some traction on Asimov. In the meantime, uh...stars, watchers, and forks are welcome! 😄

Before:

brew services
Name      Status       User  File
asimov    none         root

Then, starting the service:

sudo brew services start asimov
Warning: Taking root:admin ownership of some asimov paths:
  /opt/homebrew/Cellar/asimov/0.3.0/bin
  /opt/homebrew/Cellar/asimov/0.3.0/bin/asimov
  /opt/homebrew/opt/asimov
  /opt/homebrew/opt/asimov/bin
  /opt/homebrew/var/homebrew/linked/asimov
This will require manual removal of these paths using `sudo rm` on
brew upgrade/reinstall/uninstall.
/Library/LaunchDaemons/homebrew.mxcl.asimov.plist: service already bootstrapped
Bootstrap failed: 37: Operation already in progress
Error: Failure while executing; `/bin/launchctl bootstrap system /Library/LaunchDaemons/homebrew.mxcl.asimov.plist` exited with 37.

After, despite the error:

sudo brew services
Name      Status    User File
asimov    scheduled root /Library/LaunchDaemons/homebrew.mxcl.asimov.plist

I think it would be a great thing to actually have a place to store options that are suggested in other issues: We could do something like: ~/.asimovrc which contains options per line like .bashrc or in a JSON format.

As a developer, I often have compiled folders in my project and I would like to exclude them as well from TimeMaschine. e.g.:

• build: (from web projects compiled from webpack)
• cache, __pycache__: (some shit programs create for caching)
• tmp: (temporary files that the program needs but can easily be recreated)
• and much more, depending on your workflow...

By default Time Machine backups run around every hour, while asimov runs only daily. This likely means that Time Machine will include most dependency dirs during its run and only after that they will be excluded, while still remaining in backups.

It is possible to remove dependency dirs from backups retroactively but I think a better solution would be to wrap Time Machine into asimov as described in this answer. This will allow asimov to start before each Time Machine backup and have a chance to modify exclusion list. It also needs to launch as a low priority background task like Time Machine does so as not to interfere with what user is doing.

// Another approach I'm considering (instead of error-prone asimov setup) is to wrap npm install and similar commands such that they add an exclusion to Time Machine when they are ran.

It would be a great feature if asimov only excluded files that were children of your current directory. This would help solve the problem of asimov excluding files in ~/.vscode and other hidden directories in my home folder. If there's an existing way to accomplish this, let me know!

An intuitive off-label use of Asimov would be to prevent Spotlight from search */node_modules/. Indeed, I came across Asimov while searching for a way to get Spotlight to ignore these folders. At a glance, it's unclear whether Asimov would do this. Could you add a statement to /readme to clarify whether it would?

So I ran this as instructed and could see logs of stuff getting excluded like :

... mobile/node_modules has been excluded from Time Machine backups (506M).

But now I go to Time Machine preferences and I don't see those folders on the exclusion list.

This tool is awesome. I can finally forget about excluding new projects manually and give my backup drive a well deserved break. Good job!

It would be amazing if you could do the same with iCloud Drive Sync. I know its implementation is really stupid, when compared to Time Machine, but it would make much more sense to have these exclusions automated aswell. I don't know if there's an exclusion list, but the node_modules.nosync symlink method is really inconvenient for every day use.

Maybe you can come up with something. Cheers!

Tools like Docker and Vagrant will generally download and cache images, but these are often easily re-downloaded.

Additional directories to be excluded from Time Machine backups:

• ~/.vagrant.d/boxes
• ~/Library/Containers/com.docker.docker/Data/vms

Could be some interactive CLI option maybe

/Users/olso/Library/Application\ Support/Spotify/PersistentCache/Storage

This is an issue related to node_modules, although other dependencies may have the same issue. Basically, there's a lot of noise in the entries list unnecessarily.

An example is a directory with this structure:

root/
  node_modules/
    moduleA/
      node_modules/
        moduleD/
          package.json
      package.json
    moduleB/
      node_modules/
        moduleE/
          package.json
      package.json
    moduleC/ 
      package.json
  package.json

This will add the following directories to your exclusions:

./root/node_modules/moduleA/node_modules
./root/node_modules/moduleB/node_modules
./root/node_modules

Obviously this is redundant as the final entry already covers the former. In a single project directory, my node_modules caused ~50 entries to be added, rather than a single entry. This is a pain when it comes to reversing the process, or just inspecting the excluded entries list.

When I run command asimov or sudo asimov, its shows me what is added in time machine. So it can scan correctly. but then I check in time machine file using below command, it do not have new entries which was shown by running asimov command.

sudo mdfind "com_apple_backup_excludeItem = 'com.apple.backupd'"

Please let me know if there is fix available. thanks

It seems script skipping first matching directory because of line 37

After I delete locale repos, the excluded directories remain in the time machine exclusion list. It would be nice to purge these directories.

It fails on locations containing spaces, such as the default created by Local. Output is the following:

find: /Users/jeffmatson/Sites: No such file or directory

Finding node_modules/ directories with corresponding ../package.json files...
find: /Users/jeffmatson/Sites: No such file or directory

Finding .vagrant/ directories with corresponding ../Vagrantfile files...
find: /Users/jeffmatson/Sites: No such file or directory```

Add support for SPM (Swift Package Manager) when installed through Apple Xcode.

I had this output when running asimov:

/Users/yyy/Projects/xxx/node_modules/postcss-minify-params/node_modules has been excluded from Time Machine backups

There should be no need to do that, as /Users/yyy/Projects/xxx/node_modules/ is already ignored...

Asimov hasn't been getting the love it deserves, and I'm happy to welcome @sudar as a co-maintainer (see #48) to help things along.

I have a lot of ideas for where I'd like to see Asimov go, but want to open them up to the community for thoughts + discussion before implementing.

Near-term

Get Asimov submitted to Homebrew (#2)

This has been a goal since the beginning of the project, but Homebrew's policies dictate that the repository needs to meet a certain threshold of stars (50) on GitHub in order to be considered. Fortunately, this was exceeded some time ago, so now it's a matter of getting the Homebrew recipe in-place. I had started (and then re-started) this effort, but #34 looks the most promising. I'm totally the blocker in reviewing and approving :/

Introduce automated testing (#31)

The logic as it stands right now is pretty straight-forward (match a config/dependencies file with its corresponding installation directory and, if found, exclude from Time Machine) but having something that could run in a CI environment would be extremely useful to make sure we're not breaking things as we go.

Eventually

Introduce a configuration standard (#27)

A number of issues have hinted at the need for a user-level Asimov configuration file, enabling individual users to configure not only the patterns that should be matched but also other directories (#28), whitelist specific directories (#14), and more.

Issue #27 proposed something like an .asimovrc file, which could be used to store Asimov configurations.

Expand beyond development dependencies

Asimov was designed for developers, but there have been some great suggestions around other things that can be excluded from Time Machine (for example, Spotify caches, suggested in #39).

If Asimov becomes something of the de facto tool for keeping Time Machine backups limited in scope, there are a number of other file types that we might want to include:

• Multimedia render files
• Caches and log files
• Thumbnail metadata

Possibly rewrite in something besides Bash?

Asimov is currently written in Bash, since it's essentially performing filesystem traversal, basic pattern matching, and then interfacing with the tmutil program; if we want to get into more sophisticated functionality (for example, asimov list to see everything excluded from Time Machine backups, it's probably best to consider rewriting in a different language.

My default direction would be as a Symfony Console application, but that's mostly because PHP is my primary language (and PHP comes pre-installed on MacOS). An abstraction around tmutil will make the tool easier to work with, and Symfony components are designed to be well-tested. Distributing it as a PHAR means we still only have one file being installed.

The drawback here is that MacOS typically ships with older versions of PHP, which means the package would either need to be written using older PHP conventions, PHP would need to be marked as a dependency within Homebrew, and/or support would need to be restricted to versions of MacOS that ship more recent (read: PHP 7.2+) releases.

Make Asimov the go-to tool for interfacing with tmutil

The default tmutil tool shipped with MacOS isn't super-well documented, and is much more a developer tool than something intended for typical users. While I don't believe a GUI is necessary (Time Machine already provides that), providing a user-friendly Time Machine interface would be helpful.

Potential commands could include:

add - add files/directories to the exclusion list
config - read/write entries from the user's .asimovrc file
ls/list - Show all files/directories that have been excluded from Time Machine
prune - Locate any directories/files that have been added to the exclusion list but no longer exist (which would address #38)
rm/remove - Remove files/directories from the exclusion list
scan - Scan the filesystem for any new files/directories to be excluded. This is the default behavior of Asimov today.

Hi,

I installed this via brew, and ran it as a brew service, and nothing happened.

Looking at it, I don't understand how it ever could - ASIMOV_ROOT is defined as ~, but run as a brew service it will be running as root, and for root ~ is /var/root, which is unlikely to be where you want to look for things to exclude?

Perhaps I'm missing something?