Git Product home page Git Product logo

magic-folder's Introduction

Magic Folder

Magic Folder for Tahoe-LAFS is a Free and Open file synchronization system. It detects local changes to files and uploads those changes to a Tahoe-LAFS grid. It monitors a Tahoe-LAFS grid and downloads changes to the local filesystem.

documentation status gha_linux gha_macos gha_windows test coverage percentage

Installing

for integrators

When packaging magic-folder, please install using our pinned requirements, which are tested in CI. This should be done in a virtualenv, or other isolated python environment, so as to not interfere with system or user python packages:

git clone https://github.com/LeastAuthority/magic-folder
cd magic-folder
pip install --require-hashes -r requirements/base.txt
pip install --no-deps .

from source

To install from source (either so you can hack on it, or just to run pre-release code), you should create a virtualenv and install into that:

git clone https://github.com/LeastAuthority/magic-folder.git
cd magic-folder
virtualenv venv
venv/bin/pip install --upgrade setuptools
venv/bin/pip install --editable .
venv/bin/magic-folder --version

Usage

Magic-Folder is configured via the magic-folder command-line tool.

Magic-Folder configuration is kept in a directory. The default place for this directory is platform-dependant; on Linux it will be in ~/.config/magic-folder. Inside this directory is a database for global configuration and sub-directories to track state and temporary space for each actual magic-folder including a configuration database. All databases are SQLite.

A running Magic-Folder needs to have access to a Tahoe-LAFS client that it may use to perform operations in the Tahoe-LAFS Grid. This is referenced by the "node directory" of the Tahoe-LAFS client although actual operations are performed via the Tahoe-LAFS WebUI.

There are two ways to create a new Magic Folder instance (that is, the configuration required). Create a fresh one with magic-folder create or migrate from a Tahoe-LAFS 1.14.0 or earlier instance with magic-folder migrate.

Once a Magic-Folder is configured, functionality is provided by running a long-lived magic-folder daemon. This process is run using the magic-folder command line tool:

magic-folder --config <path to Magic Foler directory> run

As long as this process is running, whatever magic folders are configured will be functional. The process must be restarted to read configuration changes. All other interactions are via the HTTP API which listens on a local endpoint according to the configuration. Other magic-folder subcommands are typically just thin CLI wrappers around a particular HTTP endpoint.

Testing

To run the unit test suite:

  • tox

You can pass arguments to trial with an environment variable. For example, you can run the test suite on multiple cores to speed it up:

  • MAGIC_FOLDER_TRIAL_ARGS="-j4" tox

Documentation

Documentation is written as reStructuredText documents and processed using Sphinx; you will need sphinx and sphinx_rtd_theme. To generate HTML version of Magic Folder documents, do:

  • cd docs; make html

Resulting HTML files will be under docs/_build/html/.

License

Copyright 2006-2018 The Tahoe-LAFS Software Foundation Copyright 2020-2021 The Magic-Folder Developers

You may use this package under the GNU General Public License, version 2 or, at your option, any later version. You may use this package under the Transitive Grace Period Public Licence, version 1.0, or at your option, any later version. (You may choose to use this package under the terms of either licence, at your option.) See the file COPYING.GPL for the terms of the GNU General Public License, version 2. See the file COPYING.TGPPL for the terms of the Transitive Grace Period Public Licence, version 1.0.

See TGPPL.PDF for why the TGPPL exists, graphically illustrated on three slides.


magic-folder's People

Contributors

antagonismorg avatar crwood avatar ctrlaltdel avatar daira avatar david415 avatar exarkun avatar grooveadelic avatar habnabit avatar heartsucker avatar isnotajoke avatar kpreid avatar leif avatar markberger avatar meejah avatar meskio avatar sajith avatar str4d avatar tomprince avatar tpltnt avatar travis-tahoe avatar vrusinov avatar vu3rdd avatar warner avatar zooko avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

magic-folder's Issues

Make test case temporary paths beneath _trial_temp

Trial runs tests beneath a _trial_temp directory it creates. This conveniently collects test-generated artifacts in one place instead of smearing them across, say, the project source directory or your home directory or /tmp. As part of this trial's mktemp helper also creates temporary paths beneath this directory instead of in /tmp.

Make our mktemp-alike API do something similar.

magic_folder.test loads allmydata.test which loads conflicting Hypothesis profiles

magic_folder.test defines and loads a Hypothesis profile. Our CI configuration specifies a good profile to use for CI conditions.

Unfortunately allmydata.test does the same thing and our CI configuration does not specify a good profile fir it to use. allmydata.test loads second so its choice wins.

A solution would be to stop loading allmydata.test. This is probably a good idea anyway. Fortunately we make very light use of it so this is probably feasible.

Related to #43

Flaky test: magic_folder.test.test_status.StatusTests.test_unknown_magic_folder_name

Traceback (most recent call last):
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/twisted/internet/defer
.py", line 151, in maybeDeferred
    result = f(*args, **kw)                                                                                           
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/testtools/testcase.py"
, line 723, in _run_test_method                                                                                            return self._get_test_method()()                                                                                     File "/home/meejah/work/leastauthority/src/magic-folder/src/magic_folder/test/test_status.py", line 133, in test_unkn
own_magic_folder_name
    folder_names(),
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/hypothesis/core.py", l
ine 1071, in wrapped_test                                                                                             
    state.run_engine()
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/hypothesis/core.py", l
ine 789, in run_engine
    info.__expected_traceback,                             
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/hypothesis/core.py", line 656, in execute_once                                                                                               
    % (test.__name__, text_repr[0])                         
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/hypothesis/core.py", line 856, in __flaky
    raise Flaky(message)                                                       
hypothesis.errors.Flaky: Hypothesis test_unknown_magic_folder_name(self=<magic_folder.test.test_status.StatusTests.test_unknown_magic_folder_name id=0x7f024dca7e50>, folder_names=[u'\x0e\U000aea6e\x1d',                                   
 u'\U000cd527\x05',                                                    
 u'\U0003b33c&\u4bbe',
 u'\x07\x02\U000c4c63\x08',
 u'\U00087e93',                   
 u'\x17\U00038411',                                                                                                    
 u'\U0007195e+',                
 u'\U00015a2f\U000e8997\x02 \U001089eb\r\U0002b815#\U0008c5c4\u6588\U00077492\U000f2fcd\x03',                         
 u'%',                                                                                                                 
 u'\ue70f\n\x14\U000ba7d6\U000b9f5d("\ud428'], collective_dircap=u'URI:DIR2:fdis5qniqxtadltxmcnd6ta7im:3mvag25uktenhbb466jtqtediw67pdldynnhwvogxqpjqlte5kdq', upload_dircap=u'URI:DIR2:jul3r6faef4hnqc7n4mkszakdy:t33i2ahnpcsl6f67udrb5rwsw74h2tqzcxcma744hyz4cuanhdrq') produces unreliable results: Falsified on the first call but did not on a subsequent one    

Participant Restores Deleted Files

As a participant, I want to restore (either for all participants or only for myself) a version of the deleted file as it existed at some point previously so that I can restore it.

Most likely same implementation tasks as for #92

Sometimes Hypothesis tests fail on CI because of the Hypothesis deadline

The Hypothesis deadline: https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.deadline

The CI execution environment is shared. We can't make any guarantees about how long our tests will take run (more generally, none of Linux, macOS, and Windows are real-time operating systems and Python is not a real-time programming language so it's hard to argue that you can ever guarantee how long a test will take to run in any supported environment, but I digress...).

Fortunately the Hypothesis deadline can be disabled. Do this for all CI.

The "deprecations" CircleCI job fails

For example, on https://circleci.com/gh/LeastAuthority/magic-folder/3439, it encountered this exception:

Traceback (most recent call last):
  File "/tmp/magic-folder.tox/upcoming-deprecations/bin/flogtool", line 6, in <module>
    from foolscap.logging.cli import run_flogtool
  File "/tmp/magic-folder.tox/upcoming-deprecations/lib/python2.7/site-packages/foolscap/logging/cli.py", line 4, in <module>
    from twisted.python import usage
  File "/tmp/magic-folder.tox/upcoming-deprecations/lib/python2.7/site-packages/twisted/__init__.py", line 16, in <module>
    "This version of Twisted is not compatible with Python 3.4 "
Exception: This version of Twisted is not compatible with Python 3.4 or below.

Twisted trunk@HEAD has dropped Python 2.7 support entirely now. Thus, it will no longer be possible to test against development snapshots of Twisted to watch for deprecations.

The smallest change to fix this would be to remove the CI configuration that installs Twisted trunk@HEAD into this test configuration.

Make non-ASCII paths work

There is some effort in place to handle non-ASCII paths. However, on platforms that decline to specify a filesystem encoding (of UTF-8), there are still problems. Mainly, FilePath naively passes unicode around and when something tries to encode it using ASCII, there can be problems.

For now, testing against non-ASCII paths is disabled. This is done by having magic_folder.test.strategies.path_segments only build ASCII unicode paths. It should be possible to have this build non-ASCII unicode paths too and any tests that fail should be fixed.

Integration test improvements

It would be useful to use some of the "fixtures" currently defined in conftest.py as helpers for individual tests; putting most of the "real" work into a non-conftest.py location can facilitate this.

--web-port should be required

Currently, --web-port has a default value of tcp:9889. This option should be non-optional -- the user must think about where they want to listen (and anyway a public / non-local port isn't a good default). Another reason is that trying to run two copies on the same machine will fail (well, the second one will fail silently because of #67).

Participant Sees All History of Restored Files

As a participant, after I restore a deleted file, I want to be able to see the history of that file, including the history prior to its deletion, so that a mistaken choice to delete a file does not prevent me from seeing history in all of the cases I would normally want to look at history.

This more or less falls out of the snapshot design and the list-all-snapshots API.

Factor introducer/storage node setup into a reusable Fixture

ListMagicFolder does a bunch of work to set up an introducer and a client w/ storage enabled. This logic can be factored out of the TestCase into a Fixture which will be more easily reused by other TestCase subclasses (or which will at least take some clutter out of ListMagicFolder).

The "PyPy" CI job fails

The job fails with the message below. Possible fix is to upgrade the openssl version?

pypy27-coverage inst: /tmp/magic-folder.tox/.tmp/package/1/magic_folder-0.0.0.post288.zip
ERROR: invocation failed (exit code 1), logfile: /tmp/magic-folder.tox/pypy27-coverage/log/pypy27-coverage-3.log
================================== log start ===================================
Traceback (most recent call last):
  File "/usr/local/lib-python/2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/lib-python/2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal import main as _main  # isort:skip # noqa
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/cli/main_parser.py", line 12, in <module>
    from pip._internal.commands import (
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
    from pip._internal.commands.completion import CompletionCommand
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
    from pip._internal.cli.base_command import Command
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/cli/base_command.py", line 20, in <module>
    from pip._internal.download import PipSession
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/download.py", line 15, in <module>
    from pip._vendor import requests, six, urllib3
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_vendor/requests/__init__.py", line 97, in <module>
    from pip._vendor.urllib3.contrib import pyopenssl
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/OpenSSL/crypto.py", line 15, in <module>
    from OpenSSL._util import (
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 205, in <module>
    _verify_openssl_version(Binding.lib)
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 169, in _verify_openssl_version
    "You are linking against OpenSSL 1.0.1, which is no longer "
RuntimeError: You are linking against OpenSSL 1.0.1, which is no longer supported by the OpenSSL project. You need to upgrade to a newer version of OpenSSL.

Re-arrange the repository to make more sense as a stand-alone

The repository structure continues to reflect the structure of the magic-folders implementation as it existed inside Tahoe-LAFS. Without all of the rest of Tahoe-LAFS around it, this structure makes less sense. Consolidate it somewhat, removing redundancies that exist in the path hierarchies and the extremely sparse directories which now contain only one or two files.

Files / modules which could use a better name (suggested "better name" too, but maybe there's even-better names):

  • magic_folder/scripts/magic_folder_cli.py -> magic_folder/cli.py (#49)
  • magic_folder/frontends/magic_folder.py -> magic_folder/magic_folder.py (#70)
  • magic_folder/web/magic_folder.py -> .magic_folder/web.py (#69)
  • others?

Refactor magic-folder cli command handling

It would be nice if each command like create, invite, join, status etc are split into multiple modules and moved closer to the main magic-folder module than live in the scripts directory. We use Tahoe-LAFS web API whereever possible and file tickets against Tahoe-LAFS for any missing APIs.

Splitting this into multiple tickets.

  • status - Already handled in #37.
  • create: #58
  • join: #52
  • invite: #51
  • leave: #54

`magic-folder run` doesn't exit on listen error

If the magic-folder daemon fails to listen on its web-port, an exception is printed but the program keeps running (and logs "successful startup"). Magic-folder should exit if this fails.

(Also, I'm not sure that the --web-port should be optional? It currently defaults to tcp:9889. Maybe a better default would be "whatever is in magic-folder.url if that file exists in the node-dir" or just not having a default at all (i.e. it's a required option).

Refactor invite

Refactor invite command and move it closer to the main magic-folder module.

Part of #50

"magic-folder invite" doesn't check for existing user

The only hint that you've already got a user if you e.g. re-run the same magic-folder invite command twice in a row is:

2020-04-02T16:57:25-0600 [stdout#info] magic-folder: You cannot overwrite a directory with a file

We should check if the new directory entry that will be created is already used before doing that work. Looking in the "_invite()" code it uses tahoe_mv() to do this (which seems .. odd .. but might just be a naming thing?)

magic_folder.test.no_network should no longer be required

The unit test suite doesn't need to set up a "no network" Tahoe-LAFS grid and the integration tests don't use this approach (preferring to set up a network-using grid).

This module should go and anything that is still using it is bogus and needs to be fixed.

This looks like it is too large to do all at once so here are some sub-tasks:

  • magic_folder.test.cli.test_magic_folder.ListMagicFolder - #30
  • magic_folder.test.cli.test_magic_folder.StatusMagicFolder - #33
  • magic_folder.test.cli.test_magic_folder.CreateMagicFolder - #47
  • magic_folder.test.test_magic_folder.RealTest
  • magic_folder.test.test_magic_folder.RealTestAliceBob
  • magic_folder.test.test_magic_folder.MockTest
  • magic_folder.test.test_magic_folder.MockTestAliceBob

Consider unicode normalization

Unicode is used throughout the implementation but largely without consideration for how it is normalized. This means that a folder might be assigned a name normalized using one form and then later requested using a different normalization of that name. Currently this will result in the folder seeming not to exist. The two different normalizations of the text will look identical to the end user so this is a misleading result.

Until this is addressed, the Hypothesis strategies that generate text will be limited to generating text with a particular normalization so that everything appears to work.

Logs interfere with CLI tools

Whatever is happening with logging in magic-folder interfere with many of the "user oriented" command-line tools. E.g. here's the output from magic-folder status:

(venv) meejah@buyan:~/work/leastauthority/src/magic-folder$ magic-folder --node-directory ../tahoe-lafs/testgrid/alice/ status 
2020-04-02T12:59:27-0600 [twisted.application.runner._runner.Runner#info] Starting reactor...
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71cd041668>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71cd041668>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccfdda28>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccfdda28>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccffb9b0>, <HostnameEndpoint 0.0.0.0:9889>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccffb9b0>, <HostnameEndpoint 0.0.0.0:9889>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71cd041578>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71cd041578>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccffbd70>, <HostnameEndpoint 127.0.0.1:8890>)

Magic-folder status for 'default':

Local files:
  lumière.txt (625 B): good, version=0, created 37 days ago

Remote files:
  bob's remote:
  alice's remote:
  lumière.txt (625 B): good, version=0, created 37 days ago



2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccffbd70>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [-] Main loop terminated.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.