dvc should work out of box without any cloud credentials and error\warning messages.

Currently:

$ dvc run python code/df_to_matrix.py data/Posts-bin-fr-test.tsv data/Posts-test-matrix.p
Error. config 'dvc.conf' is not correctly setup.  Please fix:
Error.     can't find aws credentials.  TODO

Desired behavior:

• by default, no cloud is set in config "Cloud = ". No checking.
• if Cloud is set in config - dvc complains only in cloud related commands (sync currently and remove)

Logging level should depend on log file setting. Only INFO messages by default.
A simple solution is needed - no advanced logging libs for now.

must specify host for S3Connection; see discussion here
boto/boto#621

and list of required hosts
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

DVC does not ignore locker file:

c:\src\tmp>dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz data/
Error. [dvc-git] Commit all changed files before running reproducible command. Changed files:
Error. ?? .dvc.conf.lock

Todo: try different scenarios

ReproChangedDependency test works but it actually works incorrectly. It supposes to trigger by file content change.

Scenario:
0) I work on DVC project wich is publicly available in Github

1. I use my private cloud credentials
2. I sync the data results to a cloud and make the result available for public (how?)
3. Somebody clone my DVC repository
4. He also can use my private data from the cloud (how?)

• Do we need special settings in the stats files?
• What to do if somebody forks the repository and uses his own credentials for the next iterations.

Account: https://github.com/dataversioncontrol
Repository name: dvc

dvc import http://file1 data/
rm cache/file1_aaaaaa
dvc repro data/file1
==> Info. Reproduction is not required for data item "data/Badges.xml".

For example, we need to use dvc run ... instead of dvc-run ...

This library might help: https://github.com/google/python-fire

using a dummy script that redirects it's output to a data file, rerunning dvc run failed, unless I deleted the file and recreated it

s3 and gcp import are not implemented yet - only files and publicly available urls

Scenario: two guys are working on the same project. The first guy uploads a private dataset to S3 and uses the dataset as data source for a pipeline\workflow. The second guy should be able to reuse the dataset if he has permissions to the dataset.

The same scenario should work in GCP.

Probably not in beta version.

$ dvc import data/file*

This command imports only the first file it will recognize - file1.txt for example, not file2.txt.

In a dvc repository some of data-files were synced to cloud. Then the repo was cloned.

But I failed to sync the file:

$ dvc data-sync data/summary.txt
Downloading cache file from S3 "nlx-shared/tag_classifier/.cache/summary.txt_6405e23"
Traceback (most recent call last):
  File "/Users/dmitry/src/dvc/dvc2.py", line 77, in <module>
    Runtime.run(CmdDataSync, args_start_loc=argv_offset)
  File "/Users/dmitry/src/dvc/dvc/runtime.py", line 45, in run
    sys.exit(instance.run())
  File "/Users/dmitry/src/dvc/dvc/command/data_sync.py", line 60, in run
    return self.sync_symlink(data_item)
  File "/Users/dmitry/src/dvc/dvc/command/data_sync.py", line 84, in sync_symlink
    self.sync_from_cloud(data_item)
  File "/Users/dmitry/src/dvc/dvc/command/data_sync.py", line 184, in sync_from_cloud
    return self._sync_from_cloud_aws(item)
  File "/Users/dmitry/src/dvc/dvc/command/data_sync.py", line 106, in _sync_from_cloud_aws
    key.get_contents_to_filename(item.resolved_cache.relative, cb=percent_cb)
  File "/usr/local/lib/python2.7/site-packages/boto/s3/key.py", line 1714, in get_contents_to_filename
    os.remove(filename)
OSError: [Errno 2] No such file or directory: '.cache/summary.txt_6405e23'

Names are correct. File exist in the cloud.

Another issue - exception should be caught and outputted as a single line (or two lines).

We need two types of logs:

1. console output (by default: info and above)
2. log output (by default: debug and above)

Both of the values have to be presented in the dvc.conf file. Like

ConsoleLogLevel = Info
LogLevel = Debug
LogFile = dvc.log # what about system log do we need that?

The config log parameter ConsoleLogLevel could be override by command line:
dvc run --verbose debug Rscript plotting.R data/input.csv.

PS: do not forget Windows support.

Can be found in tutorial:

$ dvc repo data/Posts-test-matrix.p

Symlinks are not supported in Windows OS.
This hack might help: http://stackoverflow.com/questions/6260149/os-symlink-support-in-windows

test on startup and politely help if
1 - no cloud selected;
2 - no StoragePath
2 - aws / no credentials;
3 - gc / no credentials or no project;

Now dvc does recursive check for all dependencies. It should not do that by default.

Scenario. Pipeline A --> B --> C. A and C are not cached. B was synced from a cloud. I need to repro C, no A repro required.

... also we need to get rid of double commands like dvc data-sync. dvc sync is enough.

see discussion here:
boto/boto#2836

eg boto won't work with bucket 'a.b' but will with 'a-b'; requesting a key for any bucket with a period in the name returns None

Git repository might be created but git might be not configured.
The results of this failure:

1. .gitignore file was not created.
2. change was not committed

c:\src\tmp>dvc init
Directories data/, .cache/ and .state/ were created
Error. Git command error (git commit -m DVC init. data dir data, cache dir .cache, state dir .state):
*** Please tell me who you are.

Run

  git config --global user.email "[email protected]"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: empty ident name (for <(NULL)>) not allowed

It would be great to see the dependency graph for a given data item.

$ dvc visual data/result_scores.tsv

We need to clarify the output details...

currently (as much as I understand) you can configure only one directory that is tracked in dvc
it would be useful to be able to specify more then one directory (for repositories that contain more than one project)

What documentation format we should choose? Should we host the docs in the github or outside?

Examples:

• dvc gc - remove everything from the .cache dir not related to the current data files
• dvc gc data/myfile.txt - remove all caches except the current one
• dvc gc --nuke - clean up .cache dir. Basicaly it is rm -rf .cache/*.

Not in the beta version, probably.

Weird message:

c:\src\tmp>dvc import
usage: __main__.py [-h] [--no-git-actions] [--lock] [[...]]
__main__.py: error: the following arguments are required:

if I run ./build_osx.sh, the dvc executable crashes on startup; the relevant message seems to be

./dist/dvc
Traceback (most recent call last):
  File "dvc2.py", line 13, in <module>
    from dvc.command.data_import import CmdDataImport
  File "/private/var/folders/r9/4rd8csvj56s12zg3vdm360ym0000gn/T/pip-build-kSJs7p/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "dvc/command/data_import.py", line 8, in <module>
    from dvc.command.data_sync import sizeof_fmt
  File "/private/var/folders/r9/4rd8csvj56s12zg3vdm360ym0000gn/T/pip-build-kSJs7p/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "dvc/command/data_sync.py", line 6, in <module>
    from google.cloud import storage as gc
  File "/private/var/folders/r9/4rd8csvj56s12zg3vdm360ym0000gn/T/pip-build-kSJs7p/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "site-packages/google/cloud/storage/__init__.py", line 35, in <module>
  File "site-packages/pkg_resources/__init__.py", line 552, in get_distribution
  File "site-packages/pkg_resources/__init__.py", line 426, in get_provider
  File "site-packages/pkg_resources/__init__.py", line 968, in require
  File "site-packages/pkg_resources/__init__.py", line 854, in resolve
pkg_resources.DistributionNotFound: The 'google-cloud-storage' distribution was not found and is required by the application

Probably related to

google/cloud/storage/__init__.py
35:__version__ = get_distribution('google-cloud-storage').version

Create dvc pip package and a corresponded web page on pypi.python.org.

All the dvc commands (dvc-init, dvc-run) have to be available to users after the package installation.

move from data to global

ie you get yourself in the state where you have file data/X where X doesn't exist in cache or cloud

How to work on Windows without the symlinks?

Should we use cygwin or any other unix tools for win?
How about windows10 ubuntu shell? Can we support only this version?

$ run python script.py data/input data/output

script.py should be implicitly included to code dependency.

eg aws, AWS, google cloud, gc, etc