e3krisztian / bead Goto Github PK

To bead save large and complex folders, it would be important to flexibly set which files to ignore. Currently, only files inside temp/ are ignored.

Introduce a .beadignore file with the same syntax as .gitignore to set ignore patterns. (This could live inside .bead-meta, but keeping it in root would be more similar to the git model.)

bead new could create a .beadignore with only temp/ in it. This way the default behavior would remain the same, but advanced users could customize.

Remove loaded input - to temporarily save space.
The input remains defined.

input unload input-name

update an input to have data from

• latest version of package
• a concrete version of the package
• another package

Currently there is some logic for it in lib.config but it is correct only for linux!

Similar to develop, but create a new package instead of version, yet somehow keep metadata about its ancestry.

This would enable

• creating compatible fixes for old packages without confusing package timeline (although everything depending on the original would need to be branched to use the fix)
• use of template packages (e.g. tool invocations already parametrized with some databases as dependencies)

I have a folder with a .bead-meta/bead json. There are no input folders. (I did not receive it as a bead.) bead input load gives the following error.

OSError: [Errno 2] No such file or directory: '/Users/koren/Google Drive/Research/supplier-survey/input'

Expected behavior: bead input load should create input/path from the information available at bead-meta. Much like git checkout can check out entire folders.

bead-reference is one of:

• name of an existing file
• [box:]name[@absolute-version]
• @relative-version (relative references to existing input)

Where absolute-version is one of:

• unique part of timestamp-prefix
• unique part of contentid after # (# is to disambiguater from timestamp prefix)

Relative-version is absolute-version and the following

• next / +
• prev / previous / -

bead-reference examples:

• ~/beads/xyt.zip (actually ~ should be resolved by the shell)
• default, most recent version available
  • name
  • box:name
• unique timestamp prefix
  • name@202009
  • name@20200920T1113
• unique contentid part
  • name@#202009
• relative versions (useful for update)
  • @next
  • @prev
  • @previous
  • @+
  • @--

Commands mount, develop (and potentially more) should accept a package name in which case the package file is searched for and the latest version found is used.

having an existing bead x

$ bead develop x
$ bead develop x

works he second time without error/warning

100% coverage for /tools/

Work with multiple boxes, having the same bead, but potentially with different input-maps (or other future volatile meta).

[ ] develop requires box name, saves it in workspace's meta
[ ] save saves to box from which it was originated (unless it was a new or developed from file)
[ ] update updates to another version only in the originating box, must be forced to load beads from other boxes

https://pypi.python.org/pypi/argcomplete
https://github.com/kislyuk/argcomplete

@e3krisztian Please put the new bead version into release.

/tmp$ bead new asd
Created asd
/tmp$ cd asd/
/tmp/asd$ virtualenv temp/env
Using base prefix '/usr'
New python executable in /tmp/asd/temp/env/bin/python3
Also creating executable in /tmp/asd/temp/env/bin/python
Installing setuptools, pip, wheel...done.
/tmp/asd$ bead nuke
[Errno 1] Operation not permitted: '/tmp/asd/temp/env/lib/python3.6/_bootlocale.py'

might it be caused by symlinks?

/tmp/asd$ ls -l '/tmp/asd/temp/env/lib/python3.6/_bootlocale.py'
lrwxrwxrwx 1 user user 33 Feb 16 14:44 /tmp/asd/temp/env/lib/python3.6/_bootlocale.py -> /usr/lib/python3.6/_bootlocale.py

/tmp/asd$ bead new asd
/tmp/asd$ cd asd
/tmp/asd$ ln -s ../hello_world.sh .
/tmp/asd$ bash hello_world.sh  # echo hello world > output/data
/tmp/asd$ bead save
workspace contains a link: /tmp/asd/hello_world.sh
/tmp/asd$ unzip -v .../asd_20170216T145953425815+0100.zip
Archive:  .../asd_20170216T145953425815+0100.zip

This file is a BEAD zip archive.

...
----

 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
      12  Defl:N       14 -17% 2017-02-16 14:59 af083b2d  data/data
--------          -------  ---                            -------
      12               14 -17%                            1 file

Expected: error message + partial bead removed

This is known to have caused confusion as the user might be mislead from the command output and file times that the input has been changed while it is not.

Expected beaviour:

• input update says that the input is up-to-date
• input update and input load should display the full path name of the bead loaded
• maybe an input unload to have an ability to unload/load damaged input (#26)

Saving a bead with an uppercase name results in a bead that can be referenced only by
file name, not by name

When the file is developed by name its name is converted to lowercase:

/tmp$ bead develop AbcDfg_20170220T200239824886+0100.zip
Extracted source into /tmp/abcdfg

Should replace the file Workspace/.pkgmeta/repo

When sharing an open bead during development, .bead-meta may get out of sync with the content of input/. For example, we are versioning .bead-meta, but not input/*. In this case, bead input load does not behave as expected. It does not load the appropriate version listed in .bead-meta, even if the version living in input/ is older.

Currently, the only? solution to force load an input is

bead input delete whatever
bead input load whatever

But this does not respect what is already in .bead-meta and is potentially error prone.

Recommended behavior:

bead input load --force whatever

same for update.

I'm on python 2.7:

$ python --version
Python 2.7.8

And tox fails with:

$ tox
ERROR: unknown environment '[package]envs'

• multifile config
  • main config as json persisted dictionary
  • uuid map as sqlite db
• key/value store
• testing support
  • safely diverting home directory for tests (especially for future dev!)
  • key value store tests

So that names as known by a peer could be specified.

Use my x package

$ ws pack

yields

Traceback (most recent call last):
  File "XXX/libdevenv/bin/ws", line 10, in <module>
    sys.exit(main())
  File "XXX/libdevenv/lib/python2.7/site-packages/mando/core.py", line 125, in __call__
    self.execute(sys.argv[1:])
  File "XXX/libdevenv/lib/python2.7/site-packages/mando/core.py", line 121, in execute
    return command(*a)
  File "XXX/lib/lib/tool/ws.py", line 85, in pack
    ts = timestamp()
TypeError: 'module' object is not callable

nuke removes the workspace, but nuke might be a charged word

alternatives:

• nuke
• zap is a good candidate
• delete
• erase
• remove

bead input add /tmp/abc-timestamp.zip abc

Actual result: registers /tmp/abc-timestamp.zip as input name

Expected: error message, that input name contains path separator.

Although verification takes time - have to read the content of the package and calculate a hash - comparable time is also spent during the relevant operations:

• mount
• develop

to match that of json, etc.:
load, loads, dump, dumps

So that pack creates the package file in the configured directory instead of in ./temp

bead save is painfully slow with a few GB of data. Can we use parallel processing to speed up zipping?

When peers can be referenced it could be possible to import their package names and export ours.

If you try to update a specific, non-existent input (e.g. true input name is "my_input" and you try "bead input update my-input") only a not too informative AttributeError is returned.

Traceback (most recent call last):
...
AttributeError: 'NoneType' object has no attribute 'kind'

Behavior should be same as with git. If I am inside a bead workspace, say input/data1, I should still be able to do bead status and bead input add.

so that other tools can automatically run the computation in the bead after bead develop and bead input load. Like

"entry_point": "stata -b do main.do"

I downloaded a neuron package zip to my local repo. Afterwards, I issued neuron input add, and got the error message above.

CEU-8706:managerdb koren$ neuron repo list
Repositories:
-------------
sandbox: /Volumes/Home/xl/data/data-repo/repositories
local: /Volumes/Home/datastore/neuron-repos
CEU-8706:managerdb koren$ pwd
/Volumes/Home/datastore/managerdb
CEU-8706:managerdb koren$ neuron repo list
Repositories:
-------------
sandbox: /Volumes/Home/xl/data/data-repo/repositories
local: /Volumes/Home/datastore/neuron-repos
CEU-8706:managerdb koren$ neuron input add frame frame
ERROR: Not a known package name: frame
CEU-8706:managerdb koren$ ls /Volumes/Home/datastore/neuron-repos/
frame_20151028_152234_226060.zip    rovat_13_15_20151105_144001_928190.zip
CEU-8706:managerdb koren$ 

sys_argv = ['/usr/bin/bead', 'save', 'latest_de']
Traceback (most recent call last):
File "/usr/bin/bead/bead_cli/main.py", line 158, in main
retval = run(config_dir, sys.argv[1:])
File "/usr/bin/bead/bead_cli/main.py", line 135, in run
return parser.dispatch(argv)
File "/usr/bin/bead/bead_cli/cmdparse.py", line 208, in dispatch
return run(args) or 0
File "/usr/bin/bead/bead_cli/workspace.py", line 110, in run
location = box.store(workspace, timestamp())
File "/usr/bin/bead/bead/box.py", line 169, in store
workspace.pack(zipfilename, freeze_time=freeze_time, comment=ARCHIVE_COMMENT)
File "/usr/bin/bead/bead/workspace.py", line 117, in pack
_ZipCreator().create(zipfilename, self, freeze_time, comment)
File "/usr/bin/bead/bead/workspace.py", line 280, in create
with zipfile.ZipFile(
File "/usr/lib/python3.8/zipfile.py", line 1251, in init
self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: '/srv/bead-box/latest/dropbox_encrypted/firmname_search_on_complex_names_20200508_20201211T152956094686+0100.zip'

Python:

3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]

Bead source:

origin: [email protected]:e3krisztian/bead.git
branch: master
date: 2020-10-23T16:23:28+02:00
hash: 1805272
version: v0.8.1

File "/usr/lib/python2.7/zipfile.py", line 1102, in write
   zinfo.file_size))
struct.error: 'L' format requires 0 <= number <= 4294967295'

https://bugs.python.org/issue9720 (from similar bug report numpy/numpy#4481)

fixed in
https://hg.python.org/cpython/raw-file/v2.7.4/Misc/NEWS
https://docs.python.org/release/3.3.1/whatsnew/changelog.html

(apparently the machine affected is running an out-dated linux)

Currently it is the other way, violating The Principle of Least Astonishment as the order differs from the general UNIX (cp, mv, ln, mount) order of arguments of <from> <to>.

<file> can be missing, in this case <mount-name> should already be defined, although unmounted.

The current name lib from librarian fits quite well, but it is widely used in programming with another meaning.

This project definitely has some similarity how books and periodicals are created with their identifiers being ISBN, ISSN.

The project name should imply one or more of:

• having resilience to manipulation
• having stable connections
• being peer to peer
• related to data
• similarity with press business
• relaxed workflow