archive version dependencies should be a first-class citizen about datafs HOT 5 CLOSED

These are obvious points but I just want to confirm that we understand and are talking about the same things. We need to be able to set and get version metadata. This would be set at archive creation and archive update and gotten at download (for example). Right now we want to get at least dependencies within the version but ultimately, their could conceivably be arbitrary metadata for each version.

from datafs.

I am thinking this could be one implementation

metadata =  {
	'archive_name': 'Big Climate Data',
	'source': 'a big data climate instrument',
	'description': 'waves and how you can get stoked and barreled'
	'versions': [('0.0.1', 'NOAA Charts from some time period'), ('0.1.0', 'Later NOAA Charts'), ('0.2.0', 'The most reliable NOAA Charts')]
	}

from datafs.

Is this for the dependencies? I was thinking more like

{
    '_id': big_climate_data',
    'authority_name': 'osdc',
    'archive_path': '/big/climate/data',
    'versioned': True,
    'metadata': {
        'source': 'a big data climate instrument',
        'description': 'waves and how you can get stoked and barreled'},
    'versions': [
      {'version': 1.0, etc..., 'dependencies': [(arch2, 1.3), (arch7, 1.6.2a1)]},
      {'version': 0.8, etc..., 'dependencies': [(arch2, 1.2), (arch7, 1.4)]}
    ]
}

Does that make sense? Each archive depends on other archives. The version metadata might be extendable beyond just version, checksum, algorithm, time stamp, author, dependencies, but this seems like the minimum requirement.

from datafs.

On DataArchive, we'll need the following updates:

DataArchive.update

def update(
    self,
    filepath, 
    cache=False, 
    remove=False, 
    bumpversion='patch', 
    prerelease=None, 
    dependencies=None, 
    **kwargs):
    
    ...

    self._update_manager(checksum, kwargs, version=next_version, dependencies=dependencies)

DataArchive.open

def open(
    self, 
    mode='r', 
    version=None, 
    bumpversion='patch', 
    prerelease=None, 
    dependencies=None, 
    *args, 
    **kwargs):
    ...
   
    updater = lambda *args, **kwargs: self._update_manager(
            *args, version=next_version, dependencies=dependencies, **kwargs)
    ...

DataArchive.get_local_path

similar to DataArchive.open

DataArchive._update_manager

def _update_manager(self, checksum, metadata={}, version=None, dependencies=None):

    # by default, dependencies is the last version of dependencies
    if dependencies is None:
        history = self.history
        if len(history) == 0:
            dependencies = []
        else:
            dependencies = history[-1]['dependencies']

    ....

from datafs.

#74

from datafs.