ropensci / datapack Goto Github PK

When trying to load datapackage using devtools::load_all(), I get an error now that seems to have an issue with creating a union class with setClassUnion. Related issues were reported, including r-lib/devtools#657 and r-lib/devtools#168. Code below illustrates the issue, and provides session info.

> library(devtools)
> load_all('.')
Loading datapackage
Error in .walkClassGraph(ClassDef, "contains", where, attr(ext, "conflicts")) : 
  the 'superClass' list for class “character”, includes an undefined class “EnumerationValue”
In addition: Warning message:
character(0) 
Error in setClassUnion("charOrNULL", c("character", "NULL")) (from DataPackage.R#176) : 
  unable to create union class:  could not set members "character"
> getwd()
[1] "/Users/jones/development/datapackage"
> sessionInfo()
R version 3.2.0 (2015-04-16)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.3 (Yosemite)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] devtools_1.8.0

loaded via a namespace (and not attached):
 [1] Rcpp_0.11.6        roxygen2_4.1.1     XML_3.98-1.1       digest_0.6.8      
 [5] bitops_1.0-6       git2r_0.10.1       magrittr_1.5       stringi_0.4-1     
 [9] uuid_0.1-1         hash_2.2.6         tools_3.2.0        stringr_1.0.0     
[13] RCurl_1.95-4.6     rversions_1.0.0    memoise_0.2.1      redland_0.2.0.9000

Hey, I've been looking for an existing dataset available online in the bag format. Does anyone know of any examples anywhere? I've struggled to find them.

-Luke

Currently SystemMetadata does not contain methods for adding access control roles. Add methods to:

• Initialize accessPolicy with a data frame during construction
• add a single access control rule with a subject and permission
• add a set of access rules as a data frame of (subject, permission) triples

This is very confusing in my view.

It took me 2 days to find out why

getRelationships()

does not return anything.

The dateUploaded slot value should be set to the current time when the SystemMetadata object
is uploaded to DataOne, not the object creation time, and should be set by the method or function
that is performing the upload. If the date is set in this way, then we can tell if the SystemMetadata
has been successfully uploaded.

datapack:serializeSystemMetadata() should group access rules by subject, so that a more condensed,
and easier to read (by a human) accessPolicy is created. For example, the following access policy has
repeated blocks for the same subjects, from
https://arcticdata.io/metacat/d1/mn/v2/meta/arctic-data.3958.1:

<accessPolicy>
<allow>
<subject>CN=arctic-data-admins,DC=dataone,DC=org</subject>
<permission>read</permission>
</allow>
<allow>
<subject>CN=arctic-data-admins,DC=dataone,DC=org</subject>
<permission>write</permission>
</allow>
<allow>
<subject>CN=arctic-data-admins,DC=dataone,DC=org</subject>
<permission>changePermission</permission>
</allow>
<allow>
<subject>http://orcid.org/0000-0002-2625-6747</subject>
<permission>read</permission>
</allow>
<allow>
<subject>http://orcid.org/0000-0002-2625-6747</subject>
<permission>write</permission>
</allow>
<allow>
<subject>http://orcid.org/0000-0002-2625-6747</subject>
<permission>changePermission</permission>
</allow>
<allow>
<subject>public</subject>
<permission>read</permission>
</allow>
</accessPolicy>

The package contents and relationships between objects will be serialzed to OAI ORE RDF/XML
described in http://www.openarchives.org/ore/1.0/rdfxml.

This task is dependent on Redland RDF R bindings.

Access policy information is currently stored in Java objects. Refactor methods to store this info in
native R objects.

insertRelationship methods currently store relationships between package objects in Java object. These
methods will be refactored to store relationships in native R objects.

datapack is ready for any additional evaluation and then release to CRAN.

The following tasks have been completed:

• build / check() has been rerun on all platforms in cran-comments.md
• all 1.0.0 github issues closed
• datapack build tested with current dataone repo

We need a vignette illustrating typical usage. Also show relationship to the dataone package. Consider whether this vignette shoudl be generated from a .Rmd file, and whether such a file should also generate the readme from issue #24.

When a relationship is inserted and either the subject of the object of the triplet is
as.character(NA), then a blank RDF node is created. This mechanism isn't currently
working.

To illustrate this

• create a DataPackage and add a DataObject
  • list objects in the DataPackage
  • delete the DataPackage
  • create a new DataPackage
  • list objects in the new, empty DataPackage
  • the new DataPackage contains objects from the original, deleted package

To recreate

library(datapackage)
testdf <- data.frame(x=1:10,y=11:20)
csvfile <- tempfile(pattern = "file", tmpdir = tempdir(), fileext = ".csv")
write.csv(testdf, csvfile, row.names=FALSE)
cn <- CNode("SANDBOX")
mnId <- "urn:node:mnSandboxUCSB2"
dp <- new("DataPackage")
sciObj <- new("DataObject", format="text/csv", user="uid=slaughter,ou=Account,dc=ecoinformatics,dc=org", mnNodeId=mnId, filename=csvfile)
addData(dp, sciObj)
getIdentifiers(dp)
rm(dp)
newDP <- new("DataPackage")
getIdentifiers(newDP)

Initial ideas for bags:

• Spatial data
• Collection of timeseries
• Something large to exercise lazy loading
• ?

A NULL value can be specified for subject, object in both Statement and Node in the redland
package to create blank nodes for which redland will assign a unique node id. Update insertRelationship to support this.

Hello,

I'm looking for a R library to read DataPackages http://data.okfn.org/doc/data-package
I'm not sure if this package deals with them.
If that's the case, maybe you should put a link to specification.

Kind regards

The DataONEorg/D1Object class was moved into datapackage, but needs to be renamed to not use the "D1" name and to remove the java dependencies.

Wondered if you folks were aware of "Data Packages" work associated with http://data.okfn.org/ In particular, you may be interested specifically in Tabular Data Package http://data.okfn.org/doc/tabular-data-package

In my opinion, this should not happen

Update the example to use tempdir() and discontinue running of example to
prevent CRAN errors

DataObject instances store their data in memory in the data slot, which works fine for small objects, but not well for larger objects. The size threshold shifts over time, but currently its reasonable to store objects of 0.5 to 1 GB in memory, but larger is unwieldy. To fix this, add the ability to store data on the local filesystem in a temporary cache, or as a reference to a file that the caller provides. This will allow data objects to be created that are much larger than can fit in memory. We'll need a new slot to store the file reference, and modified methods to detect when data are in a file or in memory.

The datapackage package depends on the redland package, so is not included in the automatic drat
builds, with the related issue in the drat.builder repo that mentions a problem with building packages
'deep' directory structures (i.e. redland): richfitz/drat.builder#9. Ensure that datapackage builds properly in preparation for the resolution to the drat.builder issue.

The class "DataPackage" requires this form for object creation:

pkg <- DataPackage()

Should we standarize on the form used by "DataObject" and only allow object
create through new?

do <- new("DataObject", ...)

Use as few arguments in the setGeneric function signature as necessary, provide flexibility
for users that wish to write their own methods that will be dispatched from our generic.

For our own uses, we need to have adequate parameters to dispatch on, so if we have multiple
methods implemented, we may need at least the first param, and sometimes the second, i.e.
addAccessRule(x, y) requires "x" to be able to dispatch to DataObject or SystemMetadata
implementations, "y" is required to dispatch based on a character string containing one access rule
or a data.frame containing possibly multiple access rules.

See the DataONE documentation of Data Packages and BagIt for details.

i wonder if this is a good use case for the tool being created here: https://github.com/bpbond/srdb

We are currently exploring the use of the DataONE DataPackage format using using OAI-ORE for the manifest and BagIt for serialization and transport.
Related packaging efforts need to be considered. Maybe we can/should support multiple serializations. None of the following really deal with all of the facets of packaging (metadata, container structure, identifier mapping, serialization formats), but each deals with some aspects. Related packaging specifications to consider:

• NetCDF
• HDF
• data-package.json
• package.jsonld (See the discussion in frictionlessdata/datapackage#110)

Question for discussion: which serialization(s) shall we support, in addition to ORE/BagIt?

When a SystemMetadata object is initialized from a parsed sysmeta XML document, if that
document doesn't contain a element, then the following error is produced
as shown in the following example:

> library(datapack)
> library(httr)
> library(XML)
> response <- GET("https://gmn.lternet.edu/mn/v1/meta/doi:10.6073%2FAA%2Fknb-lter-hfr.145.3")
> xml <- xmlParse(content(response, as="text"))
No encoding supplied: defaulting to UTF-8.
> sysmeta <- SystemMetadata(xmlRoot(xml))

 Error in UseMethod("xmlAttrs", node) : 
  no applicable method for 'xmlAttrs' applied to an object of class "NULL" 
10 xmlAttrs(xml[["replicationPolicy"]]) at SystemMetadata.R#260
9 .local(x, ...) 
8 parseSystemMetadata(sm_obj, x) at SystemMetadata.R#202
7 parseSystemMetadata(sm_obj, x) at SystemMetadata.R#183
6 .local(...) 
5 .Method(...) 
4 eval(expr, envir, enclos) 
3 eval(.dotsCall, env) 
2 standardGeneric("SystemMetadata") at SystemMetadata.R#169
1 SystemMetadata(xmlRoot(xml)) 
> 

The resourceMap will be parsed, and all objects in the map will be fetched and added to a new datapackage object.

Evaluate the data packaging format specified a http://dataprotocols.org/data-packages and
describe http://data.okfn.org/doc/data-package, and implement parsing / serialization
if appropriate. This issue was first raised by scls19fr here: https://github.com/ropensci/datapackage/issues/39

Currently the insertRelationship method is called to associate a science object with a metadata object (which adds the isdocumented / documents relationships).

An simpler alternative to having the user call insertRelationships is to add a third parameter
to the addData method to specify a metadata object to associate with a science object:

dp <- new("DataPackage")
metaObj <- new("DataObject", id=mdId, data, format="eml://ecoinformatics.org/eml-2.1.1", user, node)
addData(dp, metaObj)
sciObj <- new("DataObject", format="text/csv", user, node, filename=csvfile)
addData(dp, sciObj, metaObj)

When the sysmeta xml is parsed, the following error occurs when the access policy is
checked:

  no applicable method for 'xmlChildren' applied to an object of class "NULL"

BTW this problem can be see in the dataone package when a sysmeta w/o access policy is
uploaded, then getSystemMetadata() is called.

Especially for the first param in a S4 generic definition, use a name that is
sufficiently generic enough to handle all potential implementations, so for the
1st parameter, use 'x' as this could be potentially any data type.

insertRelationships updated a DataObject by updating a slot that contains a hash. This allows
the method to work without returning a DataPackage object, so for example, the following works:

dp <- new("DataPackage")
# Create a relationship
insertRelationship(dp, "/Users/smith/scripts/genFields.R",
   "http://www.w3.org/ns/prov#used",
   "https://knb.ecoinformatics.org/knb/d1/mn/v1/object/doi:1234/_030MXTI009R00_20030812.40.1")

Return the updated DataPackage, to prevent confusion and allow for the standard R way of
doing things:

dp <- new("DataPackage")
# Create a relationship
dp <- insertRelationship(dp, "/Users/smith/scripts/genFields.R",
   "http://www.w3.org/ns/prov#used",
   "https://knb.ecoinformatics.org/knb/d1/mn/v1/object/doi:1234/_030MXTI009R00_20030812.40.1")

Given that the underlying data model for datapackage resourceMaps is RDF, the insertRelationships
methods don't provide a way to specify the RDF resource type (URIs, blank nodes or litteral) for the subject, and object arguments (The predicate must always be a URI).

Possibly two approaches could be supported to resolve this:
- RDF types could be specified explicitly for subject, object
- specify to the underlying RDF processing to determines the appropriate
RDF resource type based on the string passed in.

This issue is related to ropensci/redland-bindings#14

Comments?

When invoking the command

R CMD INSTALL --no-multiarch --with-keep.source datapackage

the following error is printed

Loading required package: dataone
Error in .requirePackage(package) : 
  unable to find required package ‘dataone’
In addition: Warning message:
In library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE,  :
  there is no package called ‘dataone’

This occurs if the dataone package is not available for loading, i.e. remove.packages("dataone")

DataPackage, SystemMetadata, and probably D1Object would be main classes

• Decide whether to move SystemMetadata to datapackage
• Create R-based slots for all DataPackage state variables, including systemmetadata, and ORE relationships
• Replace all java-based implementations in DataPackage with native R
• Rename D1Object to something less DataONE centric
• Replace all java-based implementation in 'D1Object with native R

A SystemMetadata object is created for a DataObject when it is initialized, but there is no method or
initialization parameter to add access rules to this sysmeta object. The proposed method would
take arguments such as:

dataObj <- addAccessRules(dataObj, user="uid=slaughter,ou=Account,dc=ecoinformatics,dc=org", access=c("read", "write"))

Create an inst/CITATION file, including a DOI for the package, with the link
as https://github.com/ropensci/datapackage

Need to add dirs for testthat.

This involves serializing using the namespace "d1_v2.0" vs "d1".

I tried to install the package with devtools::install_github("ropensci/datapackage"), but got the following errors:

Downloading github repo ropensci/datapackage@master
Installing datapackage
Skipping 1 packages not available: redland
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ  \
  --no-save --no-restore CMD INSTALL  \
  '/private/var/folders/m8/68z5d14d7mv9qyf1bd69ych40000gn/T/RtmpQWZ1bF/devtools32b45c3ccb9/ropensci-datapackage-ff07ac0'  \
  --library='/Library/Frameworks/R.framework/Versions/3.2/Resources/library'  \
  --install-tests 

ERROR: dependency ‘redland’ is not available for package ‘datapackage’
* removing ‘/Library/Frameworks/R.framework/Versions/3.2/Resources/library/datapackage’
Error: Command failed (1)

Per CRAN request, the package name will be updated to 'dpack' or 'datapack'.

@mbjones, which one do you prefer?

Downstream packages will have to be updated as well, and tickets will be entered for

• dataone
• recordr

Need to set up initial package structure.

The document http://jenkins-1.dataone.org/jenkins/job/API%20Documentation%20-%20trunk/ws/api-documentation/build/html/design/DataPackage.html#generating-resource-maps outlines the requirements
of a resource map submitted to DataONE. Confirm that the resmaps created by datapackage
conform to this spec, especially D1 identifiers in URIs and in dcterms:identifiers.

We need a complete README describing the R package. Write it. Consider whether it could or should be generated by a Vignette or an .Rmd file that generates both it and a vignette.

The DOI associated with this package "doi:10.5063/F1QV3JGM" should be published
before the package is submitted to CRAN.