kg-construct / rml-io Goto Github PK
View Code? Open in Web Editor NEWRML-IO: Input/Output declarations for RML
Home Page: https://w3id.org/rml/io/spec
License: Creative Commons Attribution 4.0 International
RML-IO: Input/Output declarations for RML
Home Page: https://w3id.org/rml/io/spec
License: Creative Commons Attribution 4.0 International
XPath allows to use XML namespaces when selecting parts of an XML document.
However, (most) implementations require to register these namespaces before doing an XPath query.
RML does not specify how does this should happen currently:
CARML has an extension for this: https://github.com/carml/carml#xml-namespace-extension
and it came up in the past already a few times without a clear solution:
I see several examples where rml:query
is used as a property of rml:Source
.
And other examples where rml:query
is used as a property of rml:LogicalSource
.
My preference would be to have rml:query
be a property of rml:LogicalSource
. Because:
rml:iterator
, in that the evaluation of an rml:iterator
produces a list of Records, and the evaluation of rml:query
also produces a list of Records (in the case of relational databases: rows). Having these on the same resource type would simplify implementations.It could be very useful to filter records when accessing a Logical Source or filter triples when exporting to a Logical Target, for example:
Logical Source
Logical Target
rmld:filter
to a Logical Source and Logical Target. If not specified, no filtering is applied.rmld:Source
or rmld:Target
.
the reference formulations were introduced in RML but they were never clarified in the end.
now that we have seen some opportunities wrt what to add in the reference formulation, it might be good to introduce some more concrete examples in the spec
We should be able to specify how a data source specifies empty literals:
"NULL"
, ""
, ``, "random word"
null
in JSON for exampleIs a job for the DataIO spec: https://rml.io/specs/dataio/
Split of kg-construct/rml-core#16
Targets also can use relative paths, 'RelativePathSource' is kinda Source specific.
Drop 'Source' from the class name and move it outside the Source section into something separate in the spec.
This spec was originally meant to cover the target but currently it is also meant to cover the source description so we need to extend it to describe the source as well and not only the target.
Replaced by rml:SQL2008Query
and rml:SQL2008Table
There's a big focus on Target examples for Subject, Predicate, Object Maps but Function and Graph Maps are under specified in the spec.
Source was copied from Target but the subsections were never updated to explain rml:null
and rml:query
.
rml:serialization
must go
Section 8 of the document is quite verbose, mostly redundant, and as a result it contains errors due to copy-paste.
For instance, the output of Example in 8.4 is wrong:
# file:///data/dump1.nt
<http://example.org/0> <http://xmlns.com/foaf/0.1/name> "Kara Danvers" .
<http://example.org/1> <http://xmlns.com/foaf/0.1/name> "Alex Danvers" .
<http://example.org/2> <http://xmlns.com/foaf/0.1/name> "J'onn J'onzz" .
<http://example.org/3> <http://xmlns.com/foaf/0.1/name> "Nia Nal" .
# file:///data/dump2.nt
<http://example.org/0> <http://xmlns.com/foaf/0.1/name> "Kara Danvers" .
<http://example.org/1> <http://xmlns.com/foaf/0.1/name> "Alex Danvers" .
<http://example.org/2> <http://xmlns.com/foaf/0.1/name> "J'onn J'onzz" .
<http://example.org/3> <http://xmlns.com/foaf/0.1/name> "Nia Nal" .
Note that the second group of triples should refer to nicknames instead.
Is it really necessary to provide all possible examples? Is there a general rule on how this works, that we could write instead?
By default, the iterator is considered a row, if not specified:
* In the case of databases, CSV or TSV data sources, the value of the rml:iterator, if not specified, is a "row".
* In the case of XML and JSON data sources, it is a valid reference to an element or an object respectively considering the reference formulation specified.
The sentence "By default, the iterator is considered a row, if not specified:" seems not necessary, since it only applies to the first bullet (databases, CSV or TSV). And in case of XML and JSON it must be specified and there is no "row" concept, right?
<http://example.org/5> <http://xmlns.com/foaf/0.1/age> "" .
<http://example.org/5> <http://xmlns.com/foaf/0.1/name> "" .
Paginated JSON/XML/... is used extensively in the wild for Web APIs.
However, we don't support this in the current Logical Source.
See:
How are we going to keep up with newer versions of SQL?
Currently we have SQL:2008 as a reference formulation, following R2RML. But there are newer versions of SQL, the latest being SQL:2023. How are we going to keep up with these?
Once #20 is merged, adjust spec with slightly changes
rml:tarXz
and rml:tarGz
, but rest of documentation and tests use rml:tarxz
and rml:targz
Examples are not all updated to the new structure of Source and Target.
related to #13
id
variable. Here is a solution: rml:iterator """
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT ?id ?name ?age WHERE {
?person foaf:name ?name .
?person foaf:age ?age .
BIND(REPLACE(STR(?person),"http://example.org/", "") AS ?id) .
}
""";
rml:targzip
instead of rml:targz
rml:targzip
instead of rml:targz
id
in a reference, but there is no ?id
in the SPARQL Query or 0003 should look as follows:<#TriplesMap> a rml:TriplesMap;
rml:logicalSource [ a rml:LogicalSource;
rml:source <#VoIDSourceAccess>;
rml:iterator """
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT ?person ?name ?age WHERE { # HERE IS THE CHANGE
?person foaf:name ?name .
?person foaf:age ?age .
}
""";
rml:referenceFormulation formats:SPARQL_Results_CSV;
];
rml:subjectMap [ a rml:SubjectMap;
rml:reference "person"; # HERE IS THE CHANGE
];
A recent paper:
R2RML and the original RML specification defined that RML processors can perform data type inference from the SQL databases. Thus, mappings did not have to specify rr:datatype for RDF Literals to have the correct data type as the processor would retrieve this automatically from the SQL database.
However, RML did not expand this to other heterogeneous datasources such as XML or JSON which both provide data types in different ways: XML schemas, native JSON types, etc. Data type inference is still under discussion but might be moved to RML-IO because this RML module focuses on accessing and iterating over the data source.
and refers to kg-construct/rml-core#87. I don't see much discussion of datatype inference there, so I'm posting this issue here.
Here are a couple of considerations:
xsd:type
, especially focusing on XSD Datatypes but not ignoring custom datatypes like geo:wktLiteral, geo:gmlLiteral etc.
In the current draft both the Source
and Target
involve properties to describe at least the following aspects of a resource:
Additionally, a distinction is made between Source
/Target
and Access
, which creates a rather complex chain of resources. And, when we look at how Access is applied in the current proposal, its intent is to model a Data Source.
I think there is room to simplify the current design, by removing the distinction between Source
/Target
and Access
. And I think we can reuse the same Source
class for both the source and target side.
So in this way, we could have a mapping like this:
<#DCATSourceAccess> a dcat:Dataset;
dcat:distribution [ a dcat:Distribution;
dcat:downloadURL "https://rml.io/specs/rml-target/Supergirl.xml";
];
.
<#TriplesMap> a rr:TriplesMap;
rml:logicalSource [ a rml:LogicalSource;
rml:source <#DCATSourceAccess> ;
rml:referenceFormulation ql:JSONPath;
rml:iterator "$.[*]";
];
rr:subjectMap [ a rr:SubjectMap;
rr:template "http://example.org/{@id}";
rml:logicalTarget <#TargetDump1>;
];
rr:predicateObjectMap [ a rr:PredicateObjectMap;
rr:predicateMap [ a rr:PredicateMap;
rr:constant foaf:name;
];
rr:objectMap [ a rr:ObjectMap;
rml:reference "name/text()";
rml:logicalTarget <#TargetDump2>;
];
];
rr:predicateObjectMap [ a rr:PredicateObjectMap;
rr:predicateMap [ a rr:PredicateMap;
rr:constant foaf:nickname;
];
rr:objectMap [ a rr:ObjectMap;
rml:reference "nickname/text()";
];
];
.
<#TargetDump1> a rmlt:LogicalTarget;
rmlt:target <#VoIDDump1>;
.
<#TargetDump2> a rmlt:LogicalTarget;
rmlt:target <#VoIDDump1>;
.
<#VoIDDump1> a void:Dataset ;
void:dataDump <file:///data/dump1.nt>;
void:feature formats:N-Triples;
rmlt:serialization formats:N-Triples;
.
<#VoIDDump2> a void:Dataset ;
void:dataDump <file:///data/dump2.nt>;
void:feature formats:N-Triples;
rmlt:serialization formats:N-Triples;
.
A specific source type specification can then define how to infer source specific properties like encoding, compression, serialization/format, etc. Many standards already have properties that define these aspects. For example:
This would allow for more simply reusing already existing source descriptions as sources and targets, without having to duplicate information.
Would be nice to have test cases covering the complete specification:
Most of the tests are actually already used as examples in the spec.
issue: not possible to describe an input stream as a source
suggestion: add support for describing input streams. This will facilitate the usage of RML in transformation pipelines.
I am not sure I understand/agree on the use of rml:access
, could you elaborate a bit more the rational behind it?
Need to be transferred from the core RML spec.
I would suggest changing
"rml:null describes which data values inside the source should be considered as NULL. Defaults to the default NULL character if available. If none is available such as CSV, no values are considered NULL, unless specified. Example: CSV does not have a default NULL character, so no value is considered NULL. However, JSON has a NULL character specified: null, this one is used together with the ones specified through rml:null."
to
"rml:null
indicate which data values inside the source should be considered as NULL values. The value for this predicate defaults to the default NULL token of the underlying data model (e.g., NULL
in relational databases and null
in JSON) and are always processed as such. Some data models have no such default NULL token, such as CSV. When that is the case, then the empty string ϵ is considered NULL."
I understood that one can declare multiple rml:null, but that the default NULL token is always used. How would one otherwise try to process these NULL tokens. The problem, however, is with the phrasing of "unless specified" for CSV files. That gives the impression that rml:null can be "overwritten" for CSV. Is ϵ always considered rml:null? If yes, then we can posit it as such. If not, then this probably needs a section with examples.
The mentioned DCAT spec does not allow to refer to relative file paths (it allows for relative resources, but this is relative to the @base of the RML mapping file which is not what I mean).
What would the description of the file ./data/input.json
, (path relative to the mapping file) look like?
(you could rephrase my question to 'relative to the cwd' if that makes things easier, I can always set my cwd to the mapping file folder).
rml:referenceFormulation rml:JSONPath
)csvw:url
-> should be file:
In Section 3.1, there is the following sentence:
"In the case of databases, CSV or TSV data sources, the value of rml:iterator is considered a "row" and must not be specified."
The sentence above seems incorrect. First of all, what is it meant by "databases"? Are graph databases databases, or only relational ones are meant here?
Further, for relational databases, rml:iterator
CAN be used to express queries according to rml:SQL2008Query
reference formulation.
For some targets it might make sense to specify some extra properties. I have this use case where I want to append output to target file x, but overwrite target file y if it exists. So for files there are the typical write modes, e.g. like the python file modes w
, a
, etc.
In the spec these properties might translate to properties that apply to other kinds of targets as well, for instance databases.
The original RML spec supported literals in rml:source
and rmlt:target
on top of objects (CSVW, SD, etc.) such as
rml:logicalSource [ a rml:LogicalSource;
rml:source "/path/to/file.csv";
];
We kept supporting this in RML for historical reasons.
However, we have now the chance to drop it in favor of only objects which allow better descriptions.
Would that be a good idea? Or better keep it?
If dropped, the following example would become:
rml:logicalSource [ a rml:LogicalSource;
rml:source [ a csvw:Table;
csvw:url "/path/to/file.csv";
csvw:dialect [ a csvw:Dialect;
csvw:delimiter ";";
csvw:header "1"^^xsd:boolean;
];
];
This approach is way more descriptive and allows a lot to be re-used of existing specifications such as CSVW, SD, DCAT, etc.
Publication from @andimou https://dl.acm.org/doi/10.1145/2814864.2814873
https://www.w3.org/ns/formats/Format
is used in many places (test cases, documentation, etc.) instead of http://www.w3.org/ns/formats/Format
.
The rdfs:domain
of rml:logicalTarget
is defined to be rml:TermMap
, yet it is also used on rml:LanguageMap
which is not a rml:TermMap
.
Options:
rml:logicalTarget
rml:TermMap
and rml:LanguageMap
Nor the documentation, nor the test cases provide such examples (same for XML and JSON results, by the way). But I question the usefulness of rml:SPARQL_RESULT_TSV
. Taking the example of 0003, we would have the following TSV:
?person ?name ?age
<http://example.org/0> "Monica Geller" "33"
<http://example.org/1> "Rachel Green" "34"
<http://example.org/2> "Joey Tribbiani" "35"
<http://example.org/3> "Chandler Bing" "36"
<http://example.org/4> "Ross Geller" "37"
How should we iterate over those? We cannot treat them as regular TSV. The angle brackets should be removed from IRIs. Literals should be "cast" to their datatypes. And I have no idea what to do with blank node identifiers. Is it possible the group thought that the TSV output would be the same as CSV output, but with tabs?
Same question for JSON and XML representations of SPARQL queries: do they have bespoke iterations (i.e., not the same iterations as for "regular" JSON or XML files), or would iterating over them require a second iterator?
Compressed sources, do we want to handle this like this:
rml:source [
rml:access [
# what we had previously for `rml:source` is now here
# location & access (ex: WoT has security and location here)
];
rml:encoding enc:UTF-8;
rml:null "THIS IS A NULL VALUE";
rml:compression comp:Zip;
]
Or do we need to handle this as nested Logical Sources? This approach won't work for a ZIP file with multiple files.
The namespace prefix ql
is used as a qualifier for http://semweb.mmlab.be/ns/ql#
. Is that still the case? Should this be documented in this spec?
See RML Star
As I'm reading the target spec I'm wondering where is the definition of rmlt:LogicalTarget and associated properties.
As a reader I'd expect to have a self-contained spec that includes everything.
Consider example in specification:
<#SPARQLEndpoint> a rml:LogicalSource;
rml:source [ a rml:Source, sd:Service;
sd:endpoint <http://example.com/sparql>;
sd:supportedLanguage sd:SPARQL11Query;
];
rml:iterator "CONSTRUCT WHERE { ?s ?p ?o. } LIMIT 100";
rml:referenceFormulation formats:SPARQL_Results_CSV;
.
Since the iterator uses a CONSTRUCT
, the reference formulation format cannot be SPARQL_Results_CSV
. Suggest either using a SPARQL SELECT
form, or change the reference formulation format(?).
Generation of triples by using CHAR datatype column, resulting RDF literal is space-padded. (specification reference)
dcat:format
for the encoding (much like CSVW encoding)It seems to me that Dataset would never be the actual source, since it is an abstract description of a dataset. The source will always be a specific distribution, or a data service. So shouldn't that be the direct source that is linked to in the mapping?
A dcat dataset could have many distributions for example.
ex:dataset1 a dcat:Dataset ;
dcat:distribution ex:distribution1 , ex:distribution 2, ex:distribution3 ;
ex:distribution1 dcat:accessUrl <https://foo.bar/1> .
ex:distribution2 dcat:accessUrl <https://foo.bar/2> .
ex:distribution3 dcat:accessUrl <https://foo.bar/3> .
With this mapping:
ex:Source a rml:Source ;
rml:access ex:dataset1 ;
You don't know which of the distributions is intended.
issue: not possible to specify source encoding
suggestion: add support for describing source encoding. With the focus on text based sources it is important to be able specify the encoding of input sources.
In the examples, the target may be a VOID data or a DCAT dataset. Yet there is no constraint here.
So how should a processor know whether the target is given by void:dataDump or dcat:downloadURL or anything else?
If this is implementation-dependent, then it should be stated explicitly.
The current RML spec says
A logical source can be one of the following:
A base source (any input source or base table),
a view (in case of databases)
should that remain as it is?
There is an inconsistency between Figure 2 and Table in Section 4.2 (e.g., the domain of rml:compression
is rml:Target
, whereas in the table it is rml:LogicalTarget
.
For consistency with logical sources, maybe the figure is "correct", however I wonder why rml:serialization
has been put as a property of rml:LogicalTarget
. Would it not make it more sense to put it together with compression and encoding details?
Furthermore, should not we not specify rml:serialization
in logical sources as well? For instance, if I am reading a binary JSON file (BSON), then rml:serialization
could be used to denote this fact which is not currently captured by rml:referenceFormulation rml:JSONPath
.
We have a lot of Logical Target examples in how to use them within RML, do the same for Logical Source
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.