attributeList needs to support custom units, unit matching about eml HOT 22 CLOSED

@mbjones Can you point me to an example defining a custom unit? E.g. how would I define a custom unit for "dollars"? Is there a linked-data solution to leverage existing definitions of currency?

Also trying to think out the workflow for a typical user here. I don't want a user to give up attempting to write data into EML just because their units aren't supported and they can't be bothered writing a STMML definition (though perhaps with a good API that can be easy enough...).

Because EML is much more relaxed about defining character attributes, this makes it tempting to encode undefined units as characters with something like <nominal><textDomain><definition>Dollars (US). I know we don't want to promote such hacks, be we also want an easy-to-use interface. Would love feedback.

from eml.

A related issue is something I encounter in my own work: how to encode simulation data in which the units are arbitrary? For instance, I might wish to indicate that I am simulating a population density or the costs of a conservation activity, but it seems wrong to label these as number Per Hectare or dollars when in fact the scaling is arbitrary.

from eml.

Here's an example of a custom unit definition:

  <additionalMetadata>
    <unitList>
      <unit id="numberPerSixtyMetersSquared" multiplerToSI=".0166667" name="numberPerSixtyMetersSquared" parentSI="numberPerMeterSquared" unitType="arealDensity">
        <description>Number per sixty meters squared</description>
      </unit>
    </unitList>
  </additionalMetadata>

which comes from a PISCO metadata document that shows pretty nice and complete metadata (available from https://knb.ecoinformatics.org/knb/metacat/pisco_subtidal.151.2/xml).

Creating custom units is easy when dealing with physical quantities like length or mass (as defined by NIST), as these all have base SI units and are easily convertible. However, it gets tricky with other things that appear to be quantities, but really are not (such as currency). There is no standard or base unit for currency, in that the value of every currency fluctuates over time (2009 dollars are different from 2010 dollars), and so currencies are not 'unit's in the traditional sense. You could define a UnitType for Currency, then define a base unit for that (say Dollars), and then define all other currencies as a conversion from the base Dollar, but of course the conversions would be wrong as the values fluctuated. You might have better luck defining a unit for a currency for a year, which are fairly well established (e.g., a '1970Dollar'), and then have each of the other annualized currency units convert from that, which would be more accurate, at least within the bounds of an annualized mean value.

Other types of units that you'll likely find problematic are for dimensionless values, such as counts, percentages, and ratios. For example, there is technically no unit for the count of animals in a plot, or for the percent cover of a plant (m^2/m^2), or for the concentration of a substance (ppm, ppb, or ml/ml). Yet its still useful to know what was measured in the numerator and denominator of the ratio. There has been lots of discussion on how to handle these cases, and its an ongoing issue we've tried to address more thoroughly in OBOE with our semantic treatment of units there, which separates the measured value from the entity being measured.

The LTER has also developed a 'Best Practices' for assigning units that you may find useful (http://im.lternet.edu/sites/im.lternet.edu/files/LTERunitBestPractices_V13.pdf).

from eml.

Ah, great information. thanks for the example. For units that are not
just combinations of basic SI units I can just ignore the attributes like
parentSI and multiplierToSI? Essentially that would just provide a name
and a description, but preferable to the character hack.

Meanwhile, for the standard units I was thinking it might be nice to allow
a reml user to configure their prefered base units, and provide a function
to automatically convert imported data tables to these units. Is there an
intelligent way to go about doing that?

Not to go too much on a currency tangent per se, since I'm mostly using
this as a way to think about addressing custom unit that isn't just a
rescaling. I assume someone in the linked data world has already figured
this out. e.g. http://sdmx.org/ provides European Central Bank data with
daily exchange rates to each currency. Seems like it would be sufficient
to indicate a currency unit (e.g. dollars) and a date (1970, today,
etc)....

In the context that I frequently encounter it, this is more about
indicating that the numbers reflect monetary value than that they reflect
meaningful dollar amounts that can be compared to other dollar amounts.
Like you suggest for other abstract units, it would probably be best to
present this in a de-dimensionalized form that still indicates the kind of
measurements involved, e.g. dollars over dollars.

On Tue, Jul 2, 2013 at 10:04 PM, Matt Jones [email protected]:

Here's an example of a custom unit definition:

Number per sixty meters squared

which comes from a PISCO metadata document that shows pretty nice and
complete metadata (available from
https://knb.ecoinformatics.org/knb/metacat/pisco_subtidal.151.2/xml).

Creating custom units is easy when dealing with physical quantities like
length or mass (as defined by NIST), as these all have base SI units and
are easily convertible. However, it gets tricky with other things that
appear to be quantities, but really are not (such as currency). There is no
standard or base unit for currency, in that the value of every currency
fluctuates over time (2009 dollars are different from 2010 dollars), and so
currencies are not 'unit's in the traditional sense. You could define a
UnitType for Currency, then define a base unit for that (say Dollars), and
then define all other currencies as a conversion from the base Dollar, but
of course the conversions would be wrong as the values fluctuated. You
might have better luck defining a unit for a currency for a year, which are
fairly well established (e.g., a '1970Dollar'), and then have each of the
other annualized currency units convert from that, which would be more
accurate, at least within th e bounds of an annualized mean value.

Other types of units that you'll likely find problematic are for
dimensionless values, such as counts, percentages, and ratios. For example,
there is technically no unit for the count of animals in a plot, or for the
percent cover of a plant (m^2/m^2), or for the concentration of a substance
(ppm, ppb, or ml/ml). Yet its still useful to know what was measured in the
numerator and denominator of the ratio. There has been lots of discussion
on how to handle these cases, and its an ongoing issue we've tried to
address more thoroughly in OBOE with our semantic treatment of units there,
which separates the measured value from the entity being measured.

The LTER has also developed a 'Best Practices' for assigning units that
you may find useful (
http://im.lternet.edu/sites/im.lternet.edu/files/LTERunitBestPractices_V13.pdf
).

—
Reply to this email directly or view it on GitHubhttps://github.com/ropensci/reml/issues/12#issuecomment-20395868
.

Carl Boettiger
UC Santa Cruz
http://carlboettiger.info/

from eml.

To convert units simply requires the conversion factors, which can be parsed out of the standard unit dictionary and then from any custom units provided. I think it might be useful to provide a separate unit conversion package that can load a list of units and their conversion factors, and then allow for straightforward conversion of any dataframe attribute that has an attached unit label. Poking around, I'm not seeing an R package for labeling units and doing conversion, but I didn't look for long. See http://stackoverflow.com/questions/7214781/converting-units-in-r

One issue of unit conversion is that people should be careful when they do conversions, as there is often a change in precision when doing unit conversion, and you can lose some sampling basis information. The custom unit example I gave above was one where the sampling design was in a sixty square meter plot, and so its important to know that for some types of analysis (e.g., species/area relationships). So, it would be good if reml would facilitate conversion, but not make automatically convert to standard units unless the user requests it.

from eml.

Excellent points! Will hold off on unit conversions for the time being.
It might be worth hitting on these unit issues in the manuscript too.

Wrapping existing unit conversion tools make senses, though it seems the
information for the conversions is already encoded in the EML... @duncantl
might have some thoughts on how we might do this elegantly..

On Wed, Jul 3, 2013 at 10:54 AM, Matt Jones [email protected]:

To convert units simply requires the conversion factors, which can be
parsed out of the standard unit dictionary and then from any custom units
provided. I think it might be useful to provide a separate unit conversion
package that can load a list of units and their conversion factors, and
then allow for straightforward conversion of any dataframe attribute that
has an attached unit label. Poking around, I'm not seeing an R package for
labeling units and doing conversion, but I didn't look for long. See
http://stackoverflow.com/questions/7214781/converting-units-in-r

One issue of unit conversion is that people should be careful when they do
conversions, as there is often a change in precision when doing unit
conversion, and you can lose some sampling basis information. The custom
unit example I gave above was one where the sampling design was in a sixty
square meter plot, and so its important to know that for some types of
analysis (e.g., species/area relationships). So, it would be good if reml
would facilitate conversion, but not make automatically convert to standard
units unless the user requests it.

—
Reply to this email directly or view it on GitHubhttps://github.com/ropensci/reml/issues/12#issuecomment-20433365
.

Carl Boettiger
UC Santa Cruz
http://carlboettiger.info/

from eml.

Hi @cboettig,

So having read the above and your and @mbjones responses to my units question, you're suggesting two ways to currently handle non-standard units:

1. I could export an STMML file description for the unit if I can find it for example in the LTER unit registry. I am not sure then how to then take this STMML definition into the metadata when using the rEML package?
2. I could write my own custom unit definition as above, providing at a minimum a unit name and description. Again though, being an introductory user, I am unclear on how to get this definition into the metadata in rEML?

Do these non-standard unit definitions always have to be put into the additionalMetadata element?

Could you also give me some pointers on how I could use the "REST service" to get unit descriptions into the EML package?

thanks again,
Karen

from eml.

Hello again,
Please could you give an example of how exactly you would specify and define a character string column in the unit.defs list in the EML package?
many thanks,
Karen

from eml.

Hi @cboettig,
Many thanks for all the helpful suggestions. The issue of sorting out units seems like a potential sticking point in getting people (including us) to use the package (and EML in general). With current capabilities it will be a lot of work to get the units for our files sorted out. We're thinking about a workflow for quickly identifying units already described and for describing those that aren't.
Daniel

from eml.

Yeah, this is definitely an issue for us to address on our end rather than
something to leave users to work around. We need a workflow that is simple
but still encourages all the best practices linked in the pdf above.
Thanks to you both for bravely trying out the package before we have closed
these issues and released a stable version to Cran, your feedback really
helps.

Hope to have both these issues you raise addressed soon.

Carl Boettiger
http://carlboettiger.info

sent from mobile device; my apologies for any terseness or typos
On Apr 7, 2014 5:24 PM, "Daniel Falster" [email protected] wrote:

Hi @cboettig https://github.com/cboettig,
Many thanks for all the helpful suggestions. The issue of sorting out
units seems like a potential sticking point in getting people (including
us) to use the package (and EML in general). With current capabilities it
will be a lot of work to get the units for our files sorted out. We're
thinking about a workflow for quickly identifying units already described
and for describing those that aren't.
Daniel

Reply to this email directly or view it on GitHubhttps://github.com//issues/12#issuecomment-39799576
.

from eml.

Hi @cboettig,

Thanks for looking into these issues - appreciate that this is a work in progress not yet released on Cran, and that these things take time to iron out.

In the meantime, it would be great if you could clarify a couple of things:

1. Could you give an example of how exactly you would specify and define a character string column in the unit.defs list in the EML package? I'm really stuck with this one and it's holding me back from getting a lot of the data described. For example, how would I specify the units for the character variables of site, collector and species in the following:

 col.defs <- c("site" = "site where sammple was taken",   
              "collector" = "who collected sample",
              "spp" = "species name",
              "height" =  "plant height",
              "diameter" = "diameter at 10% tree height, or breast height for trees >20m tall",
              "sampled" = "was individual sampled for biomass and traits?")

unit.defs = list(site, 
                 collector,
                 species, 
                 "m",
                 "cm",
                 c(y = "yes", NA ="no"))

1. Do non-standard unit definitions always have to be put into the additionalMetadata element (as in the example given by mbjones above)?

many thanks!
Karen

from eml.

Hi @karen-ross

Good questions. You can put anything you like as a description for character units -- in EML files that I've looked at, it's reasonably common to simply repeat the column definition.

 col.defs <- c("site" = "site where sample was taken",   
              "collector" = "who collected sample",
              "spp" = "species name",
              "height" =  "plant height",
              "diameter" = "diameter at 10% tree height, or breast height for trees >20m tall",
              "sampled" = "was individual sampled for biomass and traits?")

unit.defs = list("site" = "site where sample was taken",   
              "collector" = "the first name of the person who collected sample",
              "spp" = "species scientific name (as genus_species)",
                 "m",
                 "cm",
                 c(y = "yes", NA ="no"))

However, I think you should ask yourself if it would be more reasonable to consider many of the things you are treating as characters as factors instead, and define the units as such. The README shows an example of encoding species names as factors in the unit data. Likewise you might do the same for collector. This allows you to specify more information about the factor -- for instance, your collector column may only use first names for convenience of data entry, so in the metadata you might provide the full name of the person.

I believe this is an area of some contention among R users at least, with some people preferring to encode factors as character strings because of the strange things R can do to when coercing between factors if you aren't careful. But I think having them as factors makes sense from the R point of view as well, as it is somewhat more robust to be subsetting by factors than by factor == level then by string == pattern. Metadata-wise, characters would really only be things that are not levels but literal data strings that you are unlikely to subset by. (I agree that's vague, one person's characters are another person's factors).

Also, EML isn't really trying to tell the user whether they should treat data as factors or characters, or anything else really, it's just there to help them describe the data they have. So don't take my advice too seriously, and just provide a description of the characters if that is what you want.

Yes, I believe you should always provide the unit definition in additionalMetadata -- don't worry though, once I've written that utility the R package will generate that for you so it won't require any extra work!

from eml.

@mbjones I'm finally getting around to this units implementation and wondered if you might clarify a few quick things. I've reviewed the Best Practices guide but it left a few questions open about the other elements of the schema beyond naming the unit itself that I can't quite figure out from the schema documentation

• Is there a table of base class / unitTypes somewhere? I don't see them declared for the standard unit documentation
• Would a user sometimes need to define a unitType?
• Are there cases that would justify using other attributes, such as defining the unit abbreviation? (Seems like that is a case of confusing attribute and unit metadata, as the Best Practices guide discusses, but I see it is available in the STMML vocabulary so am wondering if I should implement it).

Lastly, @karen-ross's example raises an additional question where the unitType is not obvious to me. The units are m^2/hectare, I would think of defining this as dimensionless as follows, but maybe that loses the notion of area per area:

 <additionalMetadata>
 <metadata>
    <unitList>
      <unit id="metersSquaredPerHectare" multiplerToSI=".0001" 
               name="metersSquaredPerHectare"
               parentSI="dimensionless" unitType="dimensionless">
        <description>meters squared per hectare (or perhaps a more specific description, 
        like that this is a trunk area per land area or whatever it is??)
        </description>
      </unit>
    </unitList>
  </metadata>
  </additionalMetadata>

Perhaps it would it be better to define a new unitType of areaPerArea and then use that? Would that go elsewhere than in the unitList ?

Thanks for clarifying these issues!

p.s. I think the example you gave above comes from the older schema that didn't use the <metadata> element, but we need for in eml-2.1.1 like in my example, right?

from eml.

Heya @cboettig,

A few remarks on your questions:

• We've listed the unitTypes that ship with EML in the eml-unitDictionary.xml file that is included in the EML distribution. That list, in turn, is derived from the list of base quantities and derived quantities provided by NIST in their page on NIST units. NIST provides a much more complete description of the concepts behind units and quantities. The only new baseQuantity that has been discussed is currency, which isn't technically a physical quantity at all, and is only barely definable. Currencies are certainly not in the same category as for example meter in terms of their use as a unit because their values change arbitrarily over time.
• People will from time to time need to define new unitTypes. For the most part, this should only be needed when a dimensional analysis of a unit shows that there is no other unitType that shares that dimensionality. For example, if someone wants to create a unit for density in numbers per unit area, they could use the existing unitType arealDensity, and they would provide the conversion factor multiplierToSI for the base unit for that unitType, in this case numberPerMeterSquared. You can tell that a unit is the base unit for its unitType because it will have a multiplierToSI of 1.
• When defining units in STMML, it is certainly useful to include attributes such as the abbreviation; you'll see that most units in eml-unitDictionary.xml include these attributes in the STMML definitions.

Regarding @karen-ross's example, that is a classic case of a dimensionless quantity (area per area) in which the two units in the numerator and denominator cancel. In her case, a square meter is the base unit for area, and a hectare is 10^4 square meter, so the ratio of those is 10^-4, which is a dimensionless number. The problem with these dimensionless numbers is that you can't treat any two dimensionless numbers as equivalent, as sometime it is important to know what kind of entity was measured in the numerator and denominator. For example, the mass ratio of two substances (e.g., a CN ratio) will be dimensionless, but it is critical to know that the numerator is Carbon and the denominator is Nitrogen, even if they are both measured in the unit grams that cancel. With our work on OBOE, we have extended our definitions of units to include these semantics of the Entity so that these ratio measures can be better labeled, compared, and understood than is possible with traditional metadata standards like EML.

If you are creating some methods for creating unitTypes or units, it would be useful to compare them to the standard set and make sure the new unit doesn't match an existing one (in terms of parentSI and multiplierToSI), and that new unitTypes don't match an existing unitType in terms of dimensional analysis. Warnings in those areas would help reduce duplication (although it technically doesn't matter because we should be able to programmatically find any duplicates).

from eml.

Hi @mbjones!

Thanks much! More questions on unitType to come I think in a different issue, but one more quick question for now:

I'm not clear which id I should prefer to use in the describes element of additionalMetadata element containing the unitList (something that I guess didn't exist in the earlier schema in the example above). It seems most precise to use the id of the unit being defined, but if the metadata element contains the entire unitList this would create a problem. It seems that the logical options are either:

• use the id of the attributeList in the additionalMetadata/describes, or
• create one unitList per unit, such that each unit got it's own metadata element and could refer directly to the unit definition.

Regardless of the choice we should be able to figure out the unit definition by matching the unit names, so the describes reference doesn't seem crucial. What do you recommend here?

from eml.

Hi @mbjones and @cboettig,

Thanks for suggestions re new unit types. I have been looking at the
standard list of EML units with @karen-ross find suitable base SI units for
most of our variables. However, I cannot find any suitable base units for
the following:

• "g m-1",
• "mg mm-1",
• "mm-1"

So I guess I'll need to define some new unit types for those. But it's
unclear from examples above what I should list for several of the
additional meta data fields when there is no logical parent, i.e. just
leave as empty? If so, would the following suit (for g m^-1)

<additionalMetadata>
 <metadata>
    <unitList>
      <unit id="gramsPerMeter" multiplerToSI="" name="gramsPerMeter"
       parentSI="" unitType="">
        <description>Grams per meter</description>
      </unit>
    </unitList>
  </metadata>
  </additionalMetadata>

many thanks!


On 19 April 2014 13:57, Carl Boettiger <[email protected]> wrote:

> Hi @mbjones <https://github.com/mbjones>!
>
> Thanks much! More questions on unitType to come I think in a different
> issue, but one more quick question for now:
>
> I'm not clear which id I should prefer to use in the describes element of
> metadata element containing the unitList (something that I guess didn't
> exist in the earlier schema in the example above). It seems most precise to
> use the id of the unit being defined, but if the metadata element
> contains the entire unitList this would create a problem. It seems that
> the logical options are either:
>
>    - use the id of the attributeList in the metadata/describes, *or*
>    - create one unitList per unit, such that each unit got it's own
>    metadata element and could refer directly to the unit definition.
>
> Regardless of the choice we should be able to figure out the unit
> definition by matching the unit names, so the describes reference doesn't
> seem crucial. What do you recommend here?
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/ropensci/EML/issues/12#issuecomment-40860018>
> .
>

from eml.

I wanted to check with @mbjones if I have this right. A new unit type for "g m-1" would look like this:

<unitType id="massPerLength" name="massPerLength">
  <dimension name="mass"/>
  <dimension name="length" power="-1"/>
</unitType>

Then one could use that as the unit type:

 <unit id="gramsPerMeter" multiplerToSI=".001" name="gramsPerMeter" parentSI="KilogramsPerMeter" unitType="massPerLength", abbreviation="g/m">
        <description>Grams per meter</description>
</unit>

Note above we have also defined the conversion to the parent SI, since SI base unit for mass is kilogram, not gram. (See NIST). I'm not sure if it is also necessary for us to define the unit KilogramPerMeter, in this case?

 <unit id="kilogramsPerMeter" name="kilogramsPerMeter" unitType="massPerLength" abbreviation="kg/m">
        <description>Kilograms per meter</description>
</unit>

@dfalster @karen-ross Looks like I overlooked this one, sorry. I think this is a case where you need to define a new unitType, which I think I still need to add an interface to support. Will add once I pin this down. Thanks again for your questions.

from eml.

@cboettig That looks right to me -- although running it through an EML validator would help catch any errors. I think it would be good to define the kilogramsPerMeter but I'm not sure its strictly required from a schema perspective. But its useful if you are referencing it.

from eml.

@mbjones @leinfelder One more quick question, should both the <unitType> and <unit> elements go in a unitList element like Matt has in the example above, under additionalMetadata/metadata?

Thanks.

from eml.

@cboettig Yeah, it should. If you look at stmml.xsd, you'll see that UnitList is defined as:

<xsd:sequence>
        <xsd:element ref="unitType" minOccurs="0" maxOccurs="unbounded"/>
        <xsd:element ref="unit" minOccurs="0" maxOccurs="unbounded"/>
</xsd:sequence>

So basically, list your unitType's first, and then all of your units.

from eml.

Perfect, done. We now have S4 classes for the stmml:unit and stmml:unitType which should make it relatively straight forward to add new units and new unitTypes, e.g. see my unit test for an example:

I hope to add some more features to the helper functions eml_define_unit to help identify if a unit is already defined with fuzzy matching / abbreviation matching; and if not, identify if an appropriate unitType is already defined. At the moment, these functions don't provide much for training wheels.

Since these tasks are now in #110 and #120, will (finally) close this issue thread.

from eml.

Wonderful! I look forward to trying out new functionality. Thanks @cboetigg
and @mbjones

On 31 July 2014 08:38, Carl Boettiger [email protected] wrote:

Closed #12 #12.

—
Reply to this email directly or view it on GitHub
#12 (comment).

from eml.