hapi-server / uri-templates Goto Github PK
View Code? Open in Web Editor NEWRepository of tools for URI templates
License: Apache License 2.0
Repository of tools for URI templates
License: Apache License 2.0
My reading of the uri_templates spec, is that templates like
http://example.com/data_$(j;pad=none).dat
are not correct because the spec itself must contain the context. The example has 1999-01, so I think that the template should be
http://example.com/data_$(j;pad=none;Y=1999).dat
While I can see the logic that this could be formatted, ignoring the year, this causes problems for my code which assumes these are set.
See section:
https://github.com/hapi-server/uri-templates/wiki/Specification#filenames-with-missing-date-information
I could probably introduce a mode into my code which would support this, but in this case I think the spec needs to be modified.
I was given a list of Juno Jupiter equator crossing times, which are formatted like "04/06/25 12:18 AM" or "09/17/25 05:30 PM"
I'm unable to use the time parsing library with these because AM/PM flag. (Note that $y is used for two-digit years.) The date command uses %p for this, and think we should consider $p.
The iso8601 parser hasn't ever supported week-of-year and I needed it for some application. I just realized I put the code in but the change was never documented in a ticket.
So TimeUtil.java the following now parses:
Code | equivalent time |
---|---|
2022-W13/P7D | 2022-03-28T00:00Z/2022-04-04T00:00Z |
2022-W01/P7D | 2022-01-03T00:00Z/2022-01-10T00:00Z |
2021-W52/P7D | 2021-12-27T00:00Z/2022-01-03T00:00Z |
See https://github.com/autoplot/dev/blob/master/demos/2022/20220312/demotimeWeek.jy
This fails because there is a field after a field with pad=none. The position of each field in the string is no longer known, and the code relies on delimiters to position things. However the code doesn't implement this properly, clearing length for the "m" field, but not the position of the "d" and "H" fields:
doTestTimeParser1( "$y $(m;pad=none) $(d;pad=none) $(H;pad=none)", "99 1 3 0", "1999-01-03T00:00/1999-01-03T01:00" )
I have both master and a main branch. Merge the changes and delete the master, then figure out where references exist. I'm setting up an action to use master, where I just mistakenly made changes.
I've been working on a translator to Python this week, and reviewing the JavaScript version of TimeUtil.js I see that overloaded methods don't translate well. For example, formatIso8601Time( int[] nn, int offset ) has a convenience method for the typical case where the offset is zero and it's just a seven-element time, with no offset specified: formatIso8601Time( int[] nn ). These overloaded methods don't translate into JavaScript and Python nicely, and should be avoided.
Other language features which fall into this category should also be documented here.
This works:
outn= URITemplate.formatRange( 'vg1pws_lr_$Y$m$(d;delta=10;phasestart=1979-01-01)_$(Y;end)$m$d_v5.10.png', str(t), str(t.add(datum('1days')) )
But this doesn't:
ut= URITemplate( 'vg1pws_lr_$Y$m$(d;delta=10;phasestart=1979-01-01)_$(Y;end)$m$d_v5.10.png').format( ut.format( str(t), str(t.add(datum('1days')) )
Move this stuff into pip, and the javascript into node when it's done.
I'd like for us to consider allowing
"PT0.5H" should be supported but it isn't. "PT0.5S" is supported, but not minutes and hours.
TimeUtil.java was always intended to be ported to other languages, and I'm curious how the port to JavaScript would go. I've been using the site https://www.jsweet.org/jsweet-live-sandbox/ to convert the code piece by piece, because it isn't working for the entire code, and it produces somewhat unreadable code anyway.
This is a table of function name in Java and if it is ported to JavaScript:
Java | JavaScript |
---|---|
final int TIME_DIGITS = 7; | Y |
final int DATE_DIGITS = 3; | Y |
String reformatIsoTime(String exampleForm, String time) | Y |
String monthNameAbbrev(int i) | Y |
int monthNumber(String s) throws ParseException | Y |
String[] countOffDays(String startTime, String stopTime) | Y |
String nextDay(String day) | Y |
String previousDay(String day) | Y |
String ceil(String time) | Y |
String floor(String time) | Y |
String normalizeTimeString(String time) | Y |
String isoTimeFromArray(int[] nn) | Y |
String formatIso8601TimeRange( int[] nn ) | Y |
String formatIso8601Time( int[] nn, int offset ) | Y |
String formatIso8601Time( int[] nn ) | Y |
String formatIso8601Duration(int[] nn) | Y |
int[] now() | Y |
int[] isoTimeToArray(String time) | Y |
int dayOfYear(int year, int month, int day) | Y |
int monthForDayOfYear( int year, int doy ) | Y |
long toMillisecondsSince1970(String time) | Y |
void normalizeTime(int[] time) | Y |
int dayOfWeek( int year, int month, int day ) | Y |
void fromWeekOfYear( int year, int weekOfYear, int[] time ) | Y |
int[] parseISO8601Duration(String stringIn) | Y |
int[] parseISO8601Time( String string ) | Y |
int[] parseISO8601TimeRange(String stringIn) | Y |
int julianDay(int year, int month, int day) | Y |
int[] fromJulianDay(int julian) | Y |
int[] subtract(int[] base, int[] offset) | Y |
int[] add(int[] base, int[] offset) | Y |
This was moved over from hapi-server/server-java#4
I was just testing the JS implementation and ran into a potential problem:
const filename = "ac27_crn1926_199722300-199725000.gif";
const template = "ac27_crn$x_$Y$j00-$(Y;end)$(j;end)00.gif"
ut = new URITemplate(template);
console.log(ut.parse(filename));
Prints:
[1997, 8, 11, 0, 0, 0, 0, 1997, 1, 250, 0, 0, 0, 0]
So it gets the month/day of month correct for the start time, but is still using the DOY with month = 1 for the end time.
For example - let’s say we have:
>>> filename = "https://spdf.gsfc.nasa.gov/pub/pre_generated_plots/kp_plots/ace/gif/ac_199811900-199812000.gif"
>>> template = "https://spdf.gsfc.nasa.gov/pub/pre_generated_plots/kp_plots/ace/gif/ac_$Y$j00-$(Y;end)$(j;end)00.gif"
>>> from URITemplate import URITemplate
>>> ut = URITemplate(template)
>>> ut.parse(filename, {})
[1998, 4, 29, 0, 0, 0, 0, 1998, 4, 30, 0, 0, 0, 0]
This works as expected, but when the filename has additional characters after the date, it still matches and extracts the times, e.g.,
>>> filename = "https://spdf.gsfc.nasa.gov/pub/pre_generated_plots/kp_plots/ace/gif/ac_199811900-199812000-this-shouldnt-match.gif"
>>> ut.parse(filename, {})
[1998, 4, 29, 0, 0, 0, 0, 1998, 4, 30, 0, 0, 0, 0]
If it includes additional characters before the date, it throws an exception:
>>> filename = "https://spdf.gsfc.nasa.gov/pub/pre_generated_plots/kp_plots/ace/gif/ac_this-shouldnt-match-199811900-199812000.gif"
>>> ut.parse(filename, {})
Traceback (most recent call last):
File "/Users/ewgrimes/find_gif_times/URITemplate.py", line 1089, in parse
digit = int(field)
ValueError: invalid literal for int() with base 10: 'this'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/ewgrimes/find_gif_times/URITemplate.py", line 1156, in parse
raise Exception('fail to parse digit number %d: %s' % (idigit, field))
Exception: fail to parse digit number 1: this
Is there a way to check if a string matches the template exactly?
Thanks!
The Radio and Plasma Wave Group at the University of Iowa likes to put files in 100-day directories:
https://pds-ppi.igpp.ucla.edu/data/CO-V_E_J_S_SS-RPWS-4-SUMM-KEY60S-V1.0/DATA/KEY_PARAMS/
https://pds-ppi.igpp.ucla.edu/data/CO-V_E_J_S_SS-RPWS-4-SUMM-KEY60S-V1.0/DATA/KEY_PARAMS/T$Y$x/RPWS_KEY__$Y$j_0.TAB
meaning any files from for year 2000 day of year 200-266 are to be found in T20002XX/. Presently you are forced to use $x, which will result in the unnecessary listing of three out of four directories.
Also there are 4-month blocks of data. I think this can be described:
https://space.physics.uiowa.edu/voyager/data/VGPW_0101/DATA/FULL1/T0005_08/ could be
https://space.physics.uiowa.edu/voyager/data/VGPW_0101/DATA/FULL1/T$y$(m)_$(m,delta=4)
I just noticed we used "id" to allow software to refer to a enumeration field, while $x has "name". Seems like id should be used in both places.
I just noticed in the spec that $y (two-digit-year) has an argument start which can indicate where the 1900/2000 logic should switch (1950-2049 vs 1970-2069). I don't think this was implemented in the Java, Python, and JavaScript versions.
I'm thinking of using the JavaScript implementation in our new gif_walk tool at SPDF, but noticed there isn't a license yet (or one that I could find). Can we add a license (preferably something FOSS)?
A search of the spase-group.org site does not yield a hit for the offical v1 pdf document that was once there. In fact, the 2.3 SPASE xsd files refer to tsds.org/uri_templates. After we clean up this page, we should create a tagged Markdown + PDF file release that will be permanent links.
According to the spec, you can specify the case using modifiers:
fmt=abbrev|full - (optional); default=abbrev
case=lc|uc|cap - (optional); default=lc
Possibilities:
$(b;fmt=abbrev;case=lc): jan, feb, ...
$(b;fmt=abbrev;case=uc): JAN, FEB, ...
$(b;fmt=abbrev;case=cap): Jan, Feb, ...
$(b;fmt=full;case=lc): january, february, ...
$(b;fmt=full;case=cap): JANUARY, FEBRUARY, ...
$(b;fmt=full;case=uc): January, February, ...
Note: in the spec, "cap" and "uc" are flipped for the full and abbreviated cases.
More importantly, these don't seem to work for me (in Python or JavaScript); here's an example in Python, e.g., lower case:
from URITemplate import URITemplate
from datetime import datetime
month = "$(b;case=lc)"
ut = URITemplate(month)
now = datetime.now()
ut.format(now.isoformat(), now.isoformat())
Gives:
'Feb'
fmt
doesn't seem to be working either:
month = "$(b;fmt=full;case=lc)"
ut = URITemplate(month)
ut.format(now.isoformat(), now.isoformat())
Gives:
'Feb'
The URI_templates code, which is exposed as a servlet here http://jfaden.net/AutoplotServlet/URI_Templates.jsp. This should be extracted into its own independent library. I believe the problem is that the Autoplot version uses DatumRange objects to represent time spans and its FileSystem objects to do remote listings.
It doesn't look like len and pad work with x for formatting, even though the interface allows values to be passed in. For example, this Autoplot script should work:
from java.util import HashMap
import sys
addToSearchPath( sys.path,
'https://cottagesystems.com/hapi/uri_templates/dist/UriTemplatesJava.jar',
monitor )
from org.hapiserver import URITemplate
urit = URITemplate('/tmp/ap/$(x;name=sc;len=6;pad=_).dat')
m= HashMap()
m.put('sc','Apple')
print urit.format(None,None,m)
m.put('sc','Boy')
print urit.format(None,None,m)
Bob suggested adding a command line API, and I've added and extended this. This can parse and format time ranges, like so:
unix> java -jar UriTemplatesJava.jar --formatRange --timeRange='1999-01-01/1999-01-03' --template='http://example.com/data_$(d;pad=none).dat'
http://example.com/data_1.dat
http://example.com/data_2.dat
and
unix> java -jar UriTemplatesJava.jar --parse --template='http://example.com/data_$(d;pad=none;Y=1999; m=5).dat' --name='http://example.com/data_1.dat'
1999-05-01T00:00:00.000000000Z/1999-05-02T00:00:00.000000000Z
Also the name and timeRange parameters should allow for -, meaning that stdin will be used to feed in a list of names or timeRanges.
Time range strings with shared date were never implemented, like "2023-01-18T17:00/18:00" or "2013-01-01/07-01." This is part of the ISO8601 standard and should have been included.
This is done for the Java implementation, but Python and JavaScript implementations need this as well.
There's no list of the implementations of the URI_Templates specification. This should go on the readme.md page.
At the U Iowa Plasma Wave Group, we produce 10-day plots for out PDS volumes which always start on January 1st of the year. This is a nice system because you always know what the names will be for any year without having to enumerate from some phase start. I propose we introduce "phasereset=Y" for this, to match the existing phasestart:
('$Y$m$d_$(Y;end)$m$(d;delta=10;phasereset=Y)')
There would also allow for phasereset=m or phasereset=d.
This might be the first time that two intervals overlap, and I could see this causing problems. I suggest then where there is overlap at the end of the year (20191227_20200106.PNG and T20200101_20200111.PNG) then the second interval would be preferred.
The spec https://github.com/hapi-server/uri-templates/wiki/Specification has three broken links associated
The first two are
The content of the above two pages should be moved here.
There is also a reference to a file that Jon may have tried to upload and I can't find anywhere
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.