Add a geotiff writer with https://commons.apache.org/proper/commons-imaging/(?) that writes raw data values.

Most of the text for the ERDDAP user interface (other than a few static .html documents) is in messages.xml.
Long ago, in my previous job, I was able to use commercial machine translation software to make translations to French, Spanish, German, Italian, and Dutch.
The translation was crude but usable. For a couple of the languages, a human hand edited the translations. That was great.

Now there are web sites (and hopefully web services) that can translate to/from 100's of languages and they can do a much, much better job than the software I used.

The goal of this project is to create versions of ERDDAP using different languages.
(This was the part of the original design of ERDDAP. That's why most of the text is in messages.xml.)
There are two major tasks:

Major task # 1) Make a system which parses messages.xml, sends the parts to a machine translation system, and creates variants of messages.xml for other languages (e.g., messages.de.xml). With this alone, a given ERDDAP could have a user interface which uses another language.

Major task # 2) (Optional) Make a structural change to ERDDAP so that one ERDDAP can serve messages in different languages, e.g., https://baseUrl/erddap/... would be the English (original) interface while https://baseUrl/erddap/de/... would be the German interface, etc. But some other actual system might be better.

Clearly, many people outside the US would appreciate this (even the Brits, who could switch to the British spellings of some words).

There are several interrelated issues for major task # 1. Different people could take on different issues.

1. There is still text in ERDDAP which has hard-coded text. These should be converted to use text from messages.xml. The easiest way to identify these is to translate messages.xml and then see what text in the ERDDAP UI still needs to be translated.

2. Sometimes, 2+ tags in messages.xml are used to hold parts of a sentence. ERDDAP combines the parts with other information to make a sentence. This often doesn't work well with translation software because different languages put different parts of the sentence in different places (e.g., Germans put the main verb at the end). Hopefully, there are few of these. These should probably be changed to the pseudo character entity system or the MessageFormat system described below.

3. Often, a message exists with a whole sentence or block of text in messages.xml, but uses a pseudo character entity (e.g., &externalLinkHtml;) to mark where ERDDAP will plug in some piece of information at runtime. This works with translation software if it simply ignores the unknown character entity and passes it through in the results. You'll have to test that this works with the translation system you choose, or deal with the problem (e.g., substitute a nonsense word temporarily).

4. Often, a message uses Java MessageFormat-style substitution placeholders like {0} and {1}. In these messages, any single quote ' is written as ''. So the translation system will have to deal with the placeholders (or you will) and you'll have to convert to one single quote before translation and convert to 2 single quotes after translation.

5. Often the message is HTML content. The translation system will have to preserve the HTML tags.

6. To do the actual translation, you probably have to write a script which goes through messages.xml, and for each tag, cleans it up (eg convert 2 single quotes into 1 single quote), passes it to the translator, then inserts the result in a new messages.xml (e.g. with each single quote converted to 2 single quotes).

7. It would be nice to have a system which makes it easy to translate the entire messages.xml or translate/update just a single tag. Thus, if one tag is changed for a new release of ERDDAP, there should be a way to update the translated version in the other versions of messages.xml.

Skills required: Script writing (Java?). Fluency in (or at least familiarity with) a language other than English (to use as the test case) is useful. You need to know a little Java to deal with issues # 1 and # 2.

Difficulty: This is a huge task if done properly (lots of editing of message formats), but only medium technical difficulty. This is easily 3 months work (after after getting rolling with ERDDAP). Maybe this is too big. I don't know. Or just dive into this and see how far you can get in the available time. I suspect that just doing # 6 and # 7 (and dealing with issues # 3, # 4, # 5) might be a useful project on it's own (which you might be able to do crudely in 1 week). It would also highlight pieces of text that are hard coded in ERDDAP (issue # 1) or which need modifying in ERDDAP (issue # 2).

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

ERDDAP has a group of Converters which do useful things (usually conversions) related to ERDDAP. See
https://coastwatch.pfeg.noaa.gov/erddap/convert/index.html
It would be nice to add a ColorBar converter which takes a color bar specification (e.g., palette=Rainbow, Range=1 to 10000, Scale=log, NSections=4, Continuity=Continuous) and then converted a data value into a color or vice versa.

Skills required: Java programming.

Difficulty: Pretty easy. There are already Converters which could be used as templates for this converter. There is code in ERDDAP to convert a data value to a color. You probably have to write code for the reverse, but that is probably straightforward. Maybe 1-2 weeks of work (after you figure out ERDDAP).

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

This was requested by Lynne Tablewski and others.

There doesn't seem to be a release published with the war and content zip for v2.16. Can one be created? Thank you!

[This has been assigned to Harry Singh of ONC.]

ERDDAP has a Keywords converter which converts a CF Standard Name to/from a GCMD Science Keyword.
Unfortunately, the conversion file was made a long time based on CF Standard Names and GCMD keywords which are now out-of-date.
The project here is to update the conversion file (WEB-INF/classes/gov/noaa/pfel/erddap/util/CfToGcmd.txt).

Skills required: Understanding/familiarity with scientific terms

Difficulty: 2-4 weeks of work.

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

Hello Bob

As you can see on this page

http://40.68.46.120:8080/erddap/wms/erddap_dd09_b52e_bc1c/index.html

says 'To get an image file with a map with a transparent background, use

http://40.68.46.120:8080/erddap/wms/jplMURSST41/request?service=WMS&version=1.3.0&request=GetMap&bbox=-179.99,-89.99,180.0,89.99&crs=EPSG:4326&width=360&height=180&bgcolor=0x808080&layers=jplMURSST41:analysed_sst&styles=&format=image/png&transparent=TRUE
'
But when i had tried to open this link in my browser i got error : HTTP Status 404 - Resource not found: /erddap/wms/jplMURSST41/request (unmatched WMS request (2))
screenshot link: http://imgur.com/puqYrGM

I would be very grateful for help!

I always try to write code so that it runs efficiently, but there is always lots of room for improvement if someone can put more time into it. So this project involves two steps: 1) find slow spots in the code 2) rewrite those sections of code that you think you can make faster.

To find slow sections of code: 10+ years ago, I used a profiler to identify slow sections of code. That was very productive. There are more profiler options now than when I did it. See for example https://www.baeldung.com/java-profilers

It's hard to get useful information by profiling a running ERDDAP because the tasks ERDDAP is working on vary with the requests. Instead, profilers work well when you can run a specific hard task repeatedly to both look for slow spots and test your optimizations. Fortunately, ERDDAP has lots of unit tests which are speed oriented. A few of them are
Table.testReadASCIISpeed();
Table.testBigAscii();
Table.testReadJsonSpeed();
Table.testReadNDNcSpeed();
Table.testReadOpendapSequenceSpeed();
Table.testSaveAsSpeed();
but there are lots more in other classes, notably all of the EDDGrid and EDDTable subclasses.
You should run one of these tests (at a time) (via TestAll.java) and look for slow sections of code that could be optimized.

General advice: things to optimize are often the low level routines that are called a huge number of times. So be on the look out for the code in the com.cohort and e.g., SSR classes that could be optimized.

Also, much of the low level code in ERDDAP was written long ago. Changes in Java since then may offer better ways to write the code. E.g., NIO (which is now old itself) offered big speed ups -- I tried to use NIO extensively after it came out.

Regarding rewriting slow sections of code:
Of course, optimizing is a tricky issue. There is often a trade off where you get more speed, but at the expense of more memory use or more complex code. We don't want to increase memory use beyond a little bit more for a short time. And we don't want complex code that is hard for someone else to figure out and maintain in the future. And we never want optimizations that introduce bugs or are in any way less robust or less safe. Basically, we're looking for low hanging fruit.

Skills required: Java programming. You'll need to know or figure out how to use a profiler.

Difficulty: Medium difficult. It's kind of a vague project. You aren't simply writing new code to add some new feature. The challenge is finding time-consuming sections of code that could be rewritten to be faster. This is easily 1-3 months work (after you figure out ERDDAP). The more time and effort you put into it, the more results you'll get out.

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

Hi,

I was browsing though the source code and ran across a BagIt download implementation. I'm not totally sure-but is there an extra write of the data to the filesystem? I saw this comment. If so, I solved this problem in DataONE's data server by creating SpeedBagIt which gets around duplicating data for serving datasets. If so, I'd be more than happy to optimize the BagIt routine here.

Erddap can produce some very interesting bathymetry images, like this:

transparentPng?elev%5B(20.12):(20.143045462072454)%5D%5B(145.07385854272752):(145.09996556940996)%5D&.draw=surface...

But if the request includes one lat/lon extent which is out of range, the returned image is fully blank:
transparentPng?elev%5B(20.10):(20.143045462072454)%5D%5B(145.07385854272752):(145.09996556940996)%5D&.draw=surface...

The problem presents when using ERDDAP transpsarentPng service as a tile server, and the user unwittingly navigates beyond the edge.

It would be nicer if ERDDAP would anyow return the included data plus some blank data for padding (in this case blank the part of the png not represented in the data).

If cdm_data_type is a requirement for ERDDAP, can we get rid of it in favor of featureType?

We have been writing featureType attributes into our NetCDF files following this guidance for CF-1.6 Discrete Sampling Geometry (DSG) files.

So for time series data, for example, we specify featureType=timeSeries.

But we have also been writing cdm_data_type as it seems required by ERDDAP.

Following Unidata's apparent use of cdm_data_type we have been specifying cdm_data_type=Station for time series data (this is how CF-1.6 DSG files appear in ToolsUI, for example).

ERDDAP, however, apparently wants cdm_data_type to have the same values as featureType and complains that Station is not valid, suggesting it should be timeSeries instead.

@lesserwhirls, is there any reason for us to include cdm_data_type from a THREDDS perspective?

If not, we would like to get rid of it in the future and just go with featureType, and have it work in both THREDDS and ERDDAP.

Hello,
At BODC we have loaded a discrete water samples dataset -
https://linkedsystems.uk/erddap/tabledap/samples_cobs_20200901.html

The original data is stored in BODC in a relational database so to pull it together we created a denormalized view so the data are sparse now.

We haven't assigned a fill value, leaving it at NaN as they don't exist in the relational database. However we noticed on the html download these are blank which is fine but on the graph feature, it is plotting these points (greyed out).

Please see -
https://linkedsystems.uk/erddap/tabledap/samples_cobs_20200901.graph?time%2Cdepth%2CPHOSAATX&time%3E=2002-08-07T11%3A34%3A39Z&time%3C=2011-11-10T00%3A00%3A00Z&longitude%3E=-4.094&longitude%3C=-2.923&latitude%3E=53.008&latitude%3C=54.18&.draw=markers&.marker=1%7C5&.color=0xFFFFFF&.colorBar=%7C%7C%7C0%7C2%7C&.bgColor=0xffccccff

I can only get rid of the grey points by adding the parameter as a contraint which i'm not sure is all that intuitive for an end user. Would it be possible for the graphing feature to exclude NaNs on the plot?

https://linkedsystems.uk/erddap/tabledap/samples_cobs_20200901.graph?time%2Cdepth%2CPHOSAATX&time%3E=2002-08-07T11%3A34%3A39Z&time%3C=2011-11-10T00%3A00%3A00Z&longitude%3E=-4.094&longitude%3C=-2.923&latitude%3E=53.008&latitude%3C=54.18&PHOSAATX%3E=0&PHOSAATX%3C=10&.draw=markers&.marker=1%7C5&.color=0xFFFFFF&.colorBar=%7C%7C%7C0%7C2.5%7C&.bgColor=0xffccccff

Thanks,
Emma
British Oceanographic Data Centre

Consider adding rel="next" and rel="prev" links on the index page
https://support.google.com/webmasters/answer/1663744?hl=en&ref_topic=4617741

for example in the head of the page here:
http://coastwatch.pfeg.noaa.gov/erddap/info/index.html?page=1&itemsPerPage=3

EDDTableFromFiles now supports derived variables
https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#scriptSourceNames
It would be nice if EDDTableAggregateRows also supported derived variables (and perhaps also global:attName and variable:varName:attName variables).

Skills required: Java programming.

Difficulty: relatively easy. You'll have to figure out / understand how ERDDAP works, but the code to do this project is already in EDDTableFromFiles. You just need to copy and adapt it for EDDTableAggregateRows. Maybe 1 week of work after you are familiar with ERDDAP.

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

In this ncISO issue: Unidata/thredds#75 we realized that we needed to get NcISO to specify gmd:protocol in the ISO metadata so that CKAN/pyCSW would return the right link-types when queried using CSW.

@pacioos or @BobSimons, do we need to do the same thing with ERDDAP to get it to write out gmd:protocol for OpenDAP, ERDDAP, WCS, etc links so that ISO produced by ERDDAP can allow CKAN/pyCSW to return link types that are not None?

When I look at a specific example of ERDDAP ISO:
http://coastwatch.pfeg.noaa.gov/erddap/metadata/iso19115/xml/osuSstClimate_iso19115.xml
I see gmd:protocol a few times, but only with value http.

Hi,

I have been trying to read files from a shared folder on the network. I have tried various ways for ERDDAP to recognise that network share.
eg.
<fileDir>I:\</fileDir>
<fileDir>I:/\</fileDir>
<fileDir>//server_ip/shared_folder</fileDir>
<fileDir>////server_ip//shared_folder</fileDir>

But I am still struggling to get ERDDAP to read that network share. Is it a known issue or am I missing something?

Thanks.

Investigate Java to native compilers like https://www.graalvm.org/

Hi Bob,

We are trying a new method for ingesting NRT data - https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#EDDTableFromHttpGet

We have loaded some data in ERDDAP but noticed that when we used the graph feature an error occurs and the dataset is subsequently dropped from ERDDAP.

We are not sure if we have a XML issue in our setup or if this is a bug?

Details below.
Thanks,
Emma

The log errors

gov.noaa.pfel.erddap.dataset.WaitThenTryAgainException: There was a (temporary?) problem. Wait a minute, then try again. (In a browser, click the Reload button.)
(Cause: com.cohort.util.SimpleException: ERROR while reading /local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl: columnName=timestamp not found in longitude,latitude,airTemp,time.)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFilesCallable.call(EDDTableFromFilesCallable.java:154)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFilesCallable.call(EDDTableFromFilesCallable.java:31)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.getDataForDapQuery(EDDTableFromFiles.java:3696)
 at gov.noaa.pfel.erddap.dataset.EDDTable.getTwawmForDapQuery(EDDTable.java:1663)
 at gov.noaa.pfel.erddap.dataset.EDDTable.saveAsImage(EDDTable.java:3861)
 at gov.noaa.pfel.erddap.dataset.EDDTable.respondToDapQuery(EDDTable.java:3117)
 at gov.noaa.pfel.erddap.Erddap.doDap(Erddap.java:4362)
 at gov.noaa.pfel.erddap.Erddap.doGet(Erddap.java:556)
Caused by: com.cohort.util.SimpleException: ERROR while reading /local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl: columnName=timestamp not found in longitude,latitude,airTemp,time.
 at gov.noaa.pfel.erddap.dataset.EDDTableFromHttpGet.readFile(EDDTableFromHttpGet.java:370)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromHttpGet.lowGetSourceDataFromFile(EDDTableFromHttpGet.java:322)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.getSourceDataFromFile(EDDTableFromFiles.java:3029)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFilesCallable.call(EDDTableFromFilesCallable.java:109)
 ... 32 more

#0 get data from /local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl
  Table.readJsonlCSV(/local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl) done. nColumns=4 nRows=13 TIME=1ms
>> EDDTableFromHttpGet inbound_NRT_5f88_a3eb_59e0 nThreads=1 thread=https-openssl-apr-8443-exec-6 task=0: caught while reading file=/local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl: com.cohort.util.SimpleException: ERROR while reading /local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl: columnName=timestamp not found in longitude,latitude,airTemp,time.
  Table.readJsonlCSV(/local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl) done. nColumns=4 nRows=13 TIME=1ms
  Table.readFlatNc /local/tomcat/content/erddapBigDir/dataset/e0/inbound_NRT_5f88_a3eb_59e0/badFiles.nc finished. nColumns=3 nRows=1 TIME=1ms
inbound_NRT_5f88_a3eb_59e0 addBadFile: penz_2016-03.jsonl
  reason=SimpleException: ERROR while reading /local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl: columnName=timestamp not found in longitude,latitude,airTemp,time.
 at gov.noaa.pfel.erddap.dataset.EDDTableFromHttpGet.readFile(EDDTableFromHttpGet.java:370)
  Table.saveAsFlatNc /local/tomcat/content/erddapBigDir/dataset/e0/inbound_NRT_5f88_a3eb_59e0/badFiles.nc1141171690 done. nColumns=3 nRows=2 TIME=3ms
Table of badFiles successfully written. nRows=2
gov.noaa.pfel.erddap.dataset.WaitThenTryAgainException: There was a (temporary?) problem. Wait a minute, then try again. (In a browser, click the Reload button.)
(Cause: com.cohort.util.SimpleException: ERROR while reading /local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl: columnName=timestamp not found in longitude,latitude,airTemp,time.)

cat /local/tomcat_shared/content/erddap/data/inbound-NRT/penz/penz_2016-03.jsonl
["stationID","time","latitude","longitude","airTemp","waterTemp","timestamp","author","command"]
["penz","2016-03-30T12:37:55Z",10.1,-150.1,17.23,12.3,1.601565967722E9,"tomG",0]
["penz","2016-03-30T12:37:55Z",10.1,-150.1,17.3,12.3,1.601566535098E9,"tomG",0]
["penz","2016-03-30T13:37:55Z",10.1,-150.1,17.2,12.81,1.601566535098E9,"tomG",0]
["penz","2016-03-30T14:37:55Z",10.1,-150.1,17.72,13.11,1.601566535098E9,"tomG",0]
["penz","2016-03-30T15:37:55Z",10.1,-150.1,17.5,12.793,1.601566535098E9,"tomG",0]
["penz","2016-03-30T16:37:55Z",10.1,-150.1,17.1,1.873,1.601566535098E9,"tomG",0]
["penz","2016-03-30T17:37:55Z",10.1,-150.1,17.3,12.63,1.601566535098E9,"tomG",0]
["penz","2016-03-30T12:37:55Z",10.1,-150.1,17.3,12.3,1.601568148226E9,"tomG",0]
["penz","2016-03-30T13:37:55Z",10.1,-150.1,17.2,12.81,1.601568148226E9,"tomG",0]
["penz","2016-03-30T14:37:55Z",10.1,-150.1,17.72,13.11,1.601568148226E9,"tomG",0]
["penz","2016-03-30T15:37:55Z",10.1,-150.1,17.5,12.793,1.601568148226E9,"tomG",0]
["penz","2016-03-30T16:37:55Z",10.1,-150.1,17.1,1.873,1.601568148226E9,"tomG",0]
["penz","2016-03-30T17:37:55Z",10.1,-150.1,17.3,12.63,1.601568148226E9,"tomG",0]

XML snippet used

<dataset type="EDDTableFromHttpGet" datasetID="inbound_NRT_5f88_a3eb_59e0" active="true">
    <reloadEveryNMinutes>1440</reloadEveryNMinutes>
    <updateEveryNMillis>-1</updateEveryNMillis>
    <fileDir>/local/tomcat_shared/content/erddap/data/inbound-NRT/</fileDir>
    <fileNameRegex>.*\.jsonl</fileNameRegex>
    <recursive>true</recursive>
    <pathRegex>.*</pathRegex>
    <metadataFrom>last</metadataFrom>
    <sortedColumnSourceName></sortedColumnSourceName>
    <sortFilesBySourceNames>stationID, time</sortFilesBySourceNames>
    <fileTableInMemory>true</fileTableInMemory>
    <accessibleViaFiles>true</accessibleViaFiles>
    <addAttributes>
        <att name="cdm_data_type">Point</att>
        <att name="Conventions">COARDS, CF-1.6, ACDD-1.3</att>
        <att name="creator_name">BODC</att>
        <att name="creator_type">institution</att>
        <!--att name="cdm_timeseries_variables">stationID, longitude, latitude</att-->
        <att name="subsetVariables">station_id, longitude, latitude</att>
        <att name="httpGetDirectoryStructure">stationID/2months</att>
        <att name="httpGetKeys"> -- keys here --</att>
        <att name="httpGetRequiredVariables">stationID, time</att>
        <att name="infoUrl">???</att>
        <att name="institution">BODC</att>
        <att name="keywords">air, airTemp, author, bodc, british, centre, command, creamt, data, identifier, latitude, longitude, oceanographic, station, stationID, temperature, tester, time, timestamp, water, waterTemp</att>
        <att name="license">[standard]</att>
        <att name="sourceUrl">(local files)</att>
        <att name="standard_name_vocabulary">CF Standard Name Table v55</att>
        <att name="subsetVariables">stationID</att>
        <att name="summary">a test of the from http method for us by NOC and British Oceanographic Data Centre (BODC)</att>
        <att name="testOutOfDate">now-1day</att>
        <att name="title">creamT tester</att>
    </addAttributes>


    <dataVariable>
        <sourceName>stationID</sourceName>
 <destinationName>stationID</destinationName>
        <dataType>String</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="long_name">Station ID</att>
            <!--att name="cf_role">timeseries_id</att-->
        </addAttributes>
    </dataVariable>


    <dataVariable>
        <sourceName>time</sourceName>
        <destinationName>time</destinationName>
        <dataType>String</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="long_name">Time</att>
            <att name="standard_name">time</att>
            <att name="time_precision">1970-01-01T00:00:00Z</att>
            <att name="units">yyyy-MM-dd&#39;T&#39;HH:mm:ss&#39;Z&#39;</att>
        </addAttributes>
    </dataVariable>


    <dataVariable>
        <sourceName>latitude</sourceName>
        <destinationName>latitude</destinationName>
        <dataType>double</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="colorBarMaximum" type="double">90.0</att>
            <att name="colorBarMinimum" type="double">-90.0</att>
            <att name="long_name">Latitude</att>
            <att name="missing_value" type="double">NaN</att>
            <att name="standard_name">latitude</att>
            <att name="units">degrees_north</att>
        </addAttributes>
   </dataVariable>


    <dataVariable>
        <sourceName>longitude</sourceName>
        <destinationName>longitude</destinationName>
        <dataType>double</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="colorBarMaximum" type="double">180.0</att>
            <att name="colorBarMinimum" type="double">-180.0</att>
            <att name="long_name">Longitude</att>
            <att name="missing_value" type="double">NaN</att>
            <att name="standard_name">longitude</att>
            <att name="units">degrees_east</att>
        </addAttributes>
    </dataVariable>
 
    <dataVariable>
        <sourceName>airTemp</sourceName>
        <destinationName>airTemp</destinationName>
        <dataType>float</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="long_name">Air Temp</att>
            <att name="missing_value" type="float">NaN</att>
            <att name="units">degrees C</att>
        </addAttributes>
    </dataVariable>
  

  <dataVariable>
        <sourceName>waterTemp</sourceName>
        <destinationName>waterTemp</destinationName>
        <dataType>float</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="long_name">Water Temp</att>
            <att name="missing_value" type="float">NaN</att>
            <att name="units">degrees C</att>
        </addAttributes>
    </dataVariable>

    <dataVariable>
       <sourceName>timestamp</sourceName>
        <destinationName>timestamp</destinationName>
        <dataType>double</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="long_name">Timestamp</att>
            <att name="missing_value" type="double">NaN</att>
            <att name="time_precision">1970-01-01T00:00:00.000Z</att>
            <att name="units">seconds since 1970-01-01T00:00:00Z</att>
        </addAttributes>
    </dataVariable>


    <dataVariable>
        <sourceName>author</sourceName>
        <destinationName>author</destinationName>
        <dataType>String</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="ioos_category">Identifier</att>
            <att name="long_name">Author</att>
        </addAttributes>
    </dataVariable>


    <dataVariable>
        <sourceName>command</sourceName>
        <destinationName>command</destinationName>
        <dataType>byte</dataType>
        <!-- sourceAttributes>
                     </sourceAttributes -->
        <addAttributes>
            <att name="flag_meanings">insert delete</att>
            <att name="flag_values" type="byteList">0 1</att>
            <att name="ioos_category">Other</att>
            <att name="long_name">Command</att>
            <att name="missing_value" type="byte">127</att>
        </addAttributes>
    </dataVariable>


</dataset>

Make it so that ERDDAP can be a data source for MapServer (or GeoServer?) and thus use its WMS, WCS, etc services.
Make it so that suitable datasets in ERDDAP are automatically added to MapServer.
Add MapServer to the standard ERDDAP distribution (so it is all automated).

Does this makes sense? It is possible? You'll need to research and ask around: Which is the best program to link to (most widely used, best features, most up-to-date OGC services, most likely to be maintained in the long run, has an appropriate license, possible to use ERDDAP as a data source, and thus overall most suitable for ERDDAP)?

Skills required: Java programming. Some experience with the OGC world. You'll have to know or learn how to work with MapServer. You'll probably have to write code (in what language?) to enable MapServer to get data from ERDDAP datasets.

Difficulty: Hard. You'll have to do a lot of research and planning just to make sure what you do is possible and is the best approach. Then it is probably difficult to make the connection so that ERDDAP is a data source for MapServer. Then you have to make sure it works in all the difficult cases (or block off cases that don't make sense). This could easily take 2-3 months.

Mentor: Bob Simons (main author of ERDDAP). I can help you with figuring out ERDDAP, but I am not familiar with the GIS world or MapServer. You'll have to solve problems mostly on your own.

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

(Thanks to Rob Fuller for the idea.)

ERDDAP comes with a tool called ArchiveADataset which extracts a subset of a dataset (perhaps in multiple files) and packages the results in a zip or tgz file in a format suitable for submission to NOAA's NCEI archive (or for other purposes).
Unfortunately, NCEI hasn't standardized the format they want. So different groups inside and outside of NCEI have requested different formats and I have added support for them to ArchiveADataset as they have arisen. This latest request is a collaboration between NOAA's NMFS and NCEI. It is very similar to the existing BagIt option in ArchiveADataset, but they want to be able to include other files in the BagIt file.

If you chose to work on this, you'll contact and work with Nazila Merati of NMFS to make sure you make the format that they want.

Skills required: Java programming.

Difficulty: Relatively easy. ArchiveADataset already exists and has several output options. You'll just make a clone of one of them that works a little differently. Perhaps 2 weeks work (after you figure out ERDDAP and ArchiveADataset).

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

Not sure if this is intended or not but there is no download/release for 1.82 despite there being a change log entry for it on https://coastwatch.pfeg.noaa.gov/erddap/index.html

Currently setup is driven through setup.xml (read through ResourceBundle2 from cohort) which in turn builds up the EDStatic making those settings available throughout the code.

In modern microservice and container setups (e.g. axiom/docker-erddap, specially in combination with nginx or apache reverse proxy forwarding) however a new separation of concerns is pushing up. Who creates what parts of the setup.xml is not as straightforward any more. This could be facilitated by allowing the setup to be driven by environment variables like ERDDAP.baseUrl and ERDDAP.baseHttpsUrl

These would then be read by System.getenv and injected in stead of the settings in the XML file. Given that ResourceBundle2 already functions as a decorator for the java native ResourceBundle, this extra layer could maybe added on that level?

Hello Bob.

I have tried to access my NetCDF files through WCS and failed. Can you help me?

First of all i have an .nc file which you can download via following link:

ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis2/surface/mslp.1979.nc

Following the guides i've filled setup.xml file and using GenerateDatasetsXml tool i've generated body of dataset.xml file.

So, server have been set-uped successfully. And i suppose you can access it here

http://40.68.46.120:8080/erddap

However, i can't download subset of file through WCS because i have no idea which variables must be chosen here on the file's page http://40.68.46.120:8080/erddap/tabledap/allDatasets.html . Clearly, i've tried o lot of combinations of variables and fail in each try.

Please, if you have some questions fell free to ask them. And I'm really waiting for yours answer.

MQTT is a system for efficiently transmitting real time data to/from sensors.
https://en.wikipedia.org/wiki/MQTT

1. Since ERDDAP has datasets with near real time data (e.g., in data files which change periodically), it would be nice to add an MQTT real-time data service to ERDDAP. This would allow suitable EDDTableFromFiles datasets to be made available as publishers so that MQTT subscribers could be quickly and efficiently notified when there is new data. Much of this task would be to create a way for ERDDAP administrators to identify suitable datasets and to have the datasets be able to identify/cache-in-memory the most recent row of data so that it is available for the MQTT publishing. Note that 1 dataset may represent 100's of stations/sensors, e.g.,
   https://coastwatch.pfeg.noaa.gov/erddap/files/cwwcNDBCMet/nrt/
   Subscribers should be able to subscribe to one, some, or all stations in a dataset.

2. Make EDDTableFromHttpGet (which is the only type of dataset in ERDDAP that supports data ingest) into an MQTT subscriber.
   Then an EDDTableFromHttpGet dataset could connect to a sensor which is acting as MQTT publisher in order to gather data from the sensor.

ERDDAP is a Java servlet, so the solution needs to be cross-platform.
Existing libraries to be used: Paho from Eclipse? An MQTT broker (e.g., Mosquitto)?

Possible people to talk with about needs: Someone in IOOS(?) has Sensor Data Ingest Project.

Skills required: Java programming. I think you can learn about MQTT while working on the project.

Difficulty: Difficult. This could easily take a couple of months.

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

Hi,

Is there any way not to hardcode baseUrl in the setup file? As far as I can see, you have to specify baseUrl. I would like the request to continue with with the whatever url Tomcat is referencing to how can I achieve this?

Currently, if ERDDAP needs to send an email, it (in the current thread) connects to the mail server, sends the email, and disconnects. This is inefficient (too much time repeatedly connecting to the server) and slow (it slows down the current thread). Often (e.g., for subscriptions) multiple emails are sent out in a short period of time.

At the very least, it would be good to have an email queue and dedicate a separate thread to sending emails. There is already a TaskThread, which manages some background tasks (e.g., downloading remote files). There is code to ensure it isn't stalled, recreate it if needed, and stop it when ERDDAP is shutting down. The EmailThread could mimic its system for thread maintenance.

Beyond that, perhaps that thread should be smart about letting emails accumulate in the queue before sending them as a group. For example, perhaps the system should wait 5 (or 10 or ?) seconds after an email is added to the queue in case other emails are also added during that time. Thus, emails would still go out quickly but would be more likely to be sent in a batch rather than individually.

Note that all emails get sent via EDStatic.email(), so it will be easy to modify that to add an email queue.
And the TaskThread has thread management code, so you can just mimic that.

Skills required: Java programming. Since this involves threads, extra care must be taken to avoid problems.

Difficulty: Medium. You'll have to explore JavaMail's features to figure out how to optimize this. But I think it is a straightforward task. Perhaps 1-2 weeks (after you figure out ERDDAP).

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

If you query ERDDAP using allDatasets, dataset ID is returned as datasetID.

If you query ERDDAP using advanced search , dataset ID is returned as Dataset ID.

Is there a reason for this?

It's not a big deal, but it messed me up for a bit when writing my little ERDDAP time series explorer.

To pull the lon/lat values from the allDatasets table that matched the datasets returned from advanced search, I needed to do:

dfr = dfll[dfll['datasetID'].isin(dfa['Dataset ID'])]

https://github.com/reproducible-notebooks/ERDDAP_timeseries_explorer/blob/master/ERDDAP_timeseries_explorer.py#L214

Chrome browser gives an error for the mime type returned for a jsonp request.

Refused to execute script from 'https://erddap.marine.ie/erddap/tabledap/IWBNetwork.json?longitude,latitude,time,station_id&time%3E=2018-01-08T09:53:33.009Z&station_id=%22M4%22&.jsonp=jQuery20308886091928996869_1515413372716' because its MIME type ('application/json') is not executable, and strict MIME type checking is enabled.

(MIME type should be 'application/javascript' if the json is requested and .jsonp flag is set)

https://stackoverflow.com/questions/477816/what-is-the-correct-json-content-type

I'm trying to use EDDTableFromNcCFFiles to read this featureType=timeSeriesProfile data (ADCP data):
http://gamone.whoi.edu/thredds/catalog/sand/usgs/users/rsignell/data/cmg/stellwagen/CF-1.6/WFAL/catalog.html?dataset=sand/usgs/users/rsignell/data/cmg/stellwagen/CF-1.6/WFAL/8591wh-a.nc

(The direct link to download as netcdf file)

GenerateDatasetsXml.sh has no problem generating an XML snippet from this file, but DasDds.sh complains:

For cdm_data_type=TimeSeriesProfile, one variable must have cf_role=profile_id.

This is only true for timeSeriesProfile using ragged arrays.

For timeSeriesProfile datasets using multidimensional arrays, we don't need cf_role=profile_id:
http://cfconventions.org/cf-conventions/cf-conventions.html#_multidimensional_array_representations_of_time_series_profiles

@BobSimons , do you agree?

As discussed on the awesome-erddap repo - the canonical list is now here

https://github.com/IrishMarineInstitute/awesome-erddap/blob/master/erddaps.json

If you need a different format, let us know and we can build it into the issue opened on awesome-erddap.

erddap code downloaded successfully,but unzip error. help me .thank you!

This messed me up too for a while while working on the ERDDAP time series explorer.

I thought that Advanced Search, like tabledap data requests, could use time values like "now-7days" for time values like "Minimum Time" on pages like:
https://coastwatch.pfeg.noaa.gov/erddap/search/advanced.html
although I see that the pop up help does not display this as an option.

It's additionally confusing because you can type now-7days into the GUI for minimum time and do the search, and you get no errors.

But eventually you figure out that ERDDAP is just ignoring your specification there (so you might as well have typed "foobar"!) , treating this as no minimum time being specified.

It would be nice if ERDDAP complained that you typed an invalid time value.

It would be nicer if ERDDAP supported "now-7days" style time values in advanced search!

Description: ERDDAP lets users download the original source data files via a virtual file system, e.g.,
https://coastwatch.pfeg.noaa.gov/erddap/files/
It would be nice if there were an easy and efficient way for users to maintain a local copy of a dataset's files.
Note that some datasets simply gain new files periodically (although for this dataset,
https://coastwatch.pfeg.noaa.gov/erddap/files/erdATastdhday/
last years files are gzipped every year).
But for this dataset
https://coastwatch.pfeg.noaa.gov/erddap/files/cwwcNDBCMet/
the historical files are changed periodically (monthly), while other nrt (near real time) files are updated frequently (every 5 minutes).
For security reasons, we can't just allow users to log into our system and use rsync.
Instead, we need an rsync-like system. We're not going to write a new version of rsync. Hopefully, one of the rsync variants
https://en.wikipedia.org/wiki/Rsync#Variations
can be adapted for this purpose.
Ideally, users can use existing client software on their end to contact ERDDAP, find, and download the changes to a given dataset.
Obviously, the more efficient this is (processing time on the server and volume of data transferred), the better.
Security is vital. We must be able to assure the IT security people that there is absolutely no risk that a malicious user will be able to download other files from the server.

Skills required: Java programming. ERDDAP is written in Java, so at least some of the code will be in Java.

Difficulty: If you can find a suitable rsync variant (if one exists), then this is an easy (1-3 weeks of work after getting rolling with ERDDAP) or medium difficulty (4-8 weeks work) task. If no such rsync variant exists, then the task should not be attempted -- we're not going to re-write rsync or port it to Java. So the first part of the project is research and planning.

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

Investigate using Docker for ERDDAP distributions.
In particular, this may make it much easier to bundle other software with ERDDAP, e.g., ncWMS, or MapServer.

CC: @ehazen, @7yl4r,

Hi @BobSimons,

Thank you, thank you for a great and useful product of ERDDAP focused on the many ways to slice, dice and represent data for a plethora of analysis and visualization.

One of the more useful and easy representations of data is as a Web Map Service (WMS) to easily drop into a "slippy" online map using JavaScript libraries like leaflet and all the pretty basemap's like Esri.OceanBasemap and others from Leaflet Provider Demo:

However, the ERDDAP WMS currently only serves geographic projection EPSG:4326 and not the web Mercator EPSG:3857 needed to use these great basemaps. Heres' some futzing I did trying various combinations to make this work:

WMS Testing for Fixing Explorer Environmental Map

Could this be a matter of simply adding or updating instances in these Java files where 4326 shows up?

• EDDTableFromMultidimNcFiles.java

• EDDTableFromNcCFFiles.java

Thanks very much, Ben

Make a customizable web page that displays nColumns by nRows of boxes, each box gets a different ERDDAP map (or graph?), each of which can refer to some common values (e.g., latitude min/max, longitude min/max, time value) so that when any of the common values are changed, the maps and graphs are redrawn.
For example, imagine 6 (2 x 3) maps with sea surface temperature, chlorophyll, salinity, wind vectors, atmospheric pressure, current vectors.
Or image a map of an area with sea surface temperature, and a graph with sea surface temperature from 1 point on the map (you click on the map and the graph changes) [okay, that one would be super hard on ERDDAP because of the nature of how gridded data is usually stored].

The basic idea (if just maps with limited settings) is pretty simple. There is certainly lots of potential for making this fancy with lots of features:
Perhaps each graph has a dropdown box for datasetID and for variableName (the options change when datasetID changes).
Perhaps each box can be toggled between the map (or graph) and a form with settings specific to that map/graph (including all the colorBar settings).
Here's your chance to be creative and show off your JavaScript skills (while still minimizing the load on ERDDAP).

This ideal is from a poster (a finding request?) at an ESIP meeting by Matt Tisdale (NASA Langley) entitled "A Personal Hyperwall". He said I could use the idea. When you get a rough draft of this web page working, perhaps you could ask people (including Matt) for comments/suggestions/requests.

Skills required: 10% Java. 90% JavaScript.

Difficulty: Easy to hard. It's what you make of it and whether the techniques you use are simple or fancy. This could be done in 1 day (if it is a really simple setup) or a month (if it is fancy and you get and incorporate feedback from the community).

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

Currently there's a mismatch between the erddapContentDirectory environment variable checked and the name in the error message (erddapContent). Using a string constant for the name in both places would prevent inconsistencies.

https://github.com/BobSimons/erddap/blob/master/WEB-INF/classes/gov/noaa/pfel/erddap/util/EDStatic.java#L1095

To fulfill a user request for data, EDDGridFromFiles and EDDTableFromFiles both often get data from multiple data files. Both now have a system (different for each) for using multiple threads to speed up this process. See
https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#nThreads

Unfortunately, my ancient, wimpy PC is a poor computer to test the effectiveness of these systems. So the task here is to run the existing tests on a more powerful computer (eg 8+ cores and perhaps a RAID or SSD) and if possible to further optimize these systems.
For EDDTableFromFiles in particular (which can open and separately process 1000's of files to fulfill a request), it is clear that other software (eg gzip) can open and read multiple files much faster than ERDDAP can. So it seems like there is a lot of room for further optimization.

Constraint on multithreading in ERDDAP:

• The order of the results must be the same as the order without multi-threading. So the subtasks aren't completely independent (as with many multi-threading tasks, e.g., gzip).
• For EDDGridFromFiles, the amount of memory needed to process a given file may be huge. So the system needs to be more flexible and figure out how many threads to use based on the amount of data gotten from each file.

How to approach the problem: It's probably best to take on EDDTableFromFiles first. When you are done with that, look at EDDGridFromFile to see if the same techniques can be used.

Skills required: Java programming.

Difficulty: Medium. This project will involve relatively more investigation and testing, and relatively less coding, but the coding will be somewhat difficult because it involves multi-threading. The duration of this task depends a lot on how deep you want to go and/or how long it takes you to find and fix problems. So maybe this is 1-2 weeks if you are already very experienced with this type of work or 1-2 months if you are new to it and go deep into it.

Mentor: Bob Simons (main author of ERDDAP), but the current system is ERDDAP is the extent of my knowledge and expertise. I can help you understand ERDDAP and the task at hand, but you'll have to figure out how to solve the problem.

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

There is already an EDDGridLonPM180 which converts any child EDDGrid dataset with longitude values which are (something like) 0 to 360 into a dataset with longitude values from (something like) -180 to 180. For this project, you would do the opposite: convert from -180...180 to 0...360. See
https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#EDDGridLonPM180 and
https://github.com/BobSimons/erddap/blob/master/WEB-INF/classes/gov/noaa/pfel/erddap/dataset/EDDGridLonPM180.java

Skills required: Java programming.

Difficulty: The basic problem is a little tricky because there are lots of unusual cases where the source dataset has various longitude ranges and you have to do the best you can for all the cases. But the task is much easier because EDDGridLonPM180 exists so you can look at the code and do mostly the same things in the comparable situations. (So it may be just search and replace the different lon values.) Perhaps 2 weeks work (after you figure out ERDDAP).

Bonus benefit: several CoastWatch people will be very grateful if you do this.

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

Hi! Bobs,

First I appreciate your efforts towards the development of ERDDAP, wonderful Rest based, dataserver with awesome features, I have one small query, as I am new to ERDDAP, would like to make sure from you, is there any way to allow user to change X axis min and max range like Y axis, for example as you can see here, user can change Y axis min and max, but can't change interval between range, and can't change X axis range and interval, I don't know whether feature exists or not, Please clarify.

https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplAmsreSstMon.graph

Thank you, Have a nice day.

ERDDAP's graphing capabilities are simple, plain, and functional (so say I) or stodgy (so say lots of others).
ERDDAP current graphing capabilities were designed so that

• The load on the server was minimal
• Each graph can be requested with one simple RESTful URL.
• The system could be made with the technology and a library available in 2007 (very limited).

ERDDAP's current graphics system isn't going away both for continuity and because this approach meets certain basic needs,
but there is clearly room for adding a series of web pages with specialized, fancy (e.g., animated) graphics,
A good example is the animated vectors that Adam Leadbetter is (and Rob Fuller was before he retired -- we miss you Rob!) working on: https://irishmarineinstitute.github.io/erddap-js/demo/02-leaflet-time-dimension-velocity/
The discussion about that particular animation is at https://groups.google.com/g/erddap/c/WWw042tLPTs
I think Adam is still working to make this easier to reuse.

Your goal here would be make web pages for common types of animations. I don't know what's possible. You'll have to look around, figure out what is possible, and then make it work.
We'll also have to find a way to point to these new web pages from ERDDAP.
The system you make can be tightly or loosely connected to ERDDAP. Do what is best / makes sense.
It is import that the animations be able to get data from ERDDAP's services directly.
It is important that you keep the load on ERDDAP minimal. We don't want a system where one user can bog down ERDDAP (cough, like WMS, cough).

Skills required: Java and JavaScript programming. The ability to work independently.

Difficulty: Probably mostly very challenging. And I think this will suck up as much time as you care to put in it: 1 month (absolute minimum) to 3 months (more realistic).

Mentor: Bob Simons (main author of ERDDAP), but this is beyond my expertise. I'm only good with simple JavaScript. I haven't worked with any of the animation libraries. I can help with hooking things into ERDDAP and I (and others, e.g., the ERDDAP Google Group) can comment on the graphics while they are works in progress, but that's about it. You'll have to work pretty independently.

Bonus: lots of ERDDAP users will be very, very grateful to you for adding more interesting graphics to ERDDAP.

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

When a user queries for image data in ERDDAP, the file descriptor is left open indefinitely until max limit is reached. File is deleted, but handler is left open.

Hiya all,
not sure if this is my misuse of a feature or a bug?

Issue

The address https://linkedsystems.uk/erddap/tabledap/ea_sea_level_e72224_e229_390e_c509.htmlTable?&time=max(time) resulted in a error of;

{
    code=404;
    message="Not Found: Your query produced no matching results. (No data matches time=\"1.6341291E9\" because the String variable's source min=\"2020-10-30T13:15:00Z\", max=\"2021-10-13T12:30:00Z\", and hasNaN=false.)";
}

https://linkedsystems.uk/erddap/tabledap/ea_sea_level_e72224_e229_390e_c509.htmlTable?&time=max(time)+0minutes gave the same error.

interestingly the address https://linkedsystems.uk/erddap/tabledap/ea_sea_level_e72224_e229_390e_c509.htmlTable?&time=max(time)+1minutes shows an error with formatted time, and states that the requested time is outside of the range for this datatset.

{
    code=404;
    message="Not Found: Your query produced no matching results. (time=2021-10-13T12:46:00Z is outside of the variable's actual_range: 2020-10-30T13:15:00Z to 2021-10-13T12:45:00Z)";
}

The following address https://linkedsystems.uk/erddap/tabledap/ea_sea_level_e72224_e229_390e_c509.htmlTable?&time%3Cmax(time) shows all data.

and if i make time a variable like so https://linkedsystems.uk/erddap/tabledap/ea_sea_level_e72224_e229_390e_c509.htmlTable?time&time=max(time) the page worked.

Issue found

After some more digging it turns out that the erddap metadata was very slightly out of date with what was on disk. i.e the max time value was in fact 2021-10-13T12:45:00Z and as can be seen from the error reported the erddap max range was 2021-10-13T12:30:00Z. Resulting in the URL erroring out.

questions

• Why isn't the time formatted in the first error message?
• Should a request containing a + or > on a max variable be a 400 error?
• I am not entirely sure why naming the variable (in this case time) changes the behaviour between requests?

ERDDAP lets users specify the output file format. One of them is .ncoJson, which writes everything that would be in a .nc file (metadata and data) into a JSON format instead. It would be nice if there were an .ncoJsonHeader option which just contained the metadata, but no data. Many users just want the metadata, and many users want JSON, so this is a desirable feature.

Skills required: Java programming

Difficulty: easy. Perhaps 1 week. The existing code for writing .ncoJson already writes the metadata. You just have to either:

• copy that code and remove the part that writes the data
• separate out that code so both the .ncoJson and .ncoJsonHeader writers can use the same code.

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

It seems that some of the ERDDAP output formats are converting doubles (float64) to floats (float32) and running into floating point representation errors (.asc, .csv, .geoJSON, .htmlTable, .kml) while others (.nc, .ncCF) are retaining the original double type through to the output.

This seems like something that someone would have run into before! Is this a known issue?

datasets.xml
source netcdf file

Add new server side function like: &wkt("a string with the WKT polygon") ERDDAP could handle this by converting the polygon into lat and lon constraints, getting that data subset, then properly subsetting to the polygon, then return the results to the user. Thanks to Matthew Biddle.

Question: In the DMAC meeting today, I addressed issues that are related to the Trajectory Profile dataset. One of the questions is the IOOS compliance checker - NCEI Trajectory Profile Orthogonal 2.0 is asking the Trajectory Profile netCDF file has set "cdm_data_type" = "Trajectory" and then set "featureType" = "trajectoryProfile," while in ERDDAP, it is saying "cdm_data_type" = "TrajectoryProfile". Another question is if I use the standard NCEI Trajectory Profile example netCDF file, and feed the file into the ERDDAP directly, with slight modification (set global attribute cdm_trajectory_variables = trajectory), the ERDDAP will provide the error to me,

*** An error occurred while trying to load NCEI_R4_23e3_15d9_6c01: java.lang.RuntimeException: datasets.xml error on or before line #286: cdm_data_type=Trajectory, but the dataset doesn't have a longitude variable. at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:436) at gov.noaa.pfel.erddap.dataset.EDD.oneFromDatasetsXml(EDD.java:517) at gov.noaa.pfel.erddap.dataset.EDD.testDasDds(EDD.java:11068) at gov.noaa.pfel.erddap.DasDds.doIt(DasDds.java:129) at gov.noaa.pfel.erddap.DasDds.main(DasDds.java:155) Caused by: com.cohort.util.SimpleException: cdm_data_type=Trajectory, but the dataset doesn't have a longitude variable. at gov.noaa.pfel.erddap.dataset.EDDTable.accessibleViaNcCF(EDDTable.java:13105) at gov.noaa.pfel.erddap.dataset.EDD.ensureValid(EDD.java:770) at gov.noaa.pfel.erddap.dataset.EDDTable.ensureValid(EDDTable.java:612) at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.<init>(EDDTableFromFiles.java:1839) at gov.noaa.pfel.erddap.dataset.EDDTableFromNcFiles.<init>(EDDTableFromNcFiles.java:128) at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.fromXml(EDDTableFromFiles.java:467) at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:422) ... 4 more

However, if I setup the NCEI Trajectory Profile netCDF example by

`wget -qO test.nc https://data.nodc.noaa.gov/thredds/fileServer/example/v2.0/NCEI_trajectoryProfile_template_v2.0_2016-09-22_181838.014029.nc

ncatted -hO -a cdm_data_type,global,m,c,"TrajectoryProfile" test.nc
ncatted -hO -a cdm_trajectory_variables,global,a,c,"trajectory" test.nc
ncatted -hO -a cdm_profile_variables,global,a,c,"time,latitude,longitude" test.nc`

The ERDDAP server will give me the error message like below...

*** An error occurred while trying to load NCEI_R4_23e3_15d9_6c01: java.lang.RuntimeException: datasets.xml error on or before line #287: For cdm_data_type=TrajectoryProfile, one variable must have cf_role=profile_id. at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:436) at gov.noaa.pfel.erddap.dataset.EDD.oneFromDatasetsXml(EDD.java:517) at gov.noaa.pfel.erddap.dataset.EDD.testDasDds(EDD.java:11068) at gov.noaa.pfel.erddap.DasDds.doIt(DasDds.java:129) at gov.noaa.pfel.erddap.DasDds.main(DasDds.java:155) Caused by: com.cohort.util.SimpleException: For cdm_data_type=TrajectoryProfile, one variable must have cf_role=profile_id. at gov.noaa.pfel.erddap.dataset.EDDTable.accessibleViaNcCF(EDDTable.java:12954) at gov.noaa.pfel.erddap.dataset.EDD.ensureValid(EDD.java:770) at gov.noaa.pfel.erddap.dataset.EDDTable.ensureValid(EDDTable.java:612) at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.<init>(EDDTableFromFiles.java:1839) at gov.noaa.pfel.erddap.dataset.EDDTableFromNcFiles.<init>(EDDTableFromNcFiles.java:128) at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.fromXml(EDDTableFromFiles.java:467) at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:422) ... 4 more

This is because the profile_id is located on the variable "time", but the ERDDAP didn't load time variable

`grep sourceName datasets.xml

    <sourceName>trajectory</sourceName>
    <sourceName>obs</sourceName>
    <sourceName>z</sourceName>
    <sourceName>sal</sourceName>
    <sourceName>temp</sourceName>
    <sourceName>crs</sourceName>`

Hoping someone here can help me figure this out.

Add a new output file type to ERDDAP: Coverage JSON (.covjson)
See https://covjson.org/

Talk with the covjson people to see if the project is still active and if they think it is worth the effort.

Skills required: Java programming.

Difficulty: Relatively easy. ERDDAP datasets have a cdm_data_type (featureType) attribute which aligns with the Coverage JSON types. So you just need to write the code to deal with each of the cdm_data_types. Perhaps 1 month's work (after you figure out ERDDAP).

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide
especially the "Judging Your Code Contributions" section.

(Rich Signell requested this feature.)

The change in v1.82 to use String2.directReadFrom88591File() instead of creating a String from a ByteArray in the testing code for datasets introduced many compilation errors.

For example, line 516 in EDDTableFromPostDatabase.java (erddap/WEB-INF/classes/gov/noaa/pfel/erddap/dataset/EDDTableFromPostDatabase.java) changed:

before:
results = new String((new ByteArray(EDStatic.fullTestCacheDirectory + tName)).toArray());
after:
results = String2.directReadFrom88591File(EDStatic.fullTestCacheDirectory + tName)).toArray());

The ").toArray())" appears to be a leftover from a search-and-replace to make this change. It's almost certainly fixed in your local environment already...