Git Product home page Git Product logo

knmi / adaguc-server Goto Github PK

View Code? Open in Web Editor NEW
26.0 26.0 27.0 105.61 MB

ADAGUC is a geographical information system to visualize netCDF files via the web. The software consists of a server side C++ application and a client side JavaScript application. The software provides several features to access and visualize data over the web, it uses OGC standards for data dissemination.

License: Apache License 2.0

C++ 83.26% C 1.36% Makefile 0.05% Shell 0.62% Python 13.21% HTML 0.38% Dockerfile 0.12% CMake 0.74% JavaScript 0.27% CSS 0.01%

adaguc-server's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

adaguc-server's Issues

Support reading metdata from PNG imagery for correct geographical projection

Support reading following metadata from PNG and put it in the datasource:

Properties:
bbox: -5570248.7,5570248.7,5567248.3,-5567248.3
date:create: 2021-09-16T16:39:34+02:00
date:modify: 2021-09-16T16:39:31+02:00
png:IHDR.bit-depth-orig: 8
png:IHDR.bit_depth: 8
png:IHDR.color-type-orig: 6
png:IHDR.color_type: 6 (RGBA)
png:IHDR.interlace_method: 0 (Not interlaced)
png:IHDR.width,height: 3712, 3712
png:sRGB: intent=0 (Perceptual Intent)
png:text: 3 tEXt/zTXt/iTXt chunks were found
proj4_params: +proj=geos +a=6378.169 +b=6356.584 +h=35785.831 +lat_0=0 +lon_0=0.0
signature: 243b3790eabda957d49dd53573c9909da21023dda609f446df956e4ee0df1986
time: 2021-09-16T01:00:00Z

Adaguc-server configuration: Validate allowed characters in style and legend names during scanning

Style and legend names are identifiers used in urls. We have seen style names in configurations with invalid tokens.

Validate stylenames, e.g. The stylename should not contain escapable characters like <, \ , /, >, (, ), etc.

The following style should be considered invalid:

  <Style name="dBZ>0(3D)">
    <Legend fixedclasses="true" tickinterval="10">dBZ>0(3D)</Legend>
    <Min>0</Min>
    <Max>80</Max>
    <ValueRange min="-50" max="80" />
    <RenderMethod>nearest,bilinear</RenderMethod>
    <NameMapping name="nearest" title="dBZ>0, nearest" abstract="Nearest neighbour interpolation"/>
    <NameMapping name="bilinear" title="dBZ>0, bilinear" abstract="Bilinear interpolation"/>
  </Style>

Instead it should be:

  <Style name="dBZ_more_than_0_3D">
    <Legend fixedclasses="true" tickinterval="10">dBZ_more_than_0_3D</Legend>
    <Min>0</Min>
    <Max>80</Max>
    <ValueRange min="-50" max="80" />
    <RenderMethod>nearest,bilinear</RenderMethod>
    <NameMapping name="nearest" title="dBZ>0, nearest" abstract="Nearest neighbour interpolation"/>
    <NameMapping name="bilinear" title="dBZ>0, bilinear" abstract="Bilinear interpolation"/>
  </Style>

Create a wiki and pull the wiki from https://github.com/saskiawagenaar/adaguc-server.wiki.git

On my fork I have created a wiki in which I describe how to build and deploy the adaguc-server gitbook.

In order to obtain this wiki in the original adaguc-server, the following steps should be followed:

  1. Create a wiki in the adaguc-server KNMI repo
  2. Clone the adaguc-server wiki repo and merge the repo from the fork:
$ git clone https://github.com/KNMI/adaguc-server.wiki.git
$ cd adaguc-server.wiki.git
$ git pull https://github.com/saskiawagenaar/adaguc-server.wiki.git master
$ git push origin master

Trim dimension values from spaces and newlines

The following wil fail:

<Dimension name="time" units="ISO8601" interval="PT5M" quantizeperiod="PT5M" quantizemethod="low" default="max">
   time
</Dimension>

Because time is on a newline with spaces in front.

Adaguc-server doesn't like our netCDF files

Hi,

I work for the Spanish Weather Service (AEMET).

We've used the docker compose file to create an Adaguc server. We haven't touched any configuration.

We have created netCDF files with cdo from grib1 and grib2 files and placed them in the adaguc-autowms folder. When trying to visualize them, we get this message:
imagen

This information doesn't tell us what is the problem and I haven't found any log message to find out what is preventing us from visualizing the files.

We use the Lambert Conformarl Conic projection. Maybe this is relevant

Some info about the files used:
imagen
imagen

We need some way to find out what is preventing us from visualizing the files. Maybe is something about the headers in the netCDF files or the projection. Any hint will be welcome.

Cheers,

David Gil
AEMET

Contour rendering has more labels and fewer lines than in the past

I recently upgraded to the most recent version of Adaguc Server from the version in late 2019. One of the differences I noticed was with the rendering of contours. Previously it appears as though labels were being more evenly spaced across the whole domain. Now it appears as though they are being spaced evenly along a single contour line, but there is crowding in dense areas. It also appears as though there are fewer contour lines.

I've attached the files that are the result of the same backing data and configuration. I did play with the scalecontours setting but that did not have my desired effect of thinning out the labels but leaving the lines.

new
old

Please let me know if there is an existing feature or config that I can use to keep the behavior the same.

If there isn't, I'd be happy to contribute to the project by adding that functionality.

Plus sign in file name gives invalid path

Hi!,

Our netCDF files have these file names:

  • mbr003_fc2023071312+003_UV10m_grb2_proj4nco.nc
  • mbr003_fc2023071312+003_RH2m_grb2_proj4nco.nc
  • mbr003_fc2023071312+003_T2m_grb2_proj4nco.nc

When trying to visualize these files placed in adaguc-autowms, ADAGUC gives this message in the browser:
imagen

The text in the URL has the %2B correctly decoded (it is the '+' sign).

Unable to get WMS GetCapabilities[http://172.24.139.98/adagucserver?source=03_GRIB2_2netCDFproj4nco%2Fmbr003_fc2023071312%2B003_RH2m_grb2_proj4nco.nc&]

On the other hand, the logs give this text:

applicationlog 2023-07-26 10:28:23,151 - routers.wmswcs - INFO - [D:000:pid723: adagucserverEC/CServerParams.cpp:388 CServerParams]              2023-07-26T10:28:23Z LOCALFILEACCESS: Invalid path '/data/adaguc-autowms//03_GRIB2_2netCDFproj4nco/mbr003_fc2023071312 003_RH2m_grb2_proj4nco.nc'
[D:001:pid723: adagucserverEC/CServerParams.cpp:388 CServerParams]              2023-07-26T10:28:23Z LOCALFILEACCESS: Invalid path '/data/adaguc-data//03_GRIB2_2netCDFproj4nco/mbr003_fc2023071312 003_RH2m_grb2_proj4nco.nc'
[D:002:pid723: adagucserverEC/CAutoResource.cpp:213 CAutoResource]              2023-07-26T10:28:23Z Unable to resolve path for autoresource file 03_GRIB2_2netCDFproj4nco/mbr003_fc2023071312 003_RH2m_grb2_proj4nco.nc
[D:003:pid723: adagucserverEC/CAutoResource.cpp:214 CAutoResource]              2023-07-26T10:28:23Z Please note that only absolute paths without symbolic links can be used
[E:004:pid723: adagucserverEC/CRequest.cpp:2961 CRequest]                       2023-07-26T10:28:23Z AutoResource failed
applicationlog 2023-07-26 10:28:23,152 - routers.wmswcs - INFO - Adaguc status code was 1
172.20.0.4:50300 - "GET /wms?source=03_GRIB2_2netCDFproj4nco%2Fmbr003_fc2023071312%2B003_RH2m_grb2_proj4nco.nc&&service=WMS&request=GetCapabilities&random0.12704694024335694 HTTP/1.0" 500

The '+' has been replaced somewhere by a space, which gives an invalid path.

When the '+' is changed in the file name to another character, the problem disappears.

The double slash between adaguc-autowms and the subfolder is also somewhat suspiciuos, although it probably won't have any consequence.

Cheers,

David Gil

Move configuration requiring root out of Docker/start.sh into Dockerfile

This is a blocker for #59, because start.sh does lots of things that require root.

In general it's better to finish all the setup you can within the Dockerfile, so it doesn't have to be done every time the container starts. This saves startup time for the container, which can be significant because of the different lifecycles for containers over for example VMs.

Adaguc-server: Do not include files older then retention period during scanning

The retentionperiod will automatically delete files which older then the given period. It seems that this only works for files which already have been scanned and ingested in the database.

=> It would be better to not to add files which are too old to the database. During scanning and reading of the time dimension, make a decision whether to scan or to delete the file right away.

Pacific-centric Robinson projection support

Hi,

I would like to plot the following NetCDF file sample_tas_cmip6_ssp585_preIndustrial_warming2_year.nc.zip using a Pacific-centric Robinson projection.

By using the standard Robinson projection defined as <Projection id="EPSG:54030" proj4="+proj=robin +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs "/> ADAGUC generetes a perfect image: robinson_standard_ok

However, if I try with the Pacific-centric version defined as <Projection id="EPSG:54030pc" proj4="+proj=robin +lon_0=-150 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs "/> the resulting image has some problems for positive longitudes: robinson_offset_fail

As far as I know, there is not an EPSG code for the Pacific-centric Robinson projection. So, following some examples, I've just included an offset in the +lon_0 proj parameter and changed the projection name. I've tried this same proj definition with some GeoJSON files and it works well. So, I would say the proj definition is correct.

I've also tried with the +over proj parameter: in this case ADAGUC properly handles positive longitudes but the map is not centered in the Pacific: robinson_over_fail

Please, could you tell me if ADAGUC supports offsets in this +lon_0 parameter?

Thank you for sharing this great tool with the climate software community!

Features disappearing on certain BBOX

I'm trying to plot a very simple rectangular feature in two tiles, however, it only gets printed on one of the two tiles.

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "MultiPolygon",
        "coordinates": [[[
           [-9.5, 41.5],[-1, 41.5],[-1, 44],[-9.5, 44],[-9.5, 41.5]
        ]]]
      },
      "id": "cantabrican"
    }
  ]
}

image

The "right" tile gets correctly printed with either mercator or CRS84 bbox

Bbox coords

Mercator WGS84
-626172.1357121639 -5°37'30"
5009377.085697314 40°58'47.633"
0 0
5635549.221409476 45°5'20.528"

image

However the "left" part of the feature is not showing up either with mercator or CRS84 bbox:

Mercator WGS84
-1252344.2714243277 -11°15'0"
5009377.085697314 40°58'47.633"
-626172.1357121639 -5°37'30"
5635549.221409476 45°5'20.528"

image

This is how I've configured the layer and relevant style:

<Configuration>
  <Layer type="database">
    <Name>domain-mask</Name>
    <FilePath>/tmp/domains/cantabrian-14679283950060533581.json</FilePath>
    <Variable>features</Variable>
    <Styles>mask-cantabrian</Styles>
  </Layer>
</Configuration>
<Configuration>
  <Style name="mask-cantabrian">
    <Legend fixed="true">baselayer</Legend>
    <RenderMethod>nearest</RenderMethod>
    <NameMapping name="nearest" title="mask" />
    <FeatureInterval match=".*" matchid="id" bgcolor="#00000000" fillcolor="#FFFFFFFF" label="Other" />
  </Style>
</Configuration>

I've tried comparing the logs in the two cases but I don't find anything relevant. Any idea what might be going on?

Many thanks in advance

Support NetCDF files with irregular grids

We are trying to use files (NetCDF in attached) with irregular grids from WRF model.
The attached NetCDF file does not seem to work properly with adaguc-server. It has a irregular coordinate variable, with cellsizes increasing in the north-south direction.
Is it possible to support NetCDF files with irregular grids.
Thanks in advance
wrf_grid_latlon.zip

Since adaguc-server 2.10.0: using autowms gridded interpolated observations fail

Adaguc-server is unable to use GSIE gridded interpolated observations since 2.10.0:

D:002:pid20: adagucserverEC/CAutoResource.cpp:232 CAutoResource] 2023-08-21T07:09:33Z OGC REQUEST Remote resource /data/adaguc-autowms/INTER_OPER_R___RD1NRT__L3__20230818T080000_20230819T080000_0002.nc
[D:003:pid20: adagucserverEC/CRequest.cpp:752 CRequest] 2023-08-21T07:09:33Z WMS GETCAPABILITIES
[E:004:pid20: adagucserverEC/CImageWarper.cpp:334 CImageWarper] 2023-08-21T07:09:33Z Invalid projection: from " +proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +units=m +no_defs +<>"

The attached NetCDF file INTER_OPER_R___RD1NRT__L3__20230818T080000_20230819T080000_0002.nc does not work with autowms.

INTER_OPER_R___RD1NRT__L3__20230818T080000_20230819T080000_0002.zip

WMS legenda toont alleen streep en is erg hoog

Make keep-alive timeout of Adaguc-server configurable

When deploying Adaguc-Server behind an AWS ELB Appliction LoadBalancer the default keep-alive timeout of gunicorn is too short. This leads to 502 Bad Gateway errors (docs). The solution is to increase the Gunicorn keep-alive timeout to a value longer than the ELB keep-alive timeout.

The default timeout of gunicorn is 2 seconds (docs). The default keep alive of the AWS ELB is 60 seconds (docs)

What would be best? Adding an environment variable to the Docker startup script starting gunicorn?

https://github.com/KNMI/adaguc-server/blob/master/Docker/start.sh#L39

Concatenated GRIB files converted to NetCDF and served over OpenDAP Server fails to visualise

Hi,

I would like some help with a problem I am having with ADAGUC Server.

I am unable to visualise temperature data from a multi temporal NetCDF file that is served over a THREDDS Data Server v4.6.13 (TDS) via the OpenDAP protocol. The NetCDF file is generated from two separate GRIB version 1 files that originated from the ECMWF. They are:

D1D12191200122309001
D1D12191200122312001

They contain data from the same model run date and time (19/12/2018 12:00) but for two different forecast times (23/12/2018 9am and 12pm). If I convert any one of them to a NetCDF file using the eccodes grib_to_netcdf terminal command, and serve them individually over TDS. They will visualise as expected.

However if I concatenate them using the following terminal command:

cat D1D12191200122309001 D1D12191200122312001 > ECMWF_HRES_3HR_AU_VIC_20181219_1200.grib1

Then convert the resulting file to NetCDF before serving it over TDS. The result will not visualise in the ADAGUC Viewer.

If I open the same OpenDAP link in Panoply it will visualise the two time dimensions with no issues. If I take the problematic NetCDF file and place it in the ADAGUC data directory and serve it as a file rather than an OpenDAP link it will visualise the two time dimensions with no issues. The problem is only when this multi time dimension file is served over OpenDAP, in that case ADAGUC will not visualise any of the time slices.

Can someone help me with this?

I am using the latest openearth/adaguc-server docker hub image. A GetCapabilities request returns a version of 2.0.37.

My TDS catalog featureCollection configuration:


<featureCollection name="HRES 3hr Australia VIC" featureType="FMRC" path="NetCDF/ECMWF/HRES/3HR/AU/VIC">
  <metadata inherited="true">
    <serviceName>all</serviceName>
  </metadata>

  <collection spec="/usr/local/tomcat/content/thredds/public/ecmwf_nc/ECMWF_HRES_3HR_AU_VIC_#yyyyMMdd_HHmm#.nc$"
              name="ECMWF_NC_HRES_3HR_AU_VIC"
              timePartition="file"
              olderThan="5" />

</featureCollection>

My ADAGUC configuration file:

<?xml version="1.0" encoding="UTF-8" ?>
<Configuration>

  <Layer type="database">
    <FilePath>http://thredds:8080/thredds/dodsC/NetCDF/ECMWF/HRES/3HR/AU/VIC/files/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc</FilePath>
    <Variable>t2m</Variable>
    <Dimension name="time" interval="PT3H" default="max">time</Dimension>
  </Layer>

</Configuration>

The GetCapabilities request has the correct layer 't2m' listed and the correct time range:

<Dimension name="time" units="ISO8601" default="2018-12-23T12:00:00Z" multipleValues="1" nearestValue="0" current="1">2018-12-23T09:00:00Z/2018-12-23T12:00:00Z/PT3H</Dimension>

The raw request looks like this:

http://localhost:8090//adaguc-services/adagucserver?DATASET=ecmwf_2&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&LAYERS=t2m&WIDTH=1035&HEIGHT=321&CRS=EPSG%3A4326&BBOX=-49.17696610818382,91.662023552629,-17.282936741172634,194.49791263317908&STYLES=auto%2Fnearest&FORMAT=image/png&TRANSPARENT=TRUE&time=2018-12-23T09%3A00%3A00Z&0.3807920303767882

It returns this error message:

<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<ServiceExceptionReport version="1.3.0"  xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
    <ServiceException code="OperationNotSupported">
    nc_get_vara (typeconversion): : NetCDF: Access failure;
    Problem with variable t2m of type CDF_DOUBLE (requested CDF_DOUBLE):;
    nc_get_var: : NetCDF: Access failure;
    Unable to read data for variable t2m;
    Unable to read data for variable t2m in file http://thredds:8080/thredds/dodsC/NetCDF/ECMWF/HRES/3HR/AU/VIC/files/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc;
    Could not open file: http://thredds:8080/thredds/dodsC/NetCDF/ECMWF/HRES/3HR/AU/VIC/files/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc;
    warpImage for layer t2m failed;
    Unable to convert datasource t2m to image;
    Returning from line 1934;
    WMS GetMap Request failed;

  </ServiceException>
</ServiceExceptionReport>

Output from running adaguc-server-updatedatasets.sh:

*** Starting update for /data/adaguc-datasets/ecmwf_2
[D: CRequest.cpp, 116 in CRequest]                                 Dataset name based on passed configfile is [ecmwf_2]
[D: CAutoResource.cpp, 81 in CAutoResource]                        Found dataset /data/adaguc-datasets//ecmwf_2.xml
[D: CDBFileScanner.cpp, 938 in CDBFileScanner]                     *** Starting update layer 't2m' ***
[D: CDBFileScanner.cpp, 940 in CDBFileScanner]                     Using path [http://thredds:8080/thredds/dodsC/NetCDF/ECMWF/HRES/3HR/AU/VIC/files/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc], filter [] and tailpath []
[D: CDBFileScanner.cpp, 117 in CDBFileScanner]                     Checking dim [time]
[D: CDataReader.cpp, 1298 in CDataReader]                          Warning no standard name given for dimension time, using variable name instead.
[D: CDataReader.cpp, 1298 in CDataReader]                          Warning no standard name given for dimension time, using variable name instead.
[D: CDBFileScanner.cpp, 366 in CDBFileScanner]                     Found dimension 0 with name time of type 2, istimedim: [1]
[D: CDBFileScanner.cpp, 388 in CDBFileScanner]                     Marking table done for dim 'time' with table 't20190326t044525769_6afjqayh5htc9xatr7oe'.
[D: CDBFileScanner.cpp, 395 in CDBFileScanner]                     Found 1 files
[D: CDBFileScanner.cpp, 831 in CDBFileScanner]                     The database contains 1 files
[D: CDBFileScanner.cpp, 841 in CDBFileScanner]                     Comparing lists
[D: CDBFileScanner.cpp, 844 in CDBFileScanner]                     Found 0 files in DB which are missing
[D: CDBFileScanner.cpp, 1014 in CDBFileScanner]                    *** Finished update layer 't2m' ***

[D: CRequest.cpp, 3446 in CRequest]                                ***** Finished DB Update *****
[D: CDirReader.cpp, 394 in CCachedDirReader]                       Cleaningup dirReaders

The log file entries relating to the failed request:

[D: CRequest.cpp, 817 in CRequest]                                 2019-03-26T08:39:54Z/385 WMS GETMAP (0) t2m
[D: CTime.cpp, 172 in CTime]                                       2019-03-26T08:39:54Z/385 Initializing CTime with seconds since 1970
[D: CRequest.cpp, 1469 in CRequest]                                2019-03-26T08:39:54Z/385 Using maxquerylimit 512
[D: CDataSource.cpp, 1234 in CDataSource]                          2019-03-26T08:39:54Z/385 _currentStyle == NULL
[D: CAutoConfigure.cpp, 495 in CAutoConfigure]                     2019-03-26T08:39:54Z/385 Loading header [http://thredds:8080/thredds/dodsC/NetCDF/ECMWF/HRES/3HR/AU/VIC/files/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc]
[D: CDataReader.cpp, 751 in CDataReader]                           2019-03-26T08:39:54Z/385 Working on [http://thredds:8080/thredds/dodsC/NetCDF/ECMWF/HRES/3HR/AU/VIC/files/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc]
[D: CDataReader.cpp, 573 in CDataReader]                           2019-03-26T08:39:54Z/385 Assuming that for variable t2m the x dim equals longitude and the y dim equals latitude based on their position and name.
[D: CDataReader.cpp, 599 in CDataReader]                           2019-03-26T08:39:54Z/385 Using variable longitude as X variable and variable latitude as Y variable.
[D: CDataReader.cpp, 626 in CDataReader]                           2019-03-26T08:39:54Z/385 No stride defined in the RenderSettings, using a default stride of 1.
[W: CDataReader.cpp, 179 in CDataReader]                           2019-03-26T08:39:54Z/385 No correct projection found, using by default the geographic coordinate system (latitude and longitude).
[D: CDataReader.cpp, 216 in CDataReader]                           2019-03-26T08:39:54Z/385 Using the geographic coordinate system (latitude and longitude)
[E: CCDFNetCDFIO.cpp, 255 in class NetCDFReader] nc_get_vara (typeconversion): : NetCDF: Access failure
2019-03-26T08:39:54Z/385 [E: CCDFNetCDFIO.cpp, 269 in NetCDFReader]                         2019-03-26T08:39:54Z/385 Problem with variable t2m of type CDF_DOUBLE (requested CDF_DOUBLE):
[E: CCDFNetCDFIO.cpp, 270 in class NetCDFReader] nc_get_var: : NetCDF: Access failure
2019-03-26T08:39:54Z/385 [E: CCDFVariable.cpp, 351 in Variable]                             2019-03-26T08:39:54Z/385 Unable to read data for variable t2m
[E: CDataReader.cpp, 1008 in CDataReader]                          2019-03-26T08:39:54Z/385 Unable to read data for variable t2m in file http://thredds:8080/thredds/dodsC/NetCDF/ECMWF/HRES/3HR/AU/VIC/files/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc
[D: CDataReader.cpp, 1011 in CDataReader]                          2019-03-26T08:39:54Z/385 time 1 1 1
[D: CDataReader.cpp, 1011 in CDataReader]                          2019-03-26T08:39:54Z/385 latitude 0 81 1
[D: CDataReader.cpp, 1011 in CDataReader]                          2019-03-26T08:39:54Z/385 longitude 0 181 1
[E: CImageDataWriter.cpp, 1459 in CImageDataWriter]                2019-03-26T08:39:54Z/385 Could not open file: http://thredds:8080/thredds/dodsC/NetCDF/ECMWF/HRES/3HR/AU/VIC/files/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc
[E: CImageDataWriter.cpp, 2011 in CImageDataWriter]                2019-03-26T08:39:54Z/385 warpImage for layer t2m failed
[E: CRequest.cpp, 1933 in CRequest]                                2019-03-26T08:39:54Z/385 Unable to convert datasource t2m to image
[E: CRequest.cpp, 2185 in CRequest]                                2019-03-26T08:39:54Z/385 Returning from line 1934
[E: CRequest.cpp, 3144 in CRequest]                                2019-03-26T08:39:54Z/385 WMS GetMap Request failed

This zip file contains three folders 'nc_file1', 'nc_file2', and 'nc_file1_and_file2'. The problematic NetCDF file is located 'nc_file1_and_file2/ECMWF_HRES_3HR_AU_VIC_20181219_1200.nc' and is the result of concatenating the two grib files and converting them to NetCDF. The file by the same name in nc_file1/ and nc_file2/ is just each of those GRIB files converted to NetCDF and included for the purpose of demonstrating that each of those files on their own work just fine.
adaguc_opendap_problem_files.zip

WMS HTTP Status codes that provide better feedback to the users.

Although the WMS is before REST standards, it would be nice to have similar HTTP codes for monitoring and as feedback to the users of the service.

Expected Behavior

When a user gives invalid input the server responds with a HTTP status code 4xx and a message.

Current Behavior

When a user gives invalid input the server responds with a clear message about what is wrong with the input, but a different status code than I would expect (200/500).
For example, when giving temperatured as input to the COVERAGE parameter. The status code is 500. And the message is Layer [temperatured] not found WCS GetCoverage Request failed. Other examples are in the table below.

Parameter Input Status code Expected
COVERAGE temperatured 500 404
dataset Tx1d 500 404
BBOX 3.0,50.0,7.0,99999999999999.0 500 422
Time 2022-sdfsd-22T00:00:00Z/2022-07-23T00:00:00Z 500 422
CRS EPSG:4326d 500 422
RESX/RESY 0.000004 500 422
RESX/RESY 0.d4 200 422
SERVICE Tx1d 200 422
REQUEST GetCoveraged 200 422

SVG Legend not generating expected gradient

I am using the docker version of the server. A getcapabilities request returns a version of 2.0.37.

If I create a Legend directly in the configuration xml file for my WMS Service it works fine and generates an expected gradient with the colours top to bottom of red, green, blue. Here is the code for that:

<Legend name="ColorPalette" type="colorRange">
    <palette index="0"   red="0"   green="0"   blue="255"/>
    <palette index="120"  red="0"   green="255" blue="0"/>
    <palette index="240"  red="255"   green="0" blue="0"/>
</Legend>

<Style name="temperature">
    <Legend>ColorPalette</Legend>
    <RenderMethod>nearest</RenderMethod>
</Style>

However if I attempt the same Legend using instead an SVG file I get a graphic that is quite wrong. It instead generates a gradient with the colours from top to bottom of green, red, blue, green. Here is the code for that:

ColorPalette.svg file:

<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" width="300px" height="45px" viewBox="0 0 300 45">
  <g>
    <defs>
      <linearGradient id="ColorPalette" gradientUnits="objectBoundingBox" spreadMethod="pad" x1="0%" x2="100%" y1="0%" y2="0%">
        <stop offset="0.00%" stop-color="rgb(0,0,255)" stop-opacity="1.0000"/>
        <stop offset="50.0%" stop-color="rgb(0,255,0)" stop-opacity="1.0000"/>
        <stop offset="100.0%" stop-color="rgb(255,0,0)" stop-opacity="1.0000"/>
      </linearGradient>
    </defs>
    <rect fill="url(#ColorPalette)" x="4" y="4" width="292" height="37" stroke="black" stroke-width="1"/>
  </g>
</svg>

XML config file:

<Legend name="ColorPalette" type="svg" file="/data/adaguc-datasets/palettes/ColorPalette.svg"/>

<Style name="temperature">
   <Legend>ColorPalette</Legend>
   <RenderMethod>nearest</RenderMethod>
</Style>

If you use the SVG for the bcyr palette from the soliton site you also get a mangled looking gradient.

Can anyone advise if this is user error or an error in the program?

Non SVG Legend (What I expect to see):
non-svg-legend

SVG Legend (Not what I expect to see):
svg-legend

bcyr palette SVG from soliton.vm.bytemark.co.uk (Also not what I expect to see):
bcyr-soliton-svg

Contour lines left off layer if there are too many

When adding lots of custom ContourLine's there seems to be a limit to how many the server will display. If I go over that limit some lines that should show up do not.

For example, here I am trying to represent the difference in height between two levels of pressure, the height of the 500mb pressure level minus height of 1000mb level (thickness). I am using the GFS forecast, specifically the 'hgtprs' variable. Here is my style:

<Style name="thickness">
    <Legend>not_used</Legend>
    <ContourLine width=".6" linecolor="#FF0000" classes="5760" />
    <ContourLine width=".6" linecolor="#FF7700" classes="5520,5540,5560,5580,5600,5620,5640,5660,5680,5700,5720" />
    <ContourLine width=".6" linecolor="#00FFFF" textcolor="#007979" textformatting="2100m" classes="5480" />
    <ContourLine width=".6" linecolor="#00FFFF" textcolor="#007979" textformatting="1800m" classes="5440" />
    <ContourLine width=".6" linecolor="#0000FF" textcolor="#0000FF" textformatting="1500m" classes="5400" />
    <ContourLine width=".6" linecolor="#0000FF" textcolor="#0000FF" textformatting="1200m" classes="5360" />
    <ContourLine width=".6" linecolor="#DD00FF" textcolor="#DD00FF" textformatting="900m" classes="5320" />
    <ContourLine width=".6" linecolor="#DD00FF" textcolor="#DD00FF" textformatting="600m" classes="5280" />
    <ContourLine width=".6" linecolor="#BB00FF" textcolor="#BB00FF" textformatting="300m" classes="5240" />
    <ContourLine width=".6" linecolor="#BB00FF" textcolor="#BB00FF" textformatting="Sea Level" classes="5200" />
    <ContourLine width=".6" linecolor="#BB00FF" classes="5100,5120,5140,5160,5180" />

    <RenderMethod>contour</RenderMethod>
    <NameMapping name="contour" title="Thickness contour" abstract="Contourlines on a transparent background" />
</Style>

When I display this the 'Sea Level' contour line is not shown. If I remove a few of these ContourLine's it shows up again suggesting there is a hard limit on how many ContourLine's the server will show. Is it a setting or something on the server?

Include adaguc config file that is using ADAGUC_DB environment variable

ADAGUC_DB is suggested in the docker-compose example; but the update database call does not receive a config file that useses ADAGUC_DB variable and instead tries to connect to a database on localhost. Sadly the adaguc.docker.yml uses sqlite, which doesn't work with all datasets the last time I tried :)

Make it possible to switch off adaguc-server logging in production mode.

Make it possible to switch off adaguc-server logging in production mode. The logging is flooding the logging systems.

Suggested solution: extend ADAGUC_ENABLELOGBUFFER with an extra option: DISABLELOGGING

ADAGUC_ENABLELOGBUFFER=FALSE: alse means buffered logging
ADAGUC_ENABLELOGBUFFER=TRUE: means unbuffered output with live logging but means a slower service
ADAGUC_ENABLELOGBUFFER=DISABLELOGGING: nologging means no logging at all

Legend shows without ticks/labels

With the config for SAT.xml, legends show up without labels and tick marks.
The unit shows up correctly though.

image

<?xml version="1.0" encoding="UTF-8" ?>
<Configuration>
  <!-- 
  See https://dev.knmi.nl/projects/adagucserver/wiki/Dataset, for details
  This file can be included by using the adaguc.dataset.cgi?service=wms&DATASET=testdata& key value pair in the URL 
  -->
  <!-- Styles -->
  <Legend name="gray" type="colorRange">
    <palette index="0"   red="0"   green="0"   blue="0"/>
    <palette index="240" red="255" green="255"   blue="255"/>
  </Legend>
  <Legend name="invgray" type="colorRange">
    <palette index="0" red="255" green="255"   blue="255"/>
    <palette index="240"   red="0"   green="0"   blue="0"/>
  </Legend>
  <Style name="gray">
    <Legend>gray</Legend>
    <Min>313.15</Min>
    <Max>193.15</Max>
    <RenderMethod>nearest,bilinear</RenderMethod>
  </Style>

 <Layer type="database">
    <Name>GOES16-C12</Name>
    <Title>goes16-c12</Title>
    <Variable>CMI</Variable>
    <FilePath filter="^OR_ABI-L2-CMIPF-M6C12_G16.*\.nc$">/data/adaguc-data/SAT</FilePath>
    <Dimension name="t">time</Dimension>
    <Styles>gray</Styles>
  </Layer>
</Configuration>

Palette entry 239 ignored

On a interval legend the palette entyry for 239 seems to be ignored.
Example:

<Style name="pressure"> pressure nearest 10000 110000 </Style>

DataPostProc issue with time series data

We are facing issues plotting data stored on a time series NetCDF when using DataPostProc on the layer definition.

Adaguc version: 2.5.4.

We use DataPostProc to override the declared units in the NetCDF.

This is how we are setting up the process environment:

ADAGUC_CONFIG=/tmp/adagucMgrCfg/configuration.xml
ADAGUC_TMP=/tmp
ADAGUC_LOGFILE=/tmp/adaguctmp/adaguc-1362314522471699936.log
HOME=/tmp
QUERY_STRING=service=WMS&request=GetMap&version=1.1.1&layers=tasmaxhwdmax&format=png&transparent=true&dataset=AEMET-rcp45&crs=EPSG%3A4326&width=1600&height=830&srs=EPSG%3A3857&bbox=-1335507.7581985998%2C4474929.383927359%2C621280.1659019126%2C5490013.1195545
CONTENT_TYPE=null
ADAGUC_ONLINERESOURCE=http://localhost:8080/wms?
ADAGUC_PATH=/adaguc/adaguc-server-master
ADAGUC_ERRORFILE=/tmp/adaguc-server.errlog
ADAGUC_FONT=/adaguc/adaguc-server-master/data/fonts/Roboto-Light.ttf
ADAGUC_DATARESTRICTION=FALSE
ADAGUC_DB=host=db port=5432 user=user password=changeme dbname=adaguc

This is the execution log we get before the process crashes:

[D:000:pid113: /adaguc/adaguc-server-master/adagucserverEC/CAutoResource.cpp, 81 in CAutoResource] 2021-02-03T09:19:03Z Found dataset /tmp/adagucMgrCfg/AEMET-rcp45.xml
[D:001:pid113: /adaguc/adaguc-server-master/adagucserverEC/CRequest.cpp, 818 in CRequest] 2021-02-03T09:19:03Z WMS GETMAP (0) tasmaxhwdmax
[D:002:pid113: /adaguc/adaguc-server-master/adagucserverEC/CRequest.cpp, 1453 in CRequest] 2021-02-03T09:19:03Z Using maxquerylimit 365
[D:003:pid113: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 765 in CDataReader] 2021-02-03T09:19:03Z Working on [/tmp/tasmaxhwdmax_mmm_rcp45__year_1971_2100_clim.nc]
[D:004:pid113: /adaguc/adaguc-server-master/adagucserverEC/CConvertADAGUCPoint.cpp, 171 in CConvertADAGUCPoint] 2021-02-03T09:19:04Z Using CConvertADAGUCPoint.h
[D:005:pid113: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 582 in CDataReader] 2021-02-03T09:19:04Z Assuming that for variable tasmaxhwdmax the x dim equals x and the y dim equals y based on their position and name.
[D:006:pid113: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 609 in CDataReader] 2021-02-03T09:19:04Z Using variable x as X variable and variable y as Y variable.
[D:007:pid113: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 618 in CDataReader] 2021-02-03T09:19:04Z In level 2 compatibility mode, using a default stride of 1, not considering RenderSettings.
[D:008:pid113: /adaguc/adaguc-server-master/adagucserverEC/CDataPostProcessor.cpp, 51 in CDPPAXplusB] 2021-02-03T09:19:04Z Applying ax+b
[D:009:pid113: /adaguc/adaguc-server-master/adagucserverEC/CImageDataWriter.cpp, 3394 in CImageDataWriter] 2021-02-03T09:19:04Z Creating 8 bit png with alpha
[D:010:pid113: /adaguc/adaguc-server-master/adagucserverEC/CDrawImage.cpp, 216 in CDrawImage] 2021-02-03T09:19:04Z printImagePng8 CAIRO
[D:011:pid113: /adaguc/adaguc-server-master/adagucserverEC/CCairoPlotter.cpp, 899 in CCairoPlotter] 2021-02-03T09:19:04Z Number of quantized colors: 2
[D:012:pid113: /adaguc/adaguc-server-master/adagucserverEC/CCairoPlotter.cpp, 920 in CCairoPlotter] 2021-02-03T09:19:04Z Num alpha colors: 2

And the relevant configuration:

<Configuration>
   <Style name="number_of_days">
      <Legend fixed="true">temperature</Legend>
      <RenderMethod>nearest</RenderMethod>
      <NameMapping name="nearest" title="Número de días" abstract="Número de días"/>
   </Style>
   <Style name="number_of_days_point">
      <Legend fixed="true">temperature</Legend>
      <RenderMethod>nearest</RenderMethod>
      <Point plotstationid="false" pointstyle="point" discradius="8" textradius="0" dot="false" fontsize="0" textformat=" " textcolor="#000000"/>
      <RenderMethod>point</RenderMethod>
      <NameMapping name="point" title="Número de días" abstract="Número de días"/>
   </Style>
   <Layer type="database">
      <Name>tasmaxhwdmax</Name>
      <FilePath>/tmp/tasmaxhwdmax_mmm_rcp45__year_1971_2100_clim.nc</FilePath>
      <DataPostProc algorithm="ax+b" a="1.0" b="0.0" units="Dias" />
      <Variable>tasmaxhwdmax</Variable>
      <Styles>number_of_days,number_of_days_point</Styles>
      <Min>5.0</Min>
      <Max>25.0</Max>
   </Layer>
</Configuration>

TROPOMI NetCDF files give a warning

[W: CCDFNetCDFIO.cpp, 620 in NetCDFReader] Warning unknown attribute type 10 [W: CCDFNetCDFIO.cpp, 425 in NetCDFReader] Unknown attribute type for attribute time_reference_seconds_since_1970 10 11 0
Apparently newer NetCDF library version introduced a new attribute type that is being used by TROPOMI files and not yet supported by adaguc-server?

Allow "radian" as a possible value for the X/Y dimension units.

}else if(units.equals("rad")){

According with the Unidata udunits documentation for naming units, the value "rad" is used as a radiation unit for "absorbed dose".
Thus the value "rad" for radian unit shouldn't be used. Instead, the correct value should be "radian".
As one can see in the following doc:
https://www.unidata.ucar.edu/software/udunits/udunits-1/udunits.txt

Besides, the GDAL and the netcdf-java libraries are already supporting the "radian" units for the "geostationary" projection coordinates. This would make the things easier for the data providers that are using the "geostationary" projection, in order to build their netCDF files compatible with the most of the softwares. Including yours...

In order to solve this issue I'd suggest the value "radian" to be added as a possible value for the X/Y axis/coordinates unit. (The value "rad" also should be allowed, to keep retro-compatibility...)

Regards João

CSV Not reading negative floats or integers

hi
I'm trying to display a csv with negative temperatures something like this :

lat,lon,tn
-8,38,-1.02
-8.5,38,2.02
(...)

and my var tn is classified as a string where should be a float, not allowing a color scale in adaguc
Looking in the adaguc code the lines that classify the type :

CT::string col = firstLine[c].c_str();
CDFType dataType = CDF_FLOAT;
if (col.isInt()){
dataType = CDF_INT;
}else if (col.isFloat()){
dataType = CDF_FLOAT;
}else {dataType = CDF_STRING;}

and checking isFloat() function:
bool string::isFloat() {
if (this->empty()) return false;
const char digitsAllowedForFloat[] = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '.', 'f'};
const size_t numDigits = 12;
const char *inputStr = this->c_str();
const size_t inputLength = this->length();
/* Only a dot is not a float */
if (inputLength == 1 && inputStr[0] == '.') {
return false;
}
if (inputLength == 3 && this->equals("NaN")) {
return true;
}
/* Check for allowed characters, and check if '.' is included */
bool hasDotCharacter = false;
for (size_t c = 0;c< inputLength;c++) {
if (includesFunction(digitsAllowedForFloat ,numDigits, inputStr[c])!=true){
return false;
}
if (inputStr[c] == '.') hasDotCharacter = true;
}
if (!hasDotCharacter) return false;
return true;
}

I notice that the '-' sign is not in the allowed chars, therefore not classifying the negative float correctly .

I also notice the function is not checking that a float can only have one decimal separator (.) allowing '1.2.3' to be a "valid" float .

Thanks,
Cheers

timeSeries NetCDF not detecting first dimension

We have a multi-dimensional timeSeries NetCDF. The layer depends on three string-valued dimensions (member, time_filter and period):

	float tasmax(member, time_filter, period, station) ;
		tasmax:_FillValue = NaNf ;
		tasmax:long_name = "Daily Maximum Near-Surface Air Temperature" ;
		tasmax:standard_name = "air_temperature" ;
		tasmax:units = "Celsius" ;
		tasmax:coordinates = "lat lon projection" ;

This is the layer definition:

<Configuration>
    <Layer type="database">
        <Name>tasmax</Name>
        <FilePath>/temp/tasmax_multimodel_rcp45_1971_2100_clim.nc</FilePath>
        <Variable>tasmax</Variable>
        <Styles>temperature,temperature_point</Styles>
        <Min>5.0</Min>
        <Max>45.0</Max>
    </Layer>
</Configuration>

After a successful on DB update, the first dimension is not auto-detected:

[D:003:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 959 in CDBFileScanner] *** Starting update layer 'tasmax' ***
[D:004:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 961 in CDBFileScanner] Using path [/temp/tasmax_multimodel_rcp45_1971_2100_clim.nc], filter [] and tailpath []
[D:005:pid67: /adaguc/adaguc-server-master/adagucserverEC/CConvertADAGUCPoint.cpp, 171 in CConvertADAGUCPoint] Using CConvertADAGUCPoint.h
[D:006:pid67: /adaguc/adaguc-server-master/adagucserverEC/CAutoConfigure.cpp, 529 in CAutoConfigure] Loading header [/temp/tasmax_multimodel_rcp45_1971_2100_clim.nc]
[D:007:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 1346 in CDataReader] Dimension time_filter with standard_name [time_filter] is a normal dimension
[D:008:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 1346 in CDataReader] Dimension period with standard_name [period] is a normal dimension
[D:009:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 118 in CDBFileScanner] Checking dim [time_filter]
[D:010:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 1346 in CDataReader] Dimension time_filter with standard_name [time_filter] is a normal dimension
[D:011:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBAdapterPostgreSQL.cpp, 876 in CDBAdapterPostgreSQL] New table created: Set indexes
[D:012:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 118 in CDBFileScanner] Checking dim [period]
[D:013:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 1346 in CDataReader] Dimension period with standard_name [period] is a normal dimension
[D:014:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBAdapterPostgreSQL.cpp, 876 in CDBAdapterPostgreSQL] New table created: Set indexes
[D:015:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 1346 in CDataReader] Dimension time_filter with standard_name [time_filter] is a normal dimension
[D:016:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 367 in CDBFileScanner] Found dimension 0 with name time_filter of type 1, istimedim: [0]
[D:017:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 389 in CDBFileScanner] Marking table done for dim 'time_filter' with table 't20210203t165310141_wqvhuc03rg1dz6m5yaz'.
[D:018:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 1346 in CDataReader] Dimension period with standard_name [period] is a normal dimension
[D:019:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 367 in CDBFileScanner] Found dimension 1 with name period of type 1, istimedim: [0]
[D:020:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 389 in CDBFileScanner] Marking table done for dim 'period' with table 't20210203t165310153_ljxhrl2p95peyapv2pnq'.
[D:021:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 396 in CDBFileScanner] Found 1 files Adding: 0/1 (time_filter, period) /temp/tasmax_multimodel_rcp45_1971_2100_clim.nc
[D:023:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 582 in CDataReader] Assuming that for variable tasmax the x dim equals x and the y dim equals y based on their position and name.
[D:024:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 609 in CDataReader] Using variable x as X variable and variable y as Y variable.
[D:025:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 618 in CDataReader] In level 2 compatibility mode, using a default stride of 1, not considering RenderSettings.
[D:026:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 582 in CDataReader] Assuming that for variable tasmax the x dim equals x and the y dim equals y based on their position and name.
[D:027:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 609 in CDataReader] Using variable x as X variable and variable y as Y variable.
[D:028:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDataReader.cpp, 618 in CDataReader] In level 2 compatibility mode, using a default stride of 1, not considering RenderSettings.
[D:029:pid67: /adaguc/adaguc-server-master/adagucserverEC/CDBFileScanner.cpp, 1048 in CDBFileScanner] *** Finished update layer 'tasmax' ***

As a consequence only two dimensions are stored in the DB and advertised in GetCapabilities:

<Dimension name="time_filter" units="ISO8601" default="year" multipleValues="1" nearestValue="0" current="1">
	Apr,Aug,Dec,DJF,Feb,Jan,JJA,Jul,Jun,MAM,Mar,May,Nov,Oct,Sep,SON,year
</Dimension>
<Dimension name="period" units="NA" default="far_future" multipleValues="1" nearestValue="0">reference,near_future,medium_future,far_future</Dimension>

The adaguc version is: 2.5.4.

Support retentionperiod to autoclean files

Add optional retentionperiod and retentiontype attributes to FilePath to support auto cleaning of datastreams. Triggered during update of the layer. When a file is added, the cleanup is triggered. The cleanup will start with the oldest files. It will first delete the files from the databasetable, and later from the filesystem. When multiple layers come from the same file, the most strict/shortest retentionperiod is taken.

<?xml version="1.0" encoding="UTF-8" ?>
<Configuration>
  
  <Layer type="database">
    <FilePath filter=".*\.nc$" retentionperiod="PT5M" retentiontype="filetimedate">/data/adaguc-data/livetimestream/</FilePath>
    
    <Variable>data</Variable>
    <Dimension name="time" interval="PT1M" >time</Dimension>
    <RenderMethod>rgba</RenderMethod>
  </Layer>

</Configuration>

Problem with DataPostProc addfeatures

We are trying to plot a timeSeries NetCDF using geometries from an external GeoJSON as described in the documentation.

We have several layers in the NetCDF

netcdf spain-static_values-transposed {
dimensions:
	station = 8235 ;
	time = 1 ;
variables:
	string station(station) ;
		station:long_name = "station_name" ;
		station:cf_role = "timeseries_id" ;
	double lat(station, time) ;
		lat:_FillValue = NaN ;
		lat:units = "degrees_north" ;
		lat:long_name = "Latitude" ;
		lat:standard_name = "latitude" ;
	double lon(station, time) ;
		lon:_FillValue = NaN ;
		lon:units = "degrees_east" ;
		lon:long_name = "Longitude" ;
		lon:standard_name = "longitude" ;
	double fp_fd(station, time) ;
		fp_fd:units = "Number of days since year began" ;
		fp_fd:standard_name = "fp_fd" ;
		fp_fd:long_name = "Fisrt day of summer in first period" ;
		fp_fd:grid_mapping = "projection" ;
	int64 time(time) ;
		time:standard_name = "time" ;
		time:units = "days since 2021-06-07 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
	string projection ;
		projection:proj4 = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs" ;

// global attributes:
		:featureType = "timeSeries" ;
		:Conventions = "CF-1.5" ;

We are able to plot the NetCDF with points and the Geometries independently, but once we combine them using the DataPostProc we get an error:

adagucserverEC/CAutoResource.cpp, 81 in CAutoResource] 2021-06-08T07:32:30Z Found dataset /tmp/adagucMgrCfg/summer-length-zoom-3.xml
adagucserverEC/CRequest.cpp, 822 in CRequest] 2021-06-08T07:32:30Z WMS GETMAP (0) first-period-last-day
adagucserverEC/CRequest.cpp, 1457 in CRequest] 2021-06-08T07:32:30Z Using maxquerylimit 365
adagucserverEC/CDataReader.cpp, 765 in CDataReader] 2021-06-08T07:32:30Z Working on [/data/summer-length/spain-static_values-transposed.nc]
adagucserverEC/CConvertADAGUCPoint.cpp, 113 in CConvertADAGUCPoint] 2021-06-08T07:32:30Z Using CConvertADAGUCPoint.h
adagucserverEC/CDataReader.cpp, 582 in CDataReader] 2021-06-08T07:32:30Z Assuming that for variable fp_ld the x dim equals x and the y dim equals y based on their position and name.
adagucserverEC/CDataReader.cpp, 609 in CDataReader] 2021-06-08T07:32:30Z Using variable x as X variable and variable y as Y variable.
adagucserverEC/CDataReader.cpp, 618 in CDataReader] 2021-06-08T07:32:30Z In level 2 compatibility mode, using a default stride of 1, not considering RenderSettings.
adagucserverEC/CDataPostProcessor.cpp, 1170 in CDPPAddFeatures] 2021-06-08T07:32:30Z CDATAPOSTPROCESSOR_RUNBEFOREREADING::Adding features from GEOJson
adagucserverEC/CDataPostProcessor.cpp, 1208 in CDPPAddFeatures] 2021-06-08T07:32:30Z Opening /data/summer-length/new-spain-zoom-3.geojson
adagucserverEC/CDataReader.cpp, 765 in CDataReader] 2021-06-08T07:32:30Z Working on [/data/summer-length/new-spain-zoom-3.geojson]
CCDFDataModel/CCDFGeoJSONIO.cpp, 36 in GeoJSONReader] 2021-06-08T07:32:30Z New CDFGeoJSONReader
adagucserverEC/CConvertGeoJSON.cpp, 520 in CConvertGeoJSON] 2021-06-08T07:32:30Z convertGeoJSONHeader
adagucserverEC/CConvertGeoJSON.cpp, 551 in CConvertGeoJSON] 2021-06-08T07:32:30Z addCDFInfo
adagucserverEC/CDataSource.h, 384 in CDataSource] 2021-06-08T07:32:30Z attachCDFObject: variable "fp_sl" does not exist
adagucserverEC/CDataReader.cpp, 790 in CDataReader] 2021-06-08T07:32:30Z Unable to attach CDFObject
adagucserverEC/CDataPostProcessor.cpp, 1214 in CDPPAddFeatures] 2021-06-08T07:32:30Z Can't open file /data/summer-length/new-spain-zoom-3.geojson
adagucserverEC/CDataPostProcessor.cpp, 782 in CDPPExecutor] 2021-06-08T07:32:30Z Processor addfeatures failed RUNAFTERREADING, statuscode 1
adagucserverEC/CImageDataWriter.cpp, 3375 in CImageDataWriter] 2021-06-08T07:32:30Z Error occured during image data writing
adagucserverEC/CRequest.cpp, 2215 in CRequest] 2021-06-08T07:32:30Z Returning from line 2078
adagucserverEC/CRequest.cpp, 3174 in CRequest] 2021-06-08T07:32:30Z WMS GetMap Request failed
<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<!DOCTYPE ServiceExceptionReport SYSTEM "http://schemas.opengis.net/wms/1.1.1/exception_1_1_1.dtd">
<ServiceExceptionReport version="1.1.1">
  <ServiceException>
    attachCDFObject: variable "fp_sl" does not exist;
    Unable to attach CDFObject;
    Processor addfeatures failed RUNAFTERREADING, statuscode 1;
    Error occured during image data writing;
    Returning from line 2078;
    WMS GetMap Request failed;
  </ServiceException>
</ServiceExceptionReport>

We get always errors related to fp_sl layer even though we have requested other layers from the file. This is a snippet of our configuration:

<Configuration>
	<Legend name="summer-length-days" type="interval">
		<palette index="0" color="##ffffcc" />
		<palette index="30" color="#ffeda0" />
		<palette index="60" color="#fed976" />
		<palette index="90" color="#feb24c" />
		<palette index="120" color="#fd8d3c" />
		<palette index="150" color="#fc4e2a" />
		<palette index="180" color="#e31a1c" />
		<palette index="210" color="#bd0026" />
		<palette index="240" color="#800026" />
	</Legend>
	<Style name="summer-length">
		<Legend fixed="true" tickinterval="10">summer-length-days</Legend>
		<Min>30</Min>
		<Max>120</Max>
		<RenderMethod>nearest</RenderMethod>
		<NameMapping name="nearest" />
	</Style>
	<Style name="summer-length-last-day">
		<Legend fixed="true" tickinterval="10">summer-length-days</Legend>
		<Min>200</Min>
		<Max>300</Max>
		<RenderMethod>nearest</RenderMethod>
		<NameMapping name="nearest" />
	</Style>	
	<Layer type="database">
		<Name>first-period-summer-length</Name>
		<FilePath>/data/summer-length/spain-static_values-transposed.nc</FilePath>
		<DataPostProc algorithm="addfeatures" a="/data/summer-length/new-spain-zoom-3.geojson" />
		<Variable>fp_sl</Variable>
		<Styles>summer-length</Styles>
	</Layer>
	<Layer type="database">
		<Name>last-period-last-day</Name>
		<FilePath>/data/summer-length/spain-static_values-transposed.nc</FilePath>
		<DataPostProc algorithm="addfeatures" a="/data/summer-length/new-spain-zoom-3.geojson" />
		<Variable>lp_ld</Variable>
		<Styles>summer-length-last-day</Styles>
	</Layer>
</Configuration>

And the data files: summer-length.zip

Can you please advise? Many thanks in advance.

adaguc-server-updatedatasets.sh command line option parsing not extendable

Because the script relies on positional arguments and tests for a second argument if there is a first argument detected it is impossible to add command line arguments independent of the first command line argument.

I'm expecting there are currently other scripts dependent on this interface, so my current patch in pull request #70 doesn't introduce any backwards compatibility (hopefully) by using an environment variable, but I had to deal with the deletion of servicehealth files.

It would help if the command line argument parsing were refactored using the getopts bash function so that command line arguments can be added independent of already existing ones. Let me know if you want me to make a new pull request for this. For now please merge #70.

workshop on ADAGUC server and viewer

Is there a workshop sheduled ?
It's good to have a community gathered around an event
where real use cases can be exposed.

Let me know.
Regards

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.