meteofrance / meteonet Goto Github PK

View Code? Open in Web Editor NEW

118.0 118.0 22.0 243.25 MB

MeteoNet's toolbox and documentation

License: Other

Python 0.07% Jupyter Notebook 99.93%

meteonet's People

Contributors

Stargazers

Watchers

meteonet's Issues

Starting notebook fails with FileNotFoundError

Hello,
I installed, using virtualenv and PIP.
Starting open_ground_stations.ipynb, I have the following error:

(METEONET) [rechtem@diabolik-arch meteonet]$ ipython notebooks/ground_stations/open_ground_stations.ipynb 
/usr/lib/python3.8/site-packages/IPython/core/interactiveshell.py:935: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn("Attempting to work in a virtualenv. If you encounter problems, please "
2020-02-26 08:21:15,930  - WARNING - User is not known. If you experiment some problems with path for library please add the specific path in `user_configuration.py` module
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
~/python/METEONET/meteonet/notebooks/ground_stations/open_ground_stations.ipynb in <module>
      1 zone, date, param = 'SE', '20160101', 'hu'
      2 fname = "../../data_samples/ground_stations/%s_20160101.csv"%zone
----> 3 df = pd.read_csv(fname,parse_dates=[4],infer_datetime_format=True)
...

FileNotFoundError: [Errno 2] File ../../data_samples/ground_stations/SE_20160101.csv does not exist: '../../data_samples/ground_stations/SE_20160101.csv'

However, this file exists:

(METEONET) [rechtem@diabolik-arch meteonet]$ ls -l data_samples/ground_stations/SE_20160101.csv 
-rw-r--r-- 1 rechtem rechtem 7679526 25 févr. 16:45 data_samples/ground_stations/SE_20160101.csv

On the other hand, moving first to the notebook directory, and starting from there gives:

(METEONET) [rechtem@diabolik-arch meteonet]$ cd notebooks/ground_stations/
(METEONET) [rechtem@diabolik-arch ground_stations]$ ipython open_ground_stations.ipynb 
/usr/lib/python3.8/site-packages/IPython/core/interactiveshell.py:935: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn("Attempting to work in a virtualenv. If you encounter problems, please "
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
~/python/METEONET/meteonet/notebooks/ground_stations/open_ground_stations.ipynb in <module>
      1 #from data_exploration.utils.user_configuration import *
----> 2 from meteonet_toolbox.user_configuration import *

ModuleNotFoundError: No module named 'meteonet_toolbox'

I tried installing ipython in the virtualenv, but it gives the same error.
Thanks

pip install cartopy fails due to compilation error

Hello,
cartopy is missing in the project's requirements.txt.
Trying to install manually in the virtualenv gives the following error:

  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -I/usr/include -I./lib/cartopy -I/usr/include -I/usr/include/python3.8 -c lib/cartopy/trace.cpp -o build/temp.linux-x86_64-3.8/lib/cartopy/trace.o
  Dans le fichier inclus depuis lib/cartopy/trace.cpp:612:
  /usr/include/proj_api.h:37:2: erreur: #error 'To use the proj_api.h you must define the macro ACCEPT_USE_OF_DEPRECATED_PROJ_API_H'
     37 | #error 'To use the proj_api.h you must define the macro ACCEPT_USE_OF_DEPRECATED_PROJ_API_H'
        |  ^~~~~
  Dans le fichier inclus depuis lib/cartopy/_trace.h:27,
                   depuis lib/cartopy/trace.cpp:613:
  /usr/include/proj_api.h:37:2: erreur: #error 'To use the proj_api.h you must define the macro ACCEPT_USE_OF_DEPRECATED_PROJ_API_H'
     37 | #error 'To use the proj_api.h you must define the macro ACCEPT_USE_OF_DEPRECATED_PROJ_API_H'
        |  ^~~~~
  error: command 'gcc' failed with exit status 1

Thanks

Installation issues for basemap and cartopy libraries

If you want to install basemap or cartopy via Anaconda :
conda install -c conda-forge basemap
conda install -c conda-forge cartopy
you could have issues of versions compatiblities for some libraries :
-> example : about cartopy
UnsatisfiableError: The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (mock):

cartopy -> mock

The following specifications were found to be incompatible with each other:
Package mock conflicts for:
anaconda/linux-64::keras-base==2.2.4=py37_0 -> keras-applications[version='>=1.0.6'] -> keras[version='>=2.1.6'] -> tensorflow -> tensorflow-estimator=1.15 -> mock[version='>=2.0.0']
anaconda/noarch::tensorflow-estimator==2.0.0=pyh2649769_0 -> tensorflow-base[version='>=2.0.0,<2.1.0a0'] -> mock[version='>=2.0.0,<3.0a0']
conda-forge/linux-64::pytables==3.6.1=py37h9f153d1_0 -> mock
anaconda/linux-64::tensorflow-base==2.0.0=mkl_py37h9204916_0 -> keras-preprocessing[version='>=1.0.5,<2.0a0'] -> keras[version='>=2.1.6'] -> tensorflow -> tensorflow-estimator=1.15 -> mock[version='>=2.0.0']
defaults/linux-64::anaconda==custom=py37_1 -> _anaconda_depends -> pytables -> mock
anaconda/linux-64::keras==2.2.4=0 -> tensorflow -> mock[version='>=2.0.0']
defaults/linux-64::_anaconda_depends==2019.03=py37_0 -> pytables -> mock
defaults/linux-64::mock==3.0.5=py37_0

Why no data for all region?

The data is for a "NW" and "SE" quarters. Would be possible to have all the data?

Model data : anomalies and missing data

Hello,

some files in the model files are incomplete/corrupted. These anomalies are also present in the original files. So unfortunately we can not do anything.
You can detect these anomalies by checking the file sizes. For a given file category (zone, level, model type), the files must have exactly the same size. So if some files are smaller than the other one, then these files have anomalies.

This has been reported in kaggle in this discussion
https://www.kaggle.com/katerpillar/meteonet/discussion/157206

Eccodes : installation with conda fails

Several users have encountered issues when installing the library Eccodes, used to open GRIB files with xarray. It seems that this library can create conflicts with other packages.
Consequently, we made the decision to stop using the GRIB format in the dataset and to convert the GRIB files to the netCDF format which is more common and should pose less intallation issues.
This conversion concerns the land-sea and relief masks and 2D and 3D meteo model outputs.
We will try to convert the files as fast as possible, but meanwhile if you encountered issues with Eccodes, you can still try working on the other parts of the dataset : ground station observations and radar data.
We're sorry for the inconvenience.

Optimize the storage

I would suggest you to move all the datasets to other more structured file format (NetCDF for example) instead of having csv, npz, etc.

Meanwhile, for the data that you currently have, two recommendations:

I think is better to compress file by file directly (store NW2016.csv.gz directly instead of taring NW_ground_stations_2016.tar.gz). That would allow users to work directly with compressed data (with tools like zcat) and download only what they need (multiple files case).
Use xz instead of gzip to reduce the current space. The NW2016.csv file with level 9 have a final size of 98MB, for example

Radar coordinates

I couldn't find the information in the documentation, so: do the radar rainfall coordinates correspond to the center of the pixels, or one of the corners?

Page to download meteonet toolbox not found

Hi ! I would like to use the meteonet database and toolbox, however I can't download the toolbox, the page is not found when I go to the link. Do you have another way for downloading it ?

Thanks in advance !

NW data missing in ground_stations

Hi meteonet teams,

Really good job and initiative, thank you.

For now, the groundstations folder only seems to contain the data for the SE of France.
Are you planning on releasing the dataset for the NW data as well?

Thanks in advance,
G.

document ARPEGE parameters

Is it possible to have in meteonet doc more info about parameters available here https://mf-models-on-aws.org/#arpege-europe/v2/ ? For some parameters like Radiation Flux, Heat Flux, I can't find a relevant doc.

Conflict between basemap and cfgrib

Cannot install basemap with anaconda on windows, conflict with cfgrib.
(base) C:\Users\perezl>conda install -c anaconda basemap
Solving environment: failed
UnsatisfiableError: The following specifications were found to be in conflict:

basemap
cfgrib
Use "conda info " to see the dependencies for each package.

Error on loading data_samples when using open_weather_models_2D_parameters

DatasetBuildError: multiple values for unique key, try re-open the file with one of:
filter_by_keys={'stepType': 1}
filter_by_keys={'stepType': 0}

Ground station observations : extra CR in CSV files

Hello,

we notice for each line of a .csv file has TWO CR and one LR (Carriage Return and Line Return).
On Linux you should have only one LR, and on Windows you have one CR and one LR.

This extra CR at the end of each csv line does not mean anything.
We think this comes from our preprocessing task which is based on plain text file. We could have forgotten to delete some characters when extracting and converting to csv.
So, we note this as an issue and will try to fix it.

Data for other regions (SW and NE)

Hello MeteoNet team,
Thanks for the datasets already existing, great job!

I wanted to know if you plan to release also data for South West and North East part of France, especially for the ground stations data.

Thanks in advance,

Error when reading GRIB files from Météo France

We started working with meteonet data for weather forecast, then wanted to buy more archived data from Météo France to cover our use cases. We now encounter some errors when reading the files as detailed in your documentation:

import xarray as xr 
data = xr.open_dataset('E:\Download\T47648_AROME0025.2017010100.grb', engine='cfgrib', backend_kwargs={"indexpath" : ""}) 
data

Here are the errors:

skipping variable: paramId==228228 shortName='tp'
Traceback (most recent call last):
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=Variable(dimensions=('step',), data=array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
       13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
       26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
       39., 40., 41., 42.])) new_value=Variable(dimensions=('step',), data=array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,
       14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
       27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.,
       40., 41., 42.]))
skipping variable: paramId==228164 shortName='tcc'
Traceback (most recent call last):
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=Variable(dimensions=('step',), data=array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
       13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
       26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
       39., 40., 41., 42.])) new_value=Variable(dimensions=('step',), data=array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,
       14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
       27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.,
       40., 41., 42.]))
skipping variable: paramId==3099 shortName='p3099'
Traceback (most recent call last):
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=Variable(dimensions=('step',), data=array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
       13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
       26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
       39., 40., 41., 42.])) new_value=Variable(dimensions=('step',), data=array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,
       14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
       27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.,
       40., 41., 42.]))
skipping variable: paramId==167 shortName='t2m'
Traceback (most recent call last):
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='heightAboveGround' value=Variable(dimensions=(), data=10.0) new_value=Variable(dimensions=(), data=2.0)
skipping variable: paramId==157 shortName='r'
Traceback (most recent call last):
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='heightAboveGround' value=Variable(dimensions=(), data=10.0) new_value=Variable(dimensions=(), data=2.0)
skipping variable: paramId==0 shortName='unknown'
Traceback (most recent call last):
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=Variable(dimensions=('step',), data=array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
       13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
       26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
       39., 40., 41., 42.])) new_value=Variable(dimensions=('step',), data=array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,
       14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
       27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.,
       40., 41., 42.]))

We tried this workaround without success. Any help on this is welcome, thanks.

Ground stations with multiple positions

Hello,

I've noticed that some ground stations have multiple positions (lat, lon) for a single station ID.

As example, in NW2016.csv for number_sta 22005003 you have entries for (48.550,-3.388) and (48.551,-3.388). This is what should be? Or a station must have associated only one lat & lon?

To get a comple list of stations with multiple entries for lat & lon in a file, you can execute this:

$ cut -d, -f1,2,3 NW2016.csv | sort -u | awk -F , '
!count[$1]++ {save[$1] = $0; next}
  count[$1] == 2 {
    print save[$1]
    delete save[$1]
  }
  {print}'

17218001,46.315,-1.000
17218001,46.320,-1.010
22005003,48.550,-3.388
22005003,48.551,-3.388
22016001,48.857,-3.005
22016001,48.860,-3.000
22247002,48.740,-3.250
22247002,48.742,-3.251
22261002,48.519,-2.420
22261002,48.520,-2.420
22266001,48.227,-3.295
22266001,48.230,-3.310
27422001,49.220,1.280
27422001,49.224,1.280
28002001,48.190,1.850
28002001,48.191,1.847
28041001,48.300,1.270
28041001,48.303,1.271
28190001,48.081,1.666
28190001,48.090,1.670
28215001,48.329,1.817
28215001,48.330,1.820
28252002,48.250,0.990
28252002,48.253,0.989
28406001,48.196,1.680
28406001,48.200,1.680
28407001,48.258,0.910
28407001,48.260,0.910
29263002,48.228,-4.098
29263002,48.230,-4.100
29276001,48.660,-4.077
29276001,48.660,-4.080
29293001,47.840,-3.870
29293001,47.841,-3.869
37023002,47.177,1.212
37023002,47.180,1.210
37216002,47.160,0.600
37216002,47.170,0.630
49008001,47.560,-0.970
49008001,47.562,-0.970
49191001,47.250,-0.440
49191001,47.252,-0.436
50111001,49.036,-1.298
50111001,49.040,-1.300
50215002,49.108,-1.607
50215002,49.110,-1.610
50514001,48.725,-0.887
50514001,48.759,-0.843
56127001,48.070,-2.280
56127001,48.070,-2.290
56243001,47.600,-2.710
56243001,47.605,-2.714
62054001,50.870,1.580
62054001,50.870,1.583
78140001,48.960,1.920
78140001,48.964,1.925
78152001,48.840,1.980
78152001,48.860,2.000
78465001,48.860,1.690
78465001,48.860,1.694
79079013,46.924,-0.755
79079013,46.929,-0.758
79253002,46.457,-0.205
79253002,46.460,-0.200
79309001,46.617,-0.571
79309001,46.620,-0.570
79326004,46.725,-0.020
79326004,46.730,-0.020
79329001,46.982,-0.215
79329001,46.990,-0.210
85092004,46.448,-0.780
85092004,46.450,-0.780
85163001,47.000,-2.260
85163001,47.005,-2.257
86092002,46.917,0.622
86092002,46.920,0.620
86096001,46.850,0.290
86096001,46.852,0.287
86272002,46.837,0.458
86272002,46.840,0.450

Questions about Radar data in MeteoNet.

We wanted to ask a question in Slack Workspace “https://tinyurl.com/meteonet-slack”, but it couldn't be accessed.

https://meteofrance.github.io/meteonet/english/data/rain-radar/

We want to ask what is the height of the radar data in the old product. Is it radar echo data representing a fixed altitude? For example, 850hPa and 600hPa. Or is it an accumulation of radar echo data from multiple altitudes?

Looking forward to your reply!

Bilinguisme

Bonjour,

Je suis la coordonnatrice nationale des données ouvertes du Service météorologique du Canada et à ce titre et au nom de mes collègues, je tiens à souligner votre initiative.

Cependant, je regrette que la documentation soit seulement en anglais. Est-il planifié de la rendre disponible en français ?

Au gouvernement du Canada, c'est obligatoire, nous devons produire toute documentation dans les 2 langues.

Voir par exemple la documentation publique de nos données ouvertes:

https://eccc-msc.github.io/open-data/

Merci et bravo encore!

Sandrine Edouard
Coordonnatrice nationale, Accès aux données ouvertes
National coordinator, Open Data Access
Applications en traitement de données environnementales -
Environmental Data Processing Applications
Centre météorologique canadien - Canadian meteorological centre (CMC)
Environnement et changement Climatique Canada - Environment and Climate Change Canada
http://www.ec.gc.ca/

Courriel/email: [email protected]
Téléphone/Phone: 514-421-4729
2121 Trans-Canada N. Suite 230
Dorval, Québec
CANADA H9P 1J3

Weather model data : precision about precipitation parameter

Here is some precision about precipitation parameter from weather model data.

For now, it is about the total precipitation since the beginning of the run (in kg.m-2 which is equivalent to mm). It is planned in a future version to modify that parameter in order to have the total precipitation between two time steps instead.

question sur la variable temps pour les données au sol.

Bonjour, petite question un peu bête les données datetime sont en heure UT ou temps "classique" (UT+1 ou +2 l'été) ? Pour savoir si je dois faire une correction en couplant ce jeu de données avec un autre en heure normale. Merci beaucoup !

meteofrance / meteonet Goto Github PK

meteonet's People

Contributors

Stargazers

Watchers

Forkers

meteonet's Issues

Recommend Projects

Recommend Topics

Recommend Org