meteofrance / meteonet Goto Github PK
View Code? Open in Web Editor NEWMeteoNet's toolbox and documentation
License: Other
MeteoNet's toolbox and documentation
License: Other
Hello,
I installed, using virtualenv and PIP.
Starting open_ground_stations.ipynb, I have the following error:
(METEONET) [rechtem@diabolik-arch meteonet]$ ipython notebooks/ground_stations/open_ground_stations.ipynb
/usr/lib/python3.8/site-packages/IPython/core/interactiveshell.py:935: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
warn("Attempting to work in a virtualenv. If you encounter problems, please "
2020-02-26 08:21:15,930 - WARNING - User is not known. If you experiment some problems with path for library please add the specific path in `user_configuration.py` module
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
~/python/METEONET/meteonet/notebooks/ground_stations/open_ground_stations.ipynb in <module>
1 zone, date, param = 'SE', '20160101', 'hu'
2 fname = "../../data_samples/ground_stations/%s_20160101.csv"%zone
----> 3 df = pd.read_csv(fname,parse_dates=[4],infer_datetime_format=True)
...
FileNotFoundError: [Errno 2] File ../../data_samples/ground_stations/SE_20160101.csv does not exist: '../../data_samples/ground_stations/SE_20160101.csv'
However, this file exists:
(METEONET) [rechtem@diabolik-arch meteonet]$ ls -l data_samples/ground_stations/SE_20160101.csv
-rw-r--r-- 1 rechtem rechtem 7679526 25 févr. 16:45 data_samples/ground_stations/SE_20160101.csv
On the other hand, moving first to the notebook directory, and starting from there gives:
(METEONET) [rechtem@diabolik-arch meteonet]$ cd notebooks/ground_stations/
(METEONET) [rechtem@diabolik-arch ground_stations]$ ipython open_ground_stations.ipynb
/usr/lib/python3.8/site-packages/IPython/core/interactiveshell.py:935: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
warn("Attempting to work in a virtualenv. If you encounter problems, please "
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
~/python/METEONET/meteonet/notebooks/ground_stations/open_ground_stations.ipynb in <module>
1 #from data_exploration.utils.user_configuration import *
----> 2 from meteonet_toolbox.user_configuration import *
ModuleNotFoundError: No module named 'meteonet_toolbox'
I tried installing ipython in the virtualenv, but it gives the same error.
Thanks
Hello,
cartopy is missing in the project's requirements.txt.
Trying to install manually in the virtualenv gives the following error:
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -I/usr/include -I./lib/cartopy -I/usr/include -I/usr/include/python3.8 -c lib/cartopy/trace.cpp -o build/temp.linux-x86_64-3.8/lib/cartopy/trace.o
Dans le fichier inclus depuis lib/cartopy/trace.cpp:612:
/usr/include/proj_api.h:37:2: erreur: #error 'To use the proj_api.h you must define the macro ACCEPT_USE_OF_DEPRECATED_PROJ_API_H'
37 | #error 'To use the proj_api.h you must define the macro ACCEPT_USE_OF_DEPRECATED_PROJ_API_H'
| ^~~~~
Dans le fichier inclus depuis lib/cartopy/_trace.h:27,
depuis lib/cartopy/trace.cpp:613:
/usr/include/proj_api.h:37:2: erreur: #error 'To use the proj_api.h you must define the macro ACCEPT_USE_OF_DEPRECATED_PROJ_API_H'
37 | #error 'To use the proj_api.h you must define the macro ACCEPT_USE_OF_DEPRECATED_PROJ_API_H'
| ^~~~~
error: command 'gcc' failed with exit status 1
Thanks
If you want to install basemap or cartopy via Anaconda :
conda install -c conda-forge basemap
conda install -c conda-forge cartopy
you could have issues of versions compatiblities for some libraries :
-> example : about cartopy
UnsatisfiableError: The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (mock):
cartopy -> mock
The following specifications were found to be incompatible with each other:
Package mock conflicts for:
anaconda/linux-64::keras-base==2.2.4=py37_0 -> keras-applications[version='>=1.0.6'] -> keras[version='>=2.1.6'] -> tensorflow -> tensorflow-estimator=1.15 -> mock[version='>=2.0.0']
anaconda/noarch::tensorflow-estimator==2.0.0=pyh2649769_0 -> tensorflow-base[version='>=2.0.0,<2.1.0a0'] -> mock[version='>=2.0.0,<3.0a0']
conda-forge/linux-64::pytables==3.6.1=py37h9f153d1_0 -> mock
anaconda/linux-64::tensorflow-base==2.0.0=mkl_py37h9204916_0 -> keras-preprocessing[version='>=1.0.5,<2.0a0'] -> keras[version='>=2.1.6'] -> tensorflow -> tensorflow-estimator=1.15 -> mock[version='>=2.0.0']
defaults/linux-64::anaconda==custom=py37_1 -> _anaconda_depends -> pytables -> mock
anaconda/linux-64::keras==2.2.4=0 -> tensorflow -> mock[version='>=2.0.0']
defaults/linux-64::_anaconda_depends==2019.03=py37_0 -> pytables -> mock
defaults/linux-64::mock==3.0.5=py37_0
The data is for a "NW" and "SE" quarters. Would be possible to have all the data?
Hello,
some files in the model files are incomplete/corrupted. These anomalies are also present in the original files. So unfortunately we can not do anything.
You can detect these anomalies by checking the file sizes. For a given file category (zone, level, model type), the files must have exactly the same size. So if some files are smaller than the other one, then these files have anomalies.
This has been reported in kaggle in this discussion
https://www.kaggle.com/katerpillar/meteonet/discussion/157206
Several users have encountered issues when installing the library Eccodes, used to open GRIB files with xarray. It seems that this library can create conflicts with other packages.
Consequently, we made the decision to stop using the GRIB format in the dataset and to convert the GRIB files to the netCDF format which is more common and should pose less intallation issues.
This conversion concerns the land-sea and relief masks and 2D and 3D meteo model outputs.
We will try to convert the files as fast as possible, but meanwhile if you encountered issues with Eccodes, you can still try working on the other parts of the dataset : ground station observations and radar data.
We're sorry for the inconvenience.
I would suggest you to move all the datasets to other more structured file format (NetCDF for example) instead of having csv, npz, etc.
Meanwhile, for the data that you currently have, two recommendations:
NW2016.csv.gz
directly instead of taring NW_ground_stations_2016.tar.gz
). That would allow users to work directly with compressed data (with tools like zcat) and download only what they need (multiple files case).I couldn't find the information in the documentation, so: do the radar rainfall coordinates correspond to the center of the pixels, or one of the corners?
Hi ! I would like to use the meteonet database and toolbox, however I can't download the toolbox, the page is not found when I go to the link. Do you have another way for downloading it ?
Thanks in advance !
Hi meteonet teams,
Really good job and initiative, thank you.
For now, the groundstations folder only seems to contain the data for the SE of France.
Are you planning on releasing the dataset for the NW data as well?
Thanks in advance,
G.
Is it possible to have in meteonet doc more info about parameters available here https://mf-models-on-aws.org/#arpege-europe/v2/ ? For some parameters like Radiation Flux, Heat Flux, I can't find a relevant doc.
Cannot install basemap with anaconda on windows, conflict with cfgrib.
(base) C:\Users\perezl>conda install -c anaconda basemap
Solving environment: failed
UnsatisfiableError: The following specifications were found to be in conflict:
basemap
cfgrib
Use "conda info " to see the dependencies for each package.
DatasetBuildError: multiple values for unique key, try re-open the file with one of:
filter_by_keys={'stepType': 1}
filter_by_keys={'stepType': 0}
Hello,
we notice for each line of a .csv file has TWO CR and one LR (Carriage Return and Line Return).
On Linux you should have only one LR, and on Windows you have one CR and one LR.
This extra CR at the end of each csv line does not mean anything.
We think this comes from our preprocessing task which is based on plain text file. We could have forgotten to delete some characters when extracting and converting to csv.
So, we note this as an issue and will try to fix it.
Hello MeteoNet team,
Thanks for the datasets already existing, great job!
I wanted to know if you plan to release also data for South West and North East part of France, especially for the ground stations data.
Thanks in advance,
We started working with meteonet data for weather forecast, then wanted to buy more archived data from Météo France to cover our use cases. We now encounter some errors when reading the files as detailed in your documentation:
import xarray as xr
data = xr.open_dataset('E:\Download\T47648_AROME0025.2017010100.grb', engine='cfgrib', backend_kwargs={"indexpath" : ""})
data
Here are the errors:
skipping variable: paramId==228228 shortName='tp'
Traceback (most recent call last):
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
dict_merge(variables, coord_vars)
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=Variable(dimensions=('step',), data=array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.,
13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
39., 40., 41., 42.])) new_value=Variable(dimensions=('step',), data=array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13.,
14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.,
40., 41., 42.]))
skipping variable: paramId==228164 shortName='tcc'
Traceback (most recent call last):
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
dict_merge(variables, coord_vars)
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=Variable(dimensions=('step',), data=array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.,
13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
39., 40., 41., 42.])) new_value=Variable(dimensions=('step',), data=array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13.,
14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.,
40., 41., 42.]))
skipping variable: paramId==3099 shortName='p3099'
Traceback (most recent call last):
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
dict_merge(variables, coord_vars)
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=Variable(dimensions=('step',), data=array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.,
13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
39., 40., 41., 42.])) new_value=Variable(dimensions=('step',), data=array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13.,
14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.,
40., 41., 42.]))
skipping variable: paramId==167 shortName='t2m'
Traceback (most recent call last):
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
dict_merge(variables, coord_vars)
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='heightAboveGround' value=Variable(dimensions=(), data=10.0) new_value=Variable(dimensions=(), data=2.0)
skipping variable: paramId==157 shortName='r'
Traceback (most recent call last):
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
dict_merge(variables, coord_vars)
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='heightAboveGround' value=Variable(dimensions=(), data=10.0) new_value=Variable(dimensions=(), data=2.0)
skipping variable: paramId==0 shortName='unknown'
Traceback (most recent call last):
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 653, in build_dataset_components
dict_merge(variables, coord_vars)
File "D:\Applications\Anaconda3\envs\tf-gpu\lib\site-packages\cfgrib\dataset.py", line 584, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=Variable(dimensions=('step',), data=array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.,
13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.,
26., 27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38.,
39., 40., 41., 42.])) new_value=Variable(dimensions=('step',), data=array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13.,
14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
27., 28., 29., 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.,
40., 41., 42.]))
We tried this workaround without success. Any help on this is welcome, thanks.
Hello,
I've noticed that some ground stations have multiple positions (lat, lon) for a single station ID.
As example, in NW2016.csv
for number_sta
22005003 you have entries for (48.550,-3.388)
and (48.551,-3.388)
. This is what should be? Or a station must have associated only one lat & lon?
To get a comple list of stations with multiple entries for lat & lon in a file, you can execute this:
$ cut -d, -f1,2,3 NW2016.csv | sort -u | awk -F , '
!count[$1]++ {save[$1] = $0; next}
count[$1] == 2 {
print save[$1]
delete save[$1]
}
{print}'
17218001,46.315,-1.000
17218001,46.320,-1.010
22005003,48.550,-3.388
22005003,48.551,-3.388
22016001,48.857,-3.005
22016001,48.860,-3.000
22247002,48.740,-3.250
22247002,48.742,-3.251
22261002,48.519,-2.420
22261002,48.520,-2.420
22266001,48.227,-3.295
22266001,48.230,-3.310
27422001,49.220,1.280
27422001,49.224,1.280
28002001,48.190,1.850
28002001,48.191,1.847
28041001,48.300,1.270
28041001,48.303,1.271
28190001,48.081,1.666
28190001,48.090,1.670
28215001,48.329,1.817
28215001,48.330,1.820
28252002,48.250,0.990
28252002,48.253,0.989
28406001,48.196,1.680
28406001,48.200,1.680
28407001,48.258,0.910
28407001,48.260,0.910
29263002,48.228,-4.098
29263002,48.230,-4.100
29276001,48.660,-4.077
29276001,48.660,-4.080
29293001,47.840,-3.870
29293001,47.841,-3.869
37023002,47.177,1.212
37023002,47.180,1.210
37216002,47.160,0.600
37216002,47.170,0.630
49008001,47.560,-0.970
49008001,47.562,-0.970
49191001,47.250,-0.440
49191001,47.252,-0.436
50111001,49.036,-1.298
50111001,49.040,-1.300
50215002,49.108,-1.607
50215002,49.110,-1.610
50514001,48.725,-0.887
50514001,48.759,-0.843
56127001,48.070,-2.280
56127001,48.070,-2.290
56243001,47.600,-2.710
56243001,47.605,-2.714
62054001,50.870,1.580
62054001,50.870,1.583
78140001,48.960,1.920
78140001,48.964,1.925
78152001,48.840,1.980
78152001,48.860,2.000
78465001,48.860,1.690
78465001,48.860,1.694
79079013,46.924,-0.755
79079013,46.929,-0.758
79253002,46.457,-0.205
79253002,46.460,-0.200
79309001,46.617,-0.571
79309001,46.620,-0.570
79326004,46.725,-0.020
79326004,46.730,-0.020
79329001,46.982,-0.215
79329001,46.990,-0.210
85092004,46.448,-0.780
85092004,46.450,-0.780
85163001,47.000,-2.260
85163001,47.005,-2.257
86092002,46.917,0.622
86092002,46.920,0.620
86096001,46.850,0.290
86096001,46.852,0.287
86272002,46.837,0.458
86272002,46.840,0.450
We wanted to ask a question in Slack Workspace “https://tinyurl.com/meteonet-slack”, but it couldn't be accessed.
https://meteofrance.github.io/meteonet/english/data/rain-radar/
We want to ask what is the height of the radar data in the old product. Is it radar echo data representing a fixed altitude? For example, 850hPa and 600hPa. Or is it an accumulation of radar echo data from multiple altitudes?
Looking forward to your reply!
Bonjour,
Je suis la coordonnatrice nationale des données ouvertes du Service météorologique du Canada et à ce titre et au nom de mes collègues, je tiens à souligner votre initiative.
Cependant, je regrette que la documentation soit seulement en anglais. Est-il planifié de la rendre disponible en français ?
Au gouvernement du Canada, c'est obligatoire, nous devons produire toute documentation dans les 2 langues.
Voir par exemple la documentation publique de nos données ouvertes:
https://eccc-msc.github.io/open-data/
Merci et bravo encore!
Sandrine Edouard
Coordonnatrice nationale, Accès aux données ouvertes
National coordinator, Open Data Access
Applications en traitement de données environnementales -
Environmental Data Processing Applications
Centre météorologique canadien - Canadian meteorological centre (CMC)
Environnement et changement Climatique Canada - Environment and Climate Change Canada
http://www.ec.gc.ca/
Courriel/email: [email protected]
Téléphone/Phone: 514-421-4729
2121 Trans-Canada N. Suite 230
Dorval, Québec
CANADA H9P 1J3
Here is some precision about precipitation parameter from weather model data.
For now, it is about the total precipitation since the beginning of the run (in kg.m-2 which is equivalent to mm). It is planned in a future version to modify that parameter in order to have the total precipitation between two time steps instead.
Bonjour, petite question un peu bête les données datetime sont en heure UT ou temps "classique" (UT+1 ou +2 l'été) ? Pour savoir si je dois faire une correction en couplant ce jeu de données avec un autre en heure normale. Merci beaucoup !
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.