Comments (17)
aaronspring
commented on September 18, 2024
cml.version '0.7.4'
climetlab_s2s_ai_challenge.version '0.7.6'
from climetlab.
floriankrb
commented on September 18, 2024
This seems to be a concurency issue. Perhaps you updated climetlab or the plugin or the climetlab settings without restarting the kernel. Then restart the kernel.
Or would that be possible that another kernel is running using the cache ?
from climetlab.
aaronspring
commented on September 18, 2024
Or would that be possible that another kernel is running using the cache ?
Might be the case.
from climetlab.
aaronspring
commented on September 18, 2024
if I look over individual dates, it works fine.
from climetlab.
aaronspring
commented on September 18, 2024
Or would that be possible that another kernel is running using the cache ?
Might be the case.
As I done have internet connection on my compute node, I have a terminal open. I will try to see what happens when I only have one open.
from climetlab.
aaronspring
commented on September 18, 2024
Even when all data is cached and no other kernel is running, I still get this OperationalError: database is locked error when loading all dates at once.
>>> cml.load_dataset(f"s2s-ai-challenge-test-output-reference", parameter=['tp','t2m'], date=xr.cftime_range(start='20200102',freq='7D', periods=53).strftime('%Y%m%d').to_list()).to_xarray()
By downloading data from this dataset, you agree to the terms and conditions defined at https://apps.ecmwf.int/datasets/data/s2s/licence/. If you do not agree with such terms, do not download the data. This dataset has been dowloaded from IRIDL. By downloading this data you also agree to the terms and conditions defined at https://iridl.ldeo.columbia.edu.
0%| | 0/106 [00:04<?, ?it/s]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/datasets/__init__.py", line 220, in load_dataset
ds = klass(*args, **kwargs).mutate()
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab_s2s_ai_challenge/observations.py", line 96, in __init__
PreprocessedObservations.__init__(self, *args, dataset="test-output-reference", **kwargs)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/normalize.py", line 137, in inner
return func(**normalized)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab_s2s_ai_challenge/observations.py", line 75, in __init__
self.source = cml.load_source("url-pattern", PATTERN_OBS, request, merger=S2sMerger(engine="netcdf4"))
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/sources/__init__.py", line 133, in load_source
return source(name, *args, **kwargs).mutate()
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/sources/__init__.py", line 102, in __call__
source = klass(*args, **kwargs)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/sources/url_pattern.py", line 37, in __init__
sources = list(tqdm(futures, leave=True, total=len(urls)))
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/tqdm/std.py", line 1130, in __iter__
for obj in iterable:
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
yield fs.pop().result()
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/sources/url_pattern.py", line 31, in url_to_source
return Url(url)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/sources/url.py", line 51, in __init__
self.path = self.cache_file(url, extension=ext)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/sources/__init__.py", line 50, in cache_file
return cache_file(owner, *args, extension=extension)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/core/caching.py", line 296, in cache_file
if register_cache_file(path, owner, args):
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/climetlab/core/caching.py", line 221, in register_cache_file
(now, path),
sqlite3.OperationalError: database is locked
from climetlab.
floriankrb
commented on September 18, 2024
I think this has been fixed in the latest climetlab "main" branch. I will adapt the plugin to work with the new version of climetlab before investigating on this.
from climetlab.
aaronspring
commented on September 18, 2024
great. just let me know.
Tried first: pip install -U climetlab climetlab_s2s_ai_challenge and then again. but still get the locked error.
from climetlab.
floriankrb
commented on September 18, 2024
please try again with the latest climetlab 0.8.6 and plugin 0.8.0...
from climetlab.
aaronspring
commented on September 18, 2024
Threaded Downloading now works, but I get a merger error:
>>> cml.load_dataset(f"s2s-ai-challenge-test-output-reference", parameter='t2m', date=xr.cftime_range(start='20200102',freq='7D', periods=53).strftime('%Y%m%d').to_list()).to_xarray()
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 53/53 [00:01<00:00, 29.31it/s]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/aaron.spring/anaconda3/envs/climetlab/lib/python3.7/site-packages/climetlab/datasets/__init__.py", line 113, in to_xarray
return self.source.to_xarray(**kwargs)
File "/Users/aaron.spring/anaconda3/envs/climetlab/lib/python3.7/site-packages/climetlab/sources/multi.py", line 90, in to_xarray
return make_merger(self.merger, self.sources).to_xarray(**kwargs)
File "/Users/aaron.spring/anaconda3/envs/climetlab/lib/python3.7/site-packages/climetlab/mergers/__init__.py", line 204, in make_merger
raise ValueError(f"Unsupported merger {merger} ({type(merger)})")
ValueError: Unsupported merger <climetlab_s2s_ai_challenge.fields.S2sMerger object at 0x7fe88e296390> (<class 'climetlab_s2s_ai_challenge.fields.S2sMerger'>)climetlab 0.8.6 pypi_0 pypi
climetlab-s2s-ai-challenge 0.8.0 pypi_0 pypi
from climetlab.
aaronspring
commented on September 18, 2024
I dont get this error on binder.
from climetlab.
aaronspring
commented on September 18, 2024
I get that Unsupported merger on my macbook after installing from source:
Name Version Build Channel
climetlab 0.8.9 pypi_0 pypi
climetlab-s2s-ai-challenge 0.8.0 pypi_0 pypi
pip freeze:
aiobotocore @ file:///home/conda/feedstock_root/build_artifacts/aiobotocore_1615456214827/work
aiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1614280921556/work
aioitertools @ file:///home/conda/feedstock_root/build_artifacts/aioitertools_1605140085294/work
anyio==3.1.0
appdirs==1.4.4
appnope @ file:///Users/runner/miniforge3/conda-bld/appnope_1610094688183/work
argcomplete==1.12.3
argh==0.26.2
argon2-cffi @ file:///Users/runner/miniforge3/conda-bld/argon2-cffi_1610522600362/work
asciitree==0.3.3
async-generator==1.10
async-timeout==3.0.1
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1605083924122/work
Babel==2.9.1
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache==1.6.1
black==21.6b0
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1612213472466/work
bokeh @ file:///Users/runner/miniforge3/conda-bld/bokeh_1614595267908/work
botocore @ file:///home/conda/feedstock_root/build_artifacts/botocore_1610405652704/work
branca==0.3.1
brotlipy==0.7.0
cdsapi==0.5.1
certifi==2020.12.5
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1613413897768/work
cfgrib==0.9.8.5
cfgv==3.3.0
cftime==1.4.1
chardet @ file:///Users/runner/miniforge3/conda-bld/chardet_1610093485994/work
click==7.1.2
climetlab @ git+https://github.com/ecmwf/climetlab.git@8d655b4ac121a69e7244efe109c04d5e110cdf9e
climetlab-s2s-ai-challenge==0.8.0
cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1598400192773/work
colorama==0.4.4
cryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography_1615406281648/work
cycler==0.10.0
Cython==0.29.22
cytoolz==0.11.0
dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1614995065708/work
decorator==4.4.2
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
distlib==0.3.2
distributed @ file:///Users/runner/miniforge3/conda-bld/distributed_1615002843496/work
eccodes==1.1.0
ecmwf-api-client==1.6.1
ecmwflibs==0.1.2
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1605121927639/work/dist/entrypoints-0.3-py2.py3-none-any.whl
fasteners==0.14.1
filelock==3.0.12
folium==0.12.1
fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1614107550004/work
HeapDict==1.0.1
identify==2.2.10
idna==2.10
importlib-metadata==4.0.1
iniconfig==1.1.1
ipykernel @ file:///Users/runner/miniforge3/conda-bld/ipykernel_1613853195137/work/dist/ipykernel-5.5.0-py3-none-any.whl
ipython @ file:///Users/runner/miniforge3/conda-bld/ipython_1614393628205/work
ipython-genutils==0.2.0
jedi @ file:///Users/runner/miniforge3/conda-bld/jedi_1610146990595/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1612119311452/work
jmespath @ file:///home/conda/feedstock_root/build_artifacts/jmespath_1589369830981/work
json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1600692310011/work
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema_1614815863336/work
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1615693636836/work
jupyter-core @ file:///Users/runner/miniforge3/conda-bld/jupyter_core_1612125292413/work
jupyter-server==1.8.0
jupyterlab==3.0.16
jupyterlab-code-formatter==1.4.10
jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1601375948261/work
jupyterlab-server==2.6.0
kiwisolver @ file:///Users/runner/miniforge3/conda-bld/kiwisolver_1610099791311/work
locket==0.2.0
Magics==1.5.6
Markdown==3.3.4
MarkupSafe @ file:///Users/runner/miniforge3/conda-bld/markupsafe_1610127518065/work
matplotlib @ file:///Users/runner/miniforge3/conda-bld/matplotlib-suite_1611858681166/work
mistune @ file:///Users/runner/miniforge3/conda-bld/mistune_1610112898998/work
monotonic==1.5
msgpack @ file:///Users/runner/miniforge3/conda-bld/msgpack-python_1610121730520/work
multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1610319029348/work
mypy-extensions==0.4.3
nb-conda-kernels @ file:///Users/runner/miniforge3/conda-bld/nb_conda_kernels_1606762486839/work
nbclassic==0.3.1
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1614336084111/work
nbconvert @ file:///Users/runner/miniforge3/conda-bld/nbconvert_1605401854071/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1611005694671/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1605195931949/work
netCDF4==1.5.6
nodeenv==1.6.0
notebook @ file:///Users/runner/miniforge3/conda-bld/notebook_1610575486451/work
numcodecs @ file:///Users/runner/miniforge3/conda-bld/numcodecs_1611651684879/work
numpy @ file:///Users/runner/miniforge3/conda-bld/numpy_1612953042948/work
olefile @ file:///home/conda/feedstock_root/build_artifacts/olefile_1602866521163/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1612459636436/work
pandas==1.2.3
pandocfilters==1.4.2
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1607618318316/work
partd==1.1.0
pathspec==0.8.1
pathtools==0.1.2
pdbufr==0.8.2
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow @ file:///Users/runner/miniforge3/conda-bld/pillow_1615244005847/work
pipx==0.16.2.1
pluggy==0.13.1
pre-commit==2.13.0
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1605543085815/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1615486796966/work
psutil @ file:///Users/runner/miniforge3/conda-bld/psutil_1610127337489/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py==1.10.0
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1615243893546/work
pyodc==1.0.3
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1608055815057/work
pyparsing==2.4.7
pyrsistent @ file:///Users/runner/miniforge3/conda-bld/pyrsistent_1610146821108/work
PySocks @ file:///Users/runner/miniforge3/conda-bld/pysocks_1610291468418/work
pytest==6.2.4
python-dateutil==2.8.1
python-vlc==3.0.6109
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1612179539967/work
PyYAML==5.4.1
pyzmq @ file:///Users/runner/miniforge3/conda-bld/pyzmq_1614611727596/work
regex==2021.4.4
requests==2.25.1
s3fs @ file:///home/conda/feedstock_root/build_artifacts/s3fs_1608058532532/work
Send2Trash==1.5.0
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
sniffio==1.2.0
sortedcontainers @ file:///home/conda/feedstock_root/build_artifacts/sortedcontainers_1605110889605/work
tblib==1.6.0
terminado @ file:///Users/runner/miniforge3/conda-bld/terminado_1616072345354/work
testpath==0.4.4
toml==0.10.2
toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1600973991856/work
tornado @ file:///Users/runner/miniforge3/conda-bld/tornado_1610094698292/work
tqdm==4.59.0
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1602771532708/work
typed-ast==1.4.3
typing-extensions==3.10.0.0
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1615828766818/work
userpath==1.4.2
virtualenv==20.4.7
watchdog==0.9.0
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
webencodings==0.5.1
websocket-client==1.1.0
wrapt @ file:///Users/runner/miniforge3/conda-bld/wrapt_1610094931751/work
xarray==0.18.2
yarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1610354318691/work
zarr @ file:///home/conda/feedstock_root/build_artifacts/zarr_1606962441855/work
zict==2.0.0
zipp==3.4.1
from climetlab.
floriankrb
commented on September 18, 2024
The same error? that is strange. I cannot reproduce it. Any local change to the climetlab code that you checked out from git, maybe ?
from climetlab.
aaronspring
commented on September 18, 2024
I erased the env and installed everything new. However I don't need climetlab locally. I will see tomorrow how it works on the supercomputer
from climetlab.
aaronspring
commented on September 18, 2024
on the supercomputer, I now get requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
Using just one download thread seems a workaround for me.
import xarray as xr
import climetlab as cml
cml.settings.set('number-of-download-threads',1)
cml.load_dataset(f"s2s-ai-challenge-test-output-reference", parameter='t2m', date=xr.cftime_range(start='20200102',freq='7D', periods=53).strftime('%Y%m%d').to_list()).to_xarray()from climetlab.
aaronspring
commented on September 18, 2024
Please close, cannot reproduce on other machine.
from climetlab.
aaronspring
commented on September 18, 2024
@floriankrb issue can be closed
from climetlab.
Related Issues (20)
- Merge strategy/options HOT 3
- Feature preparation for weather data to use in AI/ML application HOT 7
- Loading CSV with Pandas in CML HOT 2
- ValueError: cannot include dtype 'M' in a buffer HOT 5
- add support for loading single-precision data from GRIB files HOT 1
- Pip not able to install climetlab (v0.11.31) in windows with Python 3.7 : HDF5 headers not found HOT 5
- install as many dependencies as possible with conda, then pip: a small example please ? HOT 3
- Load the dataset locally HOT 4
- Is climetlab still supported on Python 3.7 HOT 3
- Doing `pip install climetlab` does not install s3fs which is required to load zarr sources HOT 3
- climetlab should not require branca==0.3.1 HOT 5
- `load_source` does not work in my system with `'url'` data source HOT 5
- `to_pandas` warning HOT 2
- No module named 'climetlab.ml' when running cml.load_source HOT 3
- cml.plot_map doesn't show map HOT 5
- Request for Enhanced Documentation on Diverse Data Retrieval in ClimetLab HOT 1
- Changelog update
- Example of `plot_map` method in a climetlab dataset package
- Adding `cos_solar_zenith_angle_integrated` function from earthkit-meteo
- Review lat/lon iterator for rotated reduced Gaussian grids
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from climetlab.