Distribution Statement A. Approved for public release. Distribution unlimited.

Author: Naval Research Laboratory, Marine Meteorology Division

This program is free software: you can redistribute it and/or modify it under the terms of the NRLMMD License included with this program. If you did not receive the license, see https://github.com/U-S-NRL-Marine-Meteorology-Division/ for more information.

This program is distributed WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the included license for more details.

xNRL helps you read NRL NWP output into xarray Datasets nested within Pandas DataFrames.

Install

xNRL requires numpy, pandas, and xarray.

Install with pip

pip install xnrl

Clone and install with pip

git clone https://github.com/U-S-NRL-Marine-Meteorology-Division/xnrl.git
cd xnrl
pip install -e .

The -e makes the code editable.

You can update the code with
cd xnrl
git pull origin main

Install with Conda

Copy the environment_xnrl.yml file and create the Conda environment.

conda env create -f environment_xnrl.yml

Note: This only installs the current main branch and does not let you edit the code. You can update xnrl (and all packages in the environment) with
conda env update -f environment_xnrl.yml

Examples

import xnrl

# Load COAMPS flatfiles into an xarray Dataset
fp = '<path>/*pre*00120*'
ds = xnrl.open_dataset(fp, model='COAMPS')

pytest error

Hi @ahuang11

I'm working to get your pytest running again and I'm running into this issue:

The test in question:

@pytest.mark.parametrize("model", [C.NAVGEM, C.COAMPS, C.GFS])
@pytest.mark.parametrize(
    "label", ["basic", "sigma", "sigma_no_datahd", "coords", "no_latlons"]
)
@pytest.mark.parametrize("chunks", [True, False])
@pytest.mark.parametrize("only_meta", [None, True, "list"])
@pytest.mark.parametrize("temporal_dim", ["tau", "time"])
def test_open_dataset_flatfile(model, label, chunks, only_meta, temporal_dim):
    file_type = C.FLATFILE
    paths = _get_paths(model, file_type, label)
    kwds = dict(chunks=chunks, only_meta=only_meta, temporal_dim=temporal_dim)
    df = _baseline_test(paths, model, file_type, label, **kwds)

    if only_meta:
        pytest.skip("only_meta")

    ds = df["ds"][0]
    meta_df = xnrl.open_dataset(paths, only_meta="list")
    for i, row in meta_df.iterrows():
        cols = ["ini", "tau", "lev"]
        if "sig_w" in ds.coords or "lev" not in ds.dims:
            cols.remove("lev")
        if temporal_dim == "time":
            cols = ["time" if col == "tau" else col for col in cols]
        da = ds.sel(row[cols])[row["field"]].squeeze()
        offset = 8 if model == C.GFS else 0
        ex = np.fromfile(row["path"], offset=offset, dtype=">f")
        ac = da.values.ravel()
        assert np.allclose(ex, ac)

Output:

(xnrl-py3.10) [hutchins@archer tests]$ pytest test_main.py::test_open_dataset_flatfile
================================ test session starts =================================
platform linux -- Python 3.10.9, pytest-7.4.0, pluggy-1.2.0
rootdir: /users/hutchins/git/xnrl
collected 180 items                                                                  

test_main.py FFFssssssssssssFFFsssssssssssssssssssssssssssssssssssssssssssssss [ 36%]
sssssssssssssssssssssssssFFFssssssssssssFFFsssssssssssssssssssssssssssssssssss [ 79%]
sssssssssssssssssssssssssssssssssssss                                          [100%]

====================================== FAILURES ======================================
_______________ test_open_dataset_flatfile[tau-None-True-basic-navgem] _______________

model = 'navgem', label = 'basic', chunks = True, only_meta = None
temporal_dim = 'tau'

    @pytest.mark.parametrize("model", [C.NAVGEM, C.COAMPS, C.GFS])
    @pytest.mark.parametrize(
        "label", ["basic", "sigma", "sigma_no_datahd", "coords", "no_latlons"]
    )
    @pytest.mark.parametrize("chunks", [True, False])
    @pytest.mark.parametrize("only_meta", [None, True, "list"])
    @pytest.mark.parametrize("temporal_dim", ["tau", "time"])
    def test_open_dataset_flatfile(model, label, chunks, only_meta, temporal_dim):
        file_type = C.FLATFILE
        paths = _get_paths(model, file_type, label)
        kwds = dict(chunks=chunks, only_meta=only_meta, temporal_dim=temporal_dim)
        df = _baseline_test(paths, model, file_type, label, **kwds)
    
        if only_meta:
            pytest.skip("only_meta")
    
        ds = df["ds"][0]
        meta_df = xnrl.open_dataset(paths, only_meta="list")
        for i, row in meta_df.iterrows():
            cols = ["ini", "tau", "lev"]
            if "sig_w" in ds.coords or "lev" not in ds.dims:
                cols.remove("lev")
            if temporal_dim == "time":
                cols = ["time" if col == "tau" else col for col in cols]
>           da = ds.sel(row[cols])[row["field"]].squeeze()

test_main.py:191: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../.venv/lib/python3.10/site-packages/xarray/core/dataset.py:2793: in sel
    indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "sel")
../.venv/lib/python3.10/site-packages/xarray/core/utils.py:275: in either_dict_or_kwargs
    if pos_kwargs is None or pos_kwargs == {}:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ini    False
tau    False
lev    False
Name: 0, dtype: bool

    @final
    def __nonzero__(self) -> NoReturn:
>       raise ValueError(
            f"The truth value of a {type(self).__name__} is ambiguous. "
            "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
        )
E       ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

../.venv/lib/python3.10/site-packages/pandas/core/generic.py:1466: ValueError
--------------------------------- Captured log call ----------------------------------
DEBUG    xnrl:main.py:378 Searching for paths to open using ['/users/hutchins/git/xnrl/xnrl/../tests/data/navgem/flatfile/basic/airtmp_pre_0850.0_0000.0_glob360x181_2017020812_00000000_fcstfld', '/users/hutchins/git/xnrl/xnrl/../tests/data/navgem/flatfile/basic/airtmp_pre_0850.0_0000.0_glob360x181_2017020812_00240000_fcstfld', '/users/hutchins/git/xnrl/xnrl/../tests/data/navgem/flatfile/basic/geopht_pre_1000.0_0000.0_glob360x181_2017020112_14400000_fcstfld', '/users/hutchins/git/xnrl/xnrl/../tests/data/navgem/flatfile/basic/geopht_pre_1000.0_0000.0_glob360x181_2017020812_00000000_fcstfld']...
DEBUG    xnrl:main.py:387 Found 4 paths to open!
DEBUG    xnrl:main.py:158 
########################## NAVGEMFlatFile ##########################
DEBUG    xnrl:internal.py:910 Split paths successfully!
DEBUG    xnrl:internal.py:912 Extracted meta successfully!
DEBUG    xnrl:internal.py:914 Selected meta successfully!
DEBUG    xnrl:internal.py:930 Validated hypercube for pre glob360x181 successfully!
DEBUG    xnrl:internal.py:935 Created dataset for pre glob360x181 successfully!
DEBUG    xnrl:internal.py:942 Populated metadata for pre glob360x181 successfully!
DEBUG    xnrl:internal.py:947 Shifted lons for pre glob360x181 successfully!
DEBUG    xnrl:internal.py:952 Sub-selected for pre glob360x181 successfully!
DEBUG    xnrl:internal.py:974 Created row for pre glob360x181 successfully!
DEBUG    xnrl:main.py:392 xnrl finished successfully in 0:00:00.155923!
WARNING  xnrl:main.py:177 Model was not specified so iterating through available ones! Specify model for a more optimized run!

It's complaining about this line (below) which I do not know what it's doing:

da = ds.sel(row[cols])[row["field"]].squeeze()

Also of note is that I've switched to pyproject.toml using poetry.

u-s-nrl-marine-meteorology-division / xnrl Goto Github PK

xnrl's Introduction

Install

Install with pip

Clone and install with pip

Install with Conda

Examples

xnrl's People

Contributors

Stargazers

Watchers

Forkers

xnrl's Issues

Recommend Projects

Recommend Topics

Recommend Org