The two forward model have option name CONFIG_AVGMAP and CONFIG_HCMAP, while they could have been simplified to CONFIG, also in-line with the command line option when called from command line.

Currently, generate_maps (of grid3d_aggregate_maps.py), can ignore cells with properties below a certain threshold. The primary purpose of this is speed-up. However, this also affects the result: areas with no cell properties above this threshold is handled in the same manner as areas outside the extent of the grid. This can be confusing and also leads to issues when doing surface-surface operations with xtgeo (differences maps or statistics).

The default behavior should be the same for this function as it is for the functions under avghc (to the extent reasonable). One option could be to add an optional fill value to be used for undefined values, and perform this as a post-processing step before returning results in the generate_maps function. This should be relatively easy to implement, however it has the limitation that this value will also be used outside the grid extents.

https://github.com/equinor/fmu-docs/actions/runs/9364831055/job/25780692153?pr=120

When building fmu-docs, these warnings and errors were encountered:

/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/readme.rst:1: CRITICAL: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: 'README.rst'.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets... copying static files... done
copying extra files... done
done
writing output... [  9%] authors
writing output... [ 18%] config-avg
writing output... [ 27%] config-hc
writing output... [ 36%] contributing
writing output... [ 45%] examples
writing output... [ 55%] history
writing output... [ 64%] howitworks
writing output... [ 73%] index
writing output... [ 82%] installation
writing output... [ 91%] readme
writing output... [100%] usage
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated

/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
generating indices... genindex done
highlighting module code... 
/home/runner/work/fmu-docs/fmu-docs/grid3d-maps/docs/index.rst:6: WARNING: toctree contains reference to document 'readme' that doesn't have a title: no link will be generated
writing additional pages... search done

Latest build initiated at fmu-docs can be viewed at https://test-fmu-docs.azurewebsites.net/

example: https://github.com/equinor/xtgeoapp-grd3dmaps/blob/18300022056c204adce820ec9092d4e4e57224be/src/xtgeoapp_grd3dmaps/avghc/grid3d_hc_thickness.py#L30-L34

Places to update:

• FMU Docs
• Atlas docs
• FURE docs
• Drogon docs (comments in configs)
• Komodo
• GitHub workflows

• prefix should possibly change all to something else
• tag is extra information, possibly only for zones

In some cases, Eclipse keywords in e.g. UNRST contains spaces. This has been found when using E100 and tracers, where keywords with names like W8 F make occur.

This makes average maps to crash. It seems that the engine behind, xtgeo, can handle reading keywords with spaces, but the xtgeo scanning of properties seems to parse keyword with spaces in a wrong manner.

See cross-lib issue https://github.com/equinor/atlas/issues/36 and corresponding xtgeo issue equinor/xtgeo#1189.

Update actions to most recent ones:
checkout@vN —> checkout@v4, setup-python@vN —> setup-python@v5 and codecov-action@vN —> codecov-action@v4

Attempt at reconstruction (taken from an example, not the real user config):

input:
  eclroot: ../xtgeo-testdata/3dgrids/reek/REEK
  grid: $eclroot.EGRID

  # alternative approach to read a variables and dates:
  properties:
    -
      name: PORO
      source: $eclroot.INIT
    -
      name: SWAT
      source: $eclroot.UNRST
      dates: !include_from tests/yaml/global_config3a.yml::global.DATES
      diffdates: !include_from tests/yaml/global_config3a.yml::global.DIFFDATES
    -
#      name: PRESSURE
#      source: $eclroot.UNRST
#      dates: !include_from tests/yaml/global_config3a.yml::global.DATES
#      diffdates: !include_from tests/yaml/global_config3a.yml::global.DIFFDATES

The dangling dash - causes the error TypeError: argument of type 'NoneType' is not iterable from line 242 in _configparser.py which does if 'name' not in prop

Maps are often generated on exported RMS-data, hence the ERT forward model argument "--eclroot" <ECLROOT> should be optional.

The output files here are currently not with metadata. A demand now is to output with rich metadata, and driven
by fmu-dataio.

https://github.com/equinor/fmu-dataio

See examples in subscript, rms-sys, etc.

With numpy bumped from 1.23 to 1.24, some numbers change making the tests fail:

Example failure:

    def test_aggregated_map2(datatree):
        result = datatree / "aggregate2_folder"
        result.mkdir(parents=True)
        cfg = "tests/yaml/aggregate2.yml"
    
        grid3d_aggregate_map.main(
            [
                "--config",
                cfg,
                "--mapfolder",
                str(result),
                "--plotfolder",
                str(result),
            ]
        )
        swat = xtgeo.surface_from_file(result / "all--min_swat--20030101.gri")
>       assert swat.values.mean() == pytest.approx(0.7908786104444353, abs=1e-8)
E       assert 0.7309275812612058 == 0.7908786104444353 ± 1.0e-08
E         comparison failed
E         Obtained: 0.7309275812612058
E         Expected: 0.7908786104444353 ± 1.0e-08

All dicts are ordered in recent Python versions

E.g. is NTG loaded when method=use_porv in computesetting and PORV loaded when method=use_poro. Should be enough to have the keywords described for the methods in https://sdp.equinor.com/wikidocs/XTGeo/apps/xtgeoapp_grd3dmaps/html/config-hc.html

https://github.com/equinor/xtgeoapp-grd3dmaps/blob/18300022056c204adce820ec9092d4e4e57224be/src/xtgeoapp_grd3dmaps/avghc/_get_grid_props.py#L55-L83

grid3d_hc_thickness and grid3d_average_map both support filtering on dates and/or zones in the config file:

input:
  properties:
    - name: PORO
       dates: !include_from file/path/to/dates.yml

zonation:
  zproperty:
    source: path/to/zonefile.roff
    zones:
      - zone1: [1]
      - zone2: [2]

(the config file is incomplete, but should hopefully illustrate the issue)

This is not supported by grid3d_aggregate_map and grid3d_migration_time. Implementing this should help control the amount of data that is produced, which can be quite significant for large projects.

The benefits here are numerous.

Related: #26

The scripts "grid3d_average_map" and "grid_hc_thickness" could be hooked into the ERT plugin system, ala this PR: equinor/subscript#112

This would make it possible to remove custom jobs from ERT template projects.

See also https://github.com/equinor/fmu-dev/issues/4

HC plots (png) ssems to to pick up other than default color map

This does not seem to be currently done. With black, flake8, isort, etc.

Black is currently failing, let's move to ruff as with other repositories.

NumPy 2 will arrive Dec/Jan and NumPy have recommended pinning and testing against its release candidate(s) before using. Other Equinor packages are taking this action as well.

As described in dataio issue: equinor/fmu-dataio#607

we are passing unit=None into dataio.ExportData. This gives a pydantic validation error "input needs to be a valid string".

We should change the code from this:

and add a test that uses a dataset where the unit is not given

This package uses some functionality set to be deprecated with xtgeo 4.0.

grid3d_maps/avghc/_get_grid_props.py:399: DeprecatedWarning: values3d is deprecated as of 3.6 and will be removed in 4.0. Use gridprop.values instead
    xhcpv = ma.filled(initobjects[0].values3d, fill_value=0.0)

grid3d_maps/avghc/_get_grid_props.py:456: DeprecatedWarning: values3d is deprecated as of 3.6 and will be removed in 4.0. Use gridprop.values instead
    sgas[date] = ma.filled(prop.values3d, fill_value=1)

grid3d_maps/avghc/_get_grid_props.py:451: DeprecatedWarning: values3d is deprecated as of 3.6 and will be removed in 4.0. Use gridprop.values instead
    swat[date] = ma.filled(prop.values3d, fill_value=1)

...

This is a non-exhaustive list and other can be found in the test run output. These things can be fixed prior to xtgeo 4.0 being released.

Safer loading of yaml files

Documentation is built for FMU Docs. We could publish them on GitHub pages too, and then have fmu-docs point to them rather than build them.

It seems that file names can be overwritten if 2 different average runs have two different grids (e.g. geogrid and simgrid)

Is should also be possible to add user defined tags

As reported by a user, the example given for GRID3D_AGGREGATE_MAP shows

FORWARD_MODEL GRID3D_AGGREGATE_MAP(<CONFIG_AGGMAP>=conf.yml, <ECLROOT>=<ECLBASE>)

while the forward model job takes

ARGLIST "--config"  <CONFIG_AGGREGATE> "--eclroot" <ECLROOT>

Note that CONFIG_AGGREGATE != CONFIG_AGGMAP.

Both of these packages are co-existing in Komodo and ert is preferring xtgeapp-grd3dmaps. Enough time has passed that we should reasonably be able to remove xtgeoapp-grd3dmaps from future Komodo versions, i.e., it will not cause trouble if it is removed and new Komodo patch releases must be made.

lowercase under output in the configuration file should result in output files in all lower case letters. This is currently not supported by the two mentioned actions.

ERT/libres is pip-installable, and should be included in CI for running test_ert_hooks.py

On large datasets the generation of maps takes an extremely long time.

Ran generation of avg maps on the Snorre field took 8 hours to produce close to 500 maps.

Introducing async and possibly multiprocessing should cut that time significantly

Make this repository available on pypi

• Registration on pypi
• Make a github actions job that will publish on pypi when a release is made (see e.g. fmu-config)

Tests test_average_map2b() and test_average_map2c() fails. The fails occur with newer xtgeo versions >= 2.9. The exact xtgeo commit where this happens is equinor/xtgeo@77b292a

Investions suggests that results are almost identical for most maps, and the tests mentioned above are rather special. So propose to just adjust the two tests.

For hc thickess, and diffs in particular it may be that one date is found but not the other. This is difficult to trace / debug som more information and possible exceptions would be beneficial.

In accordance with new governance guidance.

https://github.com/equinor/xtgeoapp-grd3dmaps/blob/4a0709e7bbdaac8f22c9e9a79a836af64d74f6c3/src/xtgeoapp_grd3dmaps/hook_implementations/jobs.py#L3C11-L3C11

This shows up when starting ERT with komodo bleeding:

(bleeding-py38) [havb@st-linrgsn110:~/test_field_update_using_ertbox/ert/model]$ ert gui example.ert
/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/xtgeoapp_grd3dmaps/hook_implementations/jobs.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import resource_filename

Tests sometimes fail because of this.

When using several diffdates for e.g. an average property, only first and last will output. Seems like a bug

Hydrocarbon thickness computations only support three-phase, assuming both SGAS and SWAT present.

If a user tries grid3d_hc_thickness.py on a two-phase (oil-water) simulation, it ends with an error message that SGAS is not found.

The scenario is that one wants to diff between history runs and pred runs.

Xtgeoapp enables generation of average maps from 3D grids, but not other types of statistics (e.g. min/max-type of maps). This is useful in particular for CCS applications where the maximum CO2 saturation is of interest to avoid leaks into undesired parts of the reservoir. It is also useful for generating time migration maps, which requires a minimum-type of aggregation.

The idea here is to extend xtgeoapp with this functionality, possibly invoked via a new script, but based on a similar configuration file structure. For instance, a keyword could be added to the "computesettings" section of the configuration file, used to determine the type of aggregation ("mean", "min" or "max").

CCS simulations can often involve large grids to cover potential leakage over large areas. This means performance is important. Being able to ignore cells with insignificant amounts of CO2 has large potential for performance improvement as major portions of the grid will often have zero CO2.

The current approach of xtgeoapp uses a dz-weighting to ignore cells that should be excluded from the average calculation. However, this will not improve performance since the multiplication by 0 is still executed. This can be avoided, but will require significant changes to the existing functionality, including changes to the xtgeo repo. Instead, it is suggested to implement this as a completely separate submodule (alongside avghc), but making sure to create a user interface (i.e. Script arguments and configuration file structure) that is familiar to the two scripts grid3d_average_map and grid3d_hc_thickness.

The tests write to TMP in repository's root directory. This should be moved to `/tmp/'.

CONFIG_HCMAP should be CONFIG_AVGMAP in this line. That this has passed CI is a bug in itself.

https://github.com/equinor/xtgeoapp-grd3dmaps/blob/7a54d9b32b913ebe31fe16b225be3e0d4d5b8119/tests/test_ert_hooks.py#L39

These arguments can be removed per their deprecation in fmu-dataio.

This can be grouped with #118 as a generic fmu-dataio update. Note that there may possible be other deprecated things from dataio (or not).

When using fmu.dataio, the files comes with syntax:

topsome--tagname--20230101_19990912.xxx

i.e. monitor - base

Some assets use reverse dates for historical reason, and it should be possible to this here also