chaimain / asgardpy Goto Github PK

🚀 The feature, motivation and pitch

No internal Gammapy functionality to easily produce the EBL-deabsorbed spectrum. It requires proper translation of the covariance matrix as well.
These objects may be distinct attributes of AsgardpyAnalysis

Alternatives

No response

Additional context

Follow Gammapy Issue #5218
No response

As in the pipeline, we are mainly adding some functions so as to use Gammapy functions more effectively for datasets with different formats, and we would like to make it as user-friendly as possible, we may try to make the structure of the pipeline like the one initiated here gammapy/gammapy#3852.
I will make a PR implementing such a change soon.

🚀 The feature, motivation and pitch

Explore all the possible options of asgardpy.base.reduction.get_exclusion_region_mask and asgardpy.data.target.apply_selection_mask_to_models with a distinct example notebook. This was discovered to be missing while helping @joshua-baxter

Alternatives

No response

Additional context

No response

The way to read and Sky Model information which includes Spectral, Spatial and Temporal Models is not complete. We should follow Gammapy examples as close as possible.

When reading from files with different formatting/nomenclature standards, the function to read them to Gammapy SkyModel objects, we should try to make it terse and efficient.

Pydantic v2 has brought up large changes from its earlier versions (See Migration guide).

Gammapy adapted to these changes in the PR 4750 and in Asgardpy it was done in #165 (some of the changes highlighted here).

There is no "complete" solution to all the input types used in either pipeline. Is it worth investigating the time and effort to make changes in Asgardpy?

So far, no breaking changes have been seen in Asgardpy's workflow.

🚀 The feature, motivation and pitch

The current Dataset3DGeneration only supports Fermi-LAT DL3 files (with a list of XML-based sky models). It should be expanded to support more DL3 file types for creating a 3D Dataset.

• Make a check to read gadf-dl3 type of DL3 files for 3D Dataset and have it as the default format -> #94

• Also, manage the option of Sky Models inputs for different 3D dataset types, and figure out how to preferentially use all the given models with the joint dataset -> #98

🚀 The feature, motivation and pitch

To serialize an exact mapping of Fermi XML Models into Gammapy-compliant Models objects, so as to avoid the unnecessary and incorrect contribution of a Models object to a Dataset.

Declutter the functions in asgardpy/data/target.py and maybe have tests to check the validity of such functions, so as to avoid hard-coding and probably re-define each unique Model parameter.

Alternatives

Maybe a wrapper of some external function or a package import may help to not "re-invent the wheel"

Additional context

No response

🚀 The feature, motivation and pitch

Update the dependency of Gammapy to v1.2.
Some of the things to probably update -

• Including systematic
• Use updated gammapy stats functions
• Time-based objects
• Storing of DL4+ data
• Updating metadata
• Update to astropy v6
• Update to pydantic v2+
• Update on the Estimator class
• Use general decorrelation energy for any spectral model -> Adapt the custom models in here

Alternatives

No response

Additional context

No response

The "null hypothesis" (=there is no gamma signal, just background) can play no role in the evaluation of the goodness of fit.
You are mixing up "goodness of fit" (how well the spectral model reproduces the data) with "statistical significance in favour of the model containing a gamma source, vs. the model containing only background" (a measure of how likely it is that the observed excess is just a background fluctuation). The two things are only loosely related.

🚀 The feature, motivation and pitch

Follow the following standards -

• PEP 518
• PyPA specs following PEP 621
• PEP 517
• PEP 660
• Use Ruff for linting

Alternatives

No response

Additional context

No response

🚀 The feature, motivation and pitch

There are some classes defined that repeat the work done by Gammapy or other user functions and are not required in this pipeline as of now.
These are:

• AnalysisStep light-curve-estimator : Not applicable for a joint-dataset
• AnalysisStep excess-map : Not necessary for the pipeline
• Class DL4Files in asgardpy.io.io : to write various Gammapy products to files
• Module asgardpy.utils : Collection of plot functions using Gammapy

Alternatives

No response

Additional context

No response

• First issue in that function from config/generator.py (l109):

final_copy = base_config.copy() -> there seems to be no copy() definition there. I bypassed this error by using copy(base_config), however, it fails at the next point:

File ~/Software/asgardpy/src/asgardpy/config/generator.py:109, in recursive_merge_dicts(base_config, extra_config)
    107 from copy import copy
    108 final_config = copy(base_config)
--> 109 for key, value in extra_config.items():
    110     if key in final_config and isinstance(final_config[key], list):
    111         new_config = []

AttributeError: 'AnalysisStepEnum' object has no attribute 'items'

Versions

python 3.11

🐛 Describe the bug

The type of list of dict for the TimeIntervalsType config is not appropriately validated when using AsgardpyConfig.update() function in the initialization of AsgardpyAnalysis with a distinct models_file, which changes the type of pydantic models consistency in the AsgardpyConfig object.

This was introduced in #142 and worked ok with the associate pytests.

Versions

Python 3.11.4
alabaster==0.7.13
annotated-types==0.5.0
anyio==3.7.1
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
-e git+https://github.com/chaimain/asgardpy.git@e91495b0697a17753e7b060d37226667bdd56145#egg=asgardpy
astropy==5.3.1
asttokens==2.2.1
async-lru==2.0.4
attrs==23.1.0
autodoc-pydantic==1.9.0
Babel==2.12.1
backcall==0.2.0
beautifulsoup4==4.12.2
black==23.7.0
bleach==6.0.0
build==1.0.3
certifi==2023.7.22
cffi==1.15.1
charset-normalizer==3.2.0
click==8.1.6
codespell==2.2.5
colorama==0.4.6
comm==0.1.4
contourpy==1.1.0
coverage==7.2.7
cryptography==41.0.3
cycler==0.11.0
debugpy==1.6.7
decorator==5.1.1
defusedxml==0.7.1
docutils==0.20.1
execnet==2.0.2
executing==1.2.0
fastjsonschema==2.18.0
flake8==6.1.0
fonttools==4.42.0
fqdn==1.5.1
furo==2023.7.26
gammapy==1.1
idna==3.4
imagesize==1.4.1
iminuit==2.23.0
importlib-metadata==6.8.0
iniconfig==2.0.0
ipykernel==6.25.0
ipython==8.14.0
ipython-genutils==0.2.0
ipywidgets==8.1.0
isoduration==20.11.0
isort==5.12.0
jaraco.classes==3.3.0
jedi==0.19.0
jeepney==0.8.0
Jinja2==3.1.2
json5==0.9.14
jsonpointer==2.4
jsonschema==4.18.6
jsonschema-specifications==2023.7.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.7.0
jupyter-lsp==2.2.0
jupyter_client==8.3.0
jupyter_core==5.3.1
jupyter_server==2.7.0
jupyter_server_terminals==0.4.4
jupyterlab==4.0.4
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.8
jupyterlab_server==2.24.0
keyring==24.2.0
kiwisolver==1.4.4
livereload==2.6.3
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.7.2
matplotlib-inline==0.1.6
mccabe==0.7.0
mdit-py-plugins==0.4.0
mdurl==0.1.2
mistune==3.0.1
more-itertools==10.1.0
mypy==1.4.1
mypy-extensions==1.0.0
myst-parser==2.0.0
nbclient==0.8.0
nbconvert==7.7.3
nbformat==5.9.2
nest-asyncio==1.5.7
notebook==7.0.2
notebook_shim==0.2.3
numpy==1.25.2
overrides==7.3.1
packaging==23.1
pandas==2.0.3
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.11.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==10.0.0
pkginfo==1.9.6
platformdirs==3.10.0
pluggy==1.2.0
prometheus-client==0.17.1
prompt-toolkit==3.0.39
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pycodestyle==2.11.0
pycparser==2.21
pydantic==1.10.12
pydantic-settings==2.0.2
pydantic_core==2.4.0
pyerfa==2.0.0.3
pyflakes==3.1.0
Pygments==2.15.1
pyparsing==3.0.9
pyproject_hooks==1.0.0
pytest==7.4.0
pytest-cov==4.1.0
pytest-sphinx==0.5.0
pytest-xdist==3.3.1
python-dateutil==2.8.2
python-dotenv==1.0.0
python-json-logger==2.0.7
pytz==2023.3
PyYAML==6.0.1
pyzmq==25.1.0
qtconsole==5.4.3
QtPy==2.3.1
readme-renderer==40.0
referencing==0.30.2
regions==0.7
requests==2.31.0
requests-toolbelt==1.0.0
rfc3339-validator==0.1.4
rfc3986==2.0.0
rfc3986-validator==0.1.1
rich==13.5.2
rpds-py==0.9.2
ruamel.yaml==0.17.32
ruamel.yaml.clib==0.2.7
ruff==0.0.287
scipy==1.11.1
seaborn==0.12.2
SecretStorage==3.3.3
Send2Trash==1.8.2
setuptools-scm==7.1.0
six==1.16.0
sniffio==1.3.0
snowballstemmer==2.2.0
soupsieve==2.4.1
Sphinx==7.1.2
sphinx-autobuild==2021.3.14
sphinx-autodoc-typehints==1.24.0
sphinx-basic-ng==1.0.0b2
sphinx-copybutton==0.5.2
sphinxcontrib-applehelp==1.0.4
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
stack-data==0.6.2
terminado==0.17.1
tinycss2==1.2.1
tornado==6.3.2
tqdm==4.66.1
traitlets==5.9.0
twine==4.0.2
typing_extensions==4.7.1
tzdata==2023.3
uri-template==1.3.0
urllib3==2.0.4
wcwidth==0.2.6
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.1
widgetsnbextension==4.0.8
xmltodict==0.13.0
zipp==3.16.2

🚀 The feature, motivation and pitch

Move the functions get_list_objects and update_aux_info_from_xml from asgardpy.data.dataset3d.Dataset3DGeneration to asgardpy.gammapy.interoperate_models or something similar, to make the reading from a Fermi XML models file into a Gammapy Models object independent of the core Asgardpy functionality.

Alternatives

No response

Additional context

This will allow Asgardpy to retain a bit more functionality when Gammapy v2.0 (or a later version) adapts the new HLI workflow and thus reduces the need for the intermediate Analysis steps incorporated in Asgardpy.

The Map Geometry section seems a bit clunky and maybe not generalized for all possibilities.

🚀 The feature, motivation and pitch

The base template uses a static versioning system, provided by the developer. A proper usage of dynamic versioning, involving the creation of developer builds, release builds and appropriate changelog maintenance should be done.

Alternatives

No response

Additional context

No response

🚀 The feature, motivation and pitch

• #118

• #117

We need Time interval information from the config file to estimate LC flux points of a given dataset and model (with or without performing the Fit function). An efficient way has to be put in place.

🚀 The feature, motivation and pitch

Currently AsgardpyConfig only has the option to update from a different AsgardpyConfig object by using pydantic.utils.deep_update. This restricts merging the values from the second AsgardpyConfig object only when they are different from those of default and those present in the first AsgardpyConfig.

This is seen in the case where we want to have several options for the assumed initial spectral model information (LP, ECPL, PL, etc), in terms of distinct config files, such that the main AsgardpyConfig object used for the analysis can have the path of such spectral model config files, be stored in AsgardpyConfig.target.models_file, without these distinct config files having target source information like source_name and redshift.

One can use gammapy.utils.scripts.recursive_merge_dicts or maybe check for more options from pydantic.utils itself.

Alternatives

No response

Additional context

No response

🚀 The feature, motivation and pitch

With the new pydantic version, there is a need to change a few things like renaming validate_all to validate_default and replacing json_encoders with the new custom serializer. Check for other migration changes required.

Alternatives

Or constrain pydantic version to 1.9.0

Additional context

Check the error message from the docs build https://readthedocs.org/projects/asgardpy/builds/21176158/ and https://docs.pydantic.dev/latest/migration/ for support in this matter.

Properly define the use of logging information and log the main messages.

There is no standard data format or file structure yet in the VHE Astronomy field. In cases for instruments like Fermi-LAT, there are more than one standard file structures (enrico, fermipy), with no clear way of interoperability in reading the files.

Currently, support for files produced with Enrico for LAT data and GADF (standard for gammapy) compliant LST-1 DL3 files is implemented in the classes Dataset3DGeneration, Dataset1DGeneration and DL3Files. It is probably imperative to have distinct classes or set of functions for distinct file structures, which the above classes call upon to continue reading them to Gammapy-readable formats.

• Use tox or nox.
• Update tests with proper and detailed assertions.
• Maybe redistribute (all combined) tests to run on different OS and/or Python versions and thereby use pre-commit hooks