Investigate fail in FluxPointsEstimator about gammapy HOT 15 CLOSED

@AtreyeeS A first side comment: it seems the cutout for the Gaussian model becomes too small. It seems to be at just 2x2 pixels...I am not sure what happens after PSF convolution with the geometry...

from gammapy.

Thanks @AtreyeeS for investigating. I think the answer here is not obvious, because often there is a strong correlation between e.g. the amplitude and size of a model. So re-optimizing spatial parameters can make sense. however if the overall fit is unstable, things might break. The advantage of the current solution is that users can explicitly freeze the spatial parameters beforehand. This will set the model in the same state when re-optimizing for the flux point computation. If we always fix spatial parameters it is not possible to unfreeze, except if we introduce an option for this.

But I agree the most common method is to just re-optimize the norms of nearby sources. So we could show this more often in the docs:

models.parameters.freeze_all()
models.parameters.norm_parameters.unfreeze_all()

from gammapy.

This creates an issue for the upper limit computations actually. Since the fitting is done per bin, the calculated upper limits may not even correspond to the same source (eg: different sizes/positions in different bins!)

I propose that along with improving the tutorials and the docstring, we add some information in the user guide about what a flux point estimation is supposed to mean, why spectral parameters must be frozen, and the implications of freezing spatial parameters

from gammapy.

I must say I can't see a good reason why one would re-optimize the source parameters during flux estimation. Yet I can see many bad reasons :) .

I think the default should never be this behavior. It even confuses us.

from gammapy.

@QRemy @adonath do you understand this error?

from gammapy.

Does it happen also for 1.01 ? if not it could be related to #4304.

from gammapy.

@QRemy : The fail happens in 1.0.1 as well

from gammapy.

The spatial parameters should be frozen during the flux estimation, no? Why is the Gaussian width getting affected?

from gammapy.

Investigating this, the scaled model works only on the spectral parameters. We only re-set the spectral part of the scale model here

The spatial parameters remain free, and are re-optimised. Is this the correct behaviour @registerrier @bkhelifi ? The spatial position can go completely haywire, and the flux point extracted for a different region?

eg:

from gammapy.modeling.models import *
m1 = SkyModel(spectral_model=PowerLawSpectralModel(),
              spatial_model=PointSpatialModel(),
              name = "m1")
m2 = SkyModel(spectral_model=BrokenPowerLawSpectralModel(),
              spatial_model=GaussianSpatialModel(),
              name = "m2")
models = Models([m1, m2])
scale_m2 = ScaleSpectralModel(m2)
models[m2].spectral_model = scale_m2
models.to_parameters_table()["model", "type", "name", "frozen"]).show_in_browser()

clearly shows

model	type	name	frozen
m1	spectral	index	False
m1	spectral	amplitude	False
m1	spectral	reference	True
m1	spatial	lon_0	False
m1	spatial	lat_0	False
m2	spectral	norm	False
m2	spatial	lon_0	False
m2	spatial	lat_0	False
m2	spatial	sigma	False
m2	spatial	e	True
m2	spatial	phi	True

from gammapy.

And also precise it in the docstring of the estimator!
Indeed, for each estimator, a user can not scan all tutorials to understand a usage ....

from gammapy.

I think the spectral parameters of other sources are also kept free, no? For fitting in a single bin, that cannot be correct.

from gammapy.

If we really want to keep the option open, maybe reoptimize can take a string from among [None, "norms", "all"]

from gammapy.

We need to decide on an approach here:

• should impose all the source of interest paramters frozen during flux point estimation as per PR #4567
• or define a configurable approach where optimize can take more than a boolean
• Or simply improve the documentation so that all tutorials freeze the source parameters before fp estimation
  Opinions @AtreyeeS @adonath @QRemy ?

from gammapy.

We already have a sizeable user base now, and I am not sure how many of them will go back and check a fp estimation tutorial unless specifically informed about an issue there. And the current behaviour is rather erroneous (specially for weak sources), and rectifying it seems to solve the problem of "Gammapy gives very low upper limits, inconsistent with previous measurements" (I don't know if it was reported officially..)

I would at least keep a configure approach to optimize, AND improve the tutorials to show how to freeze parameters explicitly. Also, detail the behaviour in the forward folding section of the estimator docs (https://docs.gammapy.org/1.1/user-guide/estimators.html#general-method)

from gammapy.

Solved through #4567

from gammapy.