dhi-gras / atmcorr Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 2.0 10.83 MB

Atmospheric correction with Py6S and MODIS data

License: MIT License

Python 100.00%

atmosphere modis remote-sensing

atmcorr's People

Contributors

Stargazers

Watchers

Forkers

jieluoybfq ylaya

atmcorr's Issues

processing with tileSize>0 fails on Windows

Setting the tileSize parameter in the preprocessing resulted in the following error message (input was a Sen2 image)

bathymetry production options_chain_preprocessing_v07.yaml
unknown+93442f60efde999ab6f515bf2490b2e60762630f

  INFO     | Reading input data
  INFO     | Start pre-processing
  INFO     | Starting atmospheric correction ...
Getting 6S params:   0%|                                                                             | 0/605 [00:00<?, ?job/s]b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'

b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'






Getting 6S params:   0%|                                                                   | 1/605 [00:07<1:14:32,  7.41s/job]bGetting 6S params:   8%|█████▏                                                              | 46/605 [00:08<01:38,  5.67job/s]b'The system cannot execute the specified program.\r\n'

b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'







b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
  ERROR    | Uncaught exception
Traceback (most recent call last):
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\bathymetry\scripts\cli.py", line 66, in main
    cli(obj=obj)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\bathymetry\scripts\production.py", line 24, in production
    main.run_chain(options, processing_backend=backend)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\bathymetry\chain\main.py", line 109, in run_chain
    deepwater_vectorfile=o_chain.get('deepWaterShapeFile', None)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\bathymetry\chain\preprocess.py", line 56, in main
    refl_data_bandfirst, profile = atmcorr_main(**o_atmcorr)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\atmcorr\processing.py", line 266, in main
    for tilecp, adjcoef, idx in pbar(executor.map(wrap_6S.run_sixs_job, jobgen)):
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\tqdm\_tqdm.py", line 979, in __iter__
    for obj in iterable:
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\concurrent\futures\_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\concurrent\futures\thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\atmcorr\wrap_6S.py", line 141, in run_sixs_job
    mysixs.run()
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\Py6S\sixs.py", line 315, in run
    self.outputs = Outputs(outputs[0], outputs[1])
  File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\Py6S\outputs.py", line 64, in __init__
    raise OutputParsingError("6S returned an error (shown above) - check for invalid parameter inputs")
Py6S.sixs_exceptions.OutputParsingError: 6S returned an error (shown above) - check for invalid parameter inputs

Review and suppress NaN-related RuntimeWarnings

e.g.

atmcorr\wrap_6S.py:331: RuntimeWarning: invalid value encountered in less
  refl[refl < 0.0] = 0.0

atmcorr returns negative reflectance values when deglint is set to false

For some S2 images, atmcorr can return negative reflectance values for some bands (so far seen in the red band) when deglint is set to false.

Example can be found here \\DKCPH1-NCR3612\d$\Velux_DK_HabitatMapping\neg_refl

Water gets too dark in S2

Reported by @milyra.

The problem seems to be worse in winter than in summer. This could hint at a problem with the Sun angles.

Improve robustness when dealing with MODIS atmParameters and medium resolution images (S2 / L8)

When looking at overlapping S2 tiles acquired at the same time, it becomes clear that the atmospheric correction is not robust.
As a fix, we should look at atmospheric parameters for all of the swath at once, and then explode the parameters down to the correct scale. This will allow all the tiles to fail together, or pass together, instead of cases where partial results are retrieved and some of the MODIS downloads fail. In turn, we will get a consistent product across tile-borders.
We should also consider implementing a sanity check on the MODIS files. Especially we should look at the values retrieved in cloudy areas, to see how MODIS deals with these, and if we should filter them away.

This feature is lower priority than establishing a operational product chain for SEGES / NAER.

Potential issue with dg_calibration vs dg-calibration

Some places it is called dg_calibration (like in the requirements in setup.py and /atmcorr/metadata/worldview.py) but the package is called https://github.com/DHI-GRAS/dg-calibration
But in that package, there is a folder called /tree/master/dg_calibration

I'm confused. Does Python not care about the difference between - and _ ?

Install warnings when using new py and conda

(bathytest) C:\Users\mlra>pip install C:\Users\mlra\Desktop\atmcorr-master.zip
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: C:\Users\mlra\AppData\Local\Continuum\miniconda3\envs\bathytest\Include\UNKNOWN
sysconfig: C:\Users\mlra\AppData\Local\Continuum\miniconda3\envs\bathytest\Include
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Processing c:\users\mlra\desktop\atmcorr-master.zip
Requirement already satisfied: dg_calibration in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (0.4)
Requirement already satisfied: sensor_response_curves in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (unknown-657f78e2129a2eb294ca538b7d515d72edfadc31)
Requirement already satisfied: satmeta in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.3.0)
Requirement already satisfied: shapely in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.7.1)
Requirement already satisfied: python-dateutil in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (2.8.1)
Requirement already satisfied: lxml in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (4.6.3)
Requirement already satisfied: affine in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (2.3.0)
Requirement already satisfied: six>=1.5 in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from python-dateutil->satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.15.0)
Requirement already satisfied: scipy in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from sensor_response_curves->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.6.2)
Requirement already satisfied: numpy in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from sensor_response_curves->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.20.2)
Building wheels for collected packages: atmcorr
  Building wheel for atmcorr (setup.py) ... done
  Created wheel for atmcorr: filename=atmcorr-unknown_62f606e077c4539c4bf868d13bf3a00fabb4c428-py3-none-any.whl size=29575 sha256=38d4c8d5b82181d2c1976bc7efc420533854b029ab603ef2fa98072d5a1e659b
  Stored in directory: c:\users\mlra\appdata\local\pip\cache\wheels\6c\8a\ef\0fd0c53fd63871b78026fe3c66ef8aac4257dd43363a4cc346
  WARNING: Built wheel for atmcorr is invalid: Metadata 1.2 mandates PEP 440 version, but 'unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428' is not
Failed to build atmcorr
Installing collected packages: atmcorr
    Running setup.py install for atmcorr ... done
  DEPRECATION: atmcorr was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: C:\Users\mlra\AppData\Local\Continuum\miniconda3\envs\bathytest\Include\UNKNOWN
sysconfig: C:\Users\mlra\AppData\Local\Continuum\miniconda3\envs\bathytest\Include
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Successfully installed atmcorr-unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428

tileSize>0 does not help to get same values in Sentinel 2 tile overlap

When using tiling to get finer-grained viewing angles, the results do not mach for two neighboring (and overlapping) Sentinel 2 tiles from the same overpass.

Adjacency Correction run even though adjCorr = 0?

https://github.com/DHI-GRAS/Bathymetry/issues/23

use packad py6s insted of having it in dependecyes

A new package for py6s is now avalable in anaconda
https://anaconda.org/pypi/py6s

This makes it possiable to use the command below
Insted pip install -i https://pypi.anaconda.org/pypi/simple py6s

insted of having a version of py6s laying the the atmospheric correction folder

atmcorr sets 0 to 'no data' when it shouldn't

In images where the initial 'no data' value is 0, if dark waters reach 0 as a result of the atmospheric correction, the value gets set to 'no data', which then causes problems when the results from the atmcorr is sent further through the chain.
Can we possibly keep 0 and 'no data' separate? Meaning, if there were a value before, it shouldn't by accident end up as 'no data' when just using atmcorr.

Review and update WorldView calibration factors

We need to make the retrieval and selection of calibration factors for WorldView 2 and 3 more robust.

It also appears that there are new values out: https://apollomapping.com/image_downloads/DigitalGlobe_Absolute_Radiometric_Calibration_All_Satellites_2106.pdf

Should we update to those?

Addition of Landsat 4 & 5 to the list of possible sensors.

Having the option of using 6S on Landsat 4 & 5 data would be good for historical studies.
Some sort of choice has to be made for the old data, where MODIS atmospheric parameters are unavailable.