s2cloudless with sentinel L2A data about eo-learn HOT 12 CLOSED

Hi @buwuyou,

technically you could do that but you wouldn't get good cloud masks. AddCloudMaskTask in the background calls s2cloudless package for calculating cloud masks and that package uses a machine learning model for it. Because the machine learning model was trained on L1C data it cannot perform well on L2A data.

from eo-learn.

aha, do we expect an updated s2cloudless built on L2A data in the next several quarters?

from eo-learn.

No, unfortunately we won't have time to repeat the ML process for L2A any time in the near future.

But if you would be interested to train a ML model for detecting clouds from L2A data yourself, this blog post describes how we did it.

from eo-learn.

Still, many thanks for the valuable answers!

from eo-learn.

I am not sure it makes sense to train s2cloudless on L2A as L2A contains scene classification data, which are of reasonable quality (much better than what L1C data provides). It is therefore not certain that s2cloudless would be much better than that...
https://forum.sentinel-hub.com/t/l2a-scene-classification-for-sentinel-2/51

from eo-learn.

Thanks for the additional information! Can I conclude that s2 data can be prepared via two ways with a consideration of cloud masking as 1) S2 L1C + s2cloudless; 2) S2 L2A + AddSen2CorClassificationFeature('SCL', layer='BANDS-S2-L2A')? You may recommend the latter?

from eo-learn.

S2 L2A = S1L1C + sen2Cor.
S2 L2A includes atmospheric correction, which goes beyond cloud classification... In general L2A data should be of better quality (where available - since December 2018 globally, before only Europe). But there are some who might disagree and prefer to work with L1C.
Note that L2A data processed by ESA are available via Sentinel Hub as well.

from eo-learn.

@buwuyou Yes, this is what we also do. We use L2A data and for cloud masking it we combine sen2cor's scene classification (SCL) with s2cloudless' cloud mask. The latter is obtained on L1C data at 160m resolution (it's fast) and upscaled to 10m resolution.

from eo-learn.

@azupanc Could you give me an example of how the latter option you stated would look like?

from eo-learn.

@azupanc Could you give me an example of how the latter option you stated would look like?

@Arraystud This is now very easy to do, thanks to the cloud masking service.

In eo-learn use the SentinelHubInputTask for downloading the data, where you specify the L2A data source. If you check the available data bands in the documentation, you can see that you can download CLM and SCL simultaneously and merge them in some later processing step.

add_data_task = SentinelHubInputTask(
    data_source = DataSource.SENTINEL2_L2A, 
    bands_feature=(FeatureType.DATA, 'BANDS'),
    time_difference = timedelta(hours=2),
    resolution=10,
    additional_data = [
        (FeatureType.MASK, 'CLM'),
        (FeatureType.MASK, 'SCL'),
        (FeatureType.DATA, 'CLP'),
        (FeatureType.MASK, 'dataMask'),
    ]
)

The output is then:

EOPatch(
  data: {
    BANDS: numpy.ndarray(shape=(73, 321, 534, 12), dtype=float32)
    CLP: numpy.ndarray(shape=(73, 321, 534, 1), dtype=uint8)
    NDVI: numpy.ndarray(shape=(73, 321, 534, 1), dtype=float32)
  }
  mask: {
    CLM: numpy.ndarray(shape=(73, 321, 534, 1), dtype=uint8)
    SCL: numpy.ndarray(shape=(73, 321, 534, 1), dtype=uint8)
    VALID_DATA: numpy.ndarray(shape=(73, 321, 534, 1), dtype=bool)
    dataMask: numpy.ndarray(shape=(73, 321, 534, 1), dtype=uint8)
  }
  meta_info: {
    maxcc: 1.0
    service_type: 'processing'
    size_x: 534
    size_y: 321
    time_difference: datetime.timedelta(seconds=7200)
    time_interval: ('2019-01-01T00:00:00', '2019-12-31T23:59:59')
  }
  bbox: BBox(((-116.07124, 33.55431), (-116.01335, 33.58277)), crs=CRS('4326'))
  timestamp: [datetime.datetime(2019, 1, 1, 18, 34, 53), ..., datetime.datetime(2019, 12, 27, 18, 34, 54)], length=73
)

from eo-learn.

Thank you. And the combination of these two masks is done how? I can't find it on the documentation page.

from eo-learn.

Thank you. And the combination of these two masks is done how? I can't find it on the documentation page.

It's just something we used in our projects, so it's not something that we put into the docs. However, you could do the following:

1. Download eopatch with the above function
2. In python do the following

# mask of valid CLM values (0 = no cloud, 1 = cloud, 2 = no data)
valid_clm = eop.mask['CLM'] == 0

# mask of valid SCL values (4 = vegetation, 5 = baresoil, 6 = water)
valid_scl = (eop.mask['SCL'] == 4) | (eop.mask['SCL'] == 5) | (eop.mask['SCL'] == 6)

# merge valid masks together (i.e. only where both are valid)
valid_mask = valid_clm  & valid_scl

# add new mask to eopatch
eop.mask['VALID_MASK'] = valid_mask

I didn't run this code, probably should work :)

from eo-learn.