Official implementation of the Fully Convolutional Instance-aware Semantic Segmentation (FCIS) based on MXNet -> they use hickle to dump masks fast. It is wotrh checking if it is useful to us.

hickle in their code.

Modify U-Net:

• encoding part is pre-trained ResNet or VGG,
• decoding part as usual -> How to make horizontal connections?
  train entire network

Idea described in this blog post -> DSB'18 winning solution

Models are performing badly on small images

DSTL competition:

• Discussions during DSTL
• confluence (ds.ai internal)
• 1st place solution by Kyle Lee blog post
• 3rd place solution video
• 4th place solution -> ds.ai blog post

• Add epoch number to the saved model name, that is model.torch.
• Save model i.e. model_epoch123.model
• remove previously saved model (with smaller epoch number).

Currently there are not enough columns in metadata file. Fix it.

handle kernel size, strides, upsampling.

General requirements:

• generate metadata file
• in main.py option to prepare meta and masks (for train)

For the training purposes:

• prepare masks overlayed

Note, that masks for evaluation will be prepared on-the-fly in the loader from single json.

Right now each steps.pytorch.Model instance returns a dictionary of lists of outputs from the network. It causes problems when working with larger datasets. I think it should return a generator instead

• take only masks with objects smaller than 32^2
• check if we can make reasonable result on small buildings only
• if successful train small-buildings-specialist

Using random rotation in augmentation may cause trouble because:

• if pictures are taken at the same time of the day (presumable of the short period of the year) shadow will be more or less the same
• orientation of buildings/roads may be more or less the same throughout the dataset

Actions:

• go through train/valid/test examples and check if constant angle hypothesis is correct
• drop rotation from augmentations or use only a very small rotation

Instance labeler works only for one class (buildings).
In case of more classes it wouldn't work.

It is needed to adjust NeptuneMonitorSegmentation class in callbacks.py to the output format of unet.

Evaluation is currently not possible on the entire validation set.
There is however option to generate predictions in chunks.
Similar option in evaluation would help

regarding single masks for evaluation purposes:

• load json to memory and convert to mask on-the-fly.

Currently function overlay_masks_from_annotations in preparation.py assumes single class case. It has to be modified and adapted to multiclass case. Some tuning of loaders may be necessary.

Loader has a method .dataset() where one passes objects like:

• self.transform
• self.transform_with_target

etc.
That is clumsy. Since those methods are available via self. why pass them?

using mosaic padding with stream_mode=1 causes errors at the score evaluation stage.

Current loaders do not support multiple classes just one class with multiple outputs

Class instances that are close are connected

We may be training on too large images.
Go through:

• https://medium.com/mlreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807
• https://arxiv.org/abs/1701.04128

Decide whether we are training on correct size images:

• Mayby smaller crops are enough in which case faster training is possible

• erode 30% of the mask
• train on eroded mask
• no dilation

Loss params like:

• BCE weight
• DICE weight
• distance weighted loss params
• size weighted loss params
• weight schedule

need to be investigated

Pillow can be replaced by a faster Pillow-SIMD with little to no effort.
I think it is worth trying https://github.com/uploadcare/pillow-simd

• calculate only for threshold = 0.5

Funky augmentation where small buildings are added randomly as long as they are not touching any real image

We have to find a good way to save nn.DataParallel models and later load them to transformers.

• Currently normalization constants mean=0 std=1 are hard-coded, they should be calculated on train and passed to loaders
• Substitute mean and std with values from pretrained PyTorch models (i.e. ResNet).

It seems that in the pipeline_config.py:

MEAN = [0., 0., 0.]
STD = [1., 1., 1.]

But the pytorch pretrained models have:

transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]))

I think this may cause suboptimal results

Current submission generator assumes that predictions is a list of single-channel images (one image per test image), where pixels with values 1,2,3... correspond to building instances 1,2,3...

New submission generator has to be adjusted to handle output from MulticlassLabeler (that will be prepared in issue #23

One of the options to speed-up data loaders is to:

• transform your images and target masks into one large HDF5 file
• refactor/add pytorch Datasets that read from the HDF5 file

Now it is just torch.ones(). It should handle class imbalance.

• modify preparation.py
• modify loaders.py

Build training that implements this:

• erode masks (so all masks were separated, they didn’t touch),
• train U-Net with weighted cross entropy loss function. Idea described in this kaggle post. While training add more weight in loss function to points that we would assign to contours touching category,
• make predictions,
• dilate predictions.

Weights calculated as in the U-Net paper (page 5, eq. 2).

U-Net performs poorly on edges. Effect described in this blog post.

Idea is to crop central part of the image during predicting. Of course it must be followed by sliding-window predictions with reflection padding of the original image.

Implement mosaic-padding-based inference to tackle the troublesome edge regions in unets

Add morphological pre/post processing to deal with connected instances.

Using DSB-18 code