sjyhne / mapai-competition Goto Github PK

Both the metrics for IoU and BIoU are incapable of evaluating samples with no buildings because True Negatives are not accounted for. This also leads to very unstable metrics for samples with almost exclusively background.

For a sample with no buildings, any prediction, regardless of its accuracy, will give a score of 0. This happens because the metrics are only calculated for the positive class, but not the negative class.

Since True Negatives and False Positives are important measures of performance, I highly encourage you to change how you evaluate predictions. I hope you see the urgency here; a competition with flawed metrics is not fair and can't be taken seriously.

Please implement class-wise metrics following Keras' MeanIOU metric. Its documentation states:

Note that this class first computes IoUs for all individual classes, then returns the mean of these values.

Additionally (in a class-wise implementation), both IoU and BIoU should ignore a class (or default to 1) if its target has no positive examples.

The user home directories are actually not a good place to store data.
This is configurable in the huggingface interface with something like this:

import datasets
datasets.load_dataset("sjyhne/mapai_training_data", cache_dir="/mnt/experiment-3/huggingface")

However, it seems there is a bug in the mapai_training_data.py, and it can't properly handle nondefault directory.
I would suggest to add an optional cache_dir to the create_dataset function which is propagated
into the load_dataset, and also fixing the bug in the huggingface datasets.

It is hackable, but especially on long term, after the competition, it would be nice to have a conformant dataset
which could be used in the future easily.

Reproduction:

• execute the above lines

Possible cause:

• very likely in the _split_genetators function the current working dir is not the one which is assumed,
  therefore the os.makedirs refer to wrong location.

There is a 6 hour limit for jobs in Github Actions. Since the main step in evaluate.yml iterates through every submission it is practically guaranteed that it will time out before the majority of submissions have been tested.

Hello, I am getting an error when running main.py with the following command:

"python main.py --device=cuda --task=2 --submission='submission/' --data-type=validation"

The error is:
Traceback (most recent call last):
File "C:\Users<user>\MapAI-Competition-Private\team_template\src\main.py", line 31, in
evaluate_model_2(args=args)
File "C:\Users<user>\MapAI-Competition-Private\team_template\src\model_task_2.py", line 107, in main
biou_score = biou(label, prediction)
File "C:\Users<user>\MapAI-Competition-Private\venv\lib\site-packages\competition_toolkit\eval_functions.py", line 76, in biou
gt_boundary = _mask_to_boundary(target.astype(np.uint8), dilation_ratio)
File "C:\Users<user>\MapAI-Competition-Private\venv\lib\site-packages\competition_toolkit\eval_functions.py", line 43, in _mask_to_boundary
h, w = mask.shape
ValueError: too many values to unpack (expected 2)

The shape of mask is [4,500,500], and changes based on the batch size and image size. I am using the model as provided in the example in "model_task_2.py", should not this work "out-of-the-box"?

It would be wise to remind people not to commit changes outside their team folder,
especially not in the github workflows or in the competition toolkit...

Just out of curiosity, are the .png figures from plt a mandatory part of the submission in the submission path?

The default task in main.py, task 3 which runs tasks 1+ 2 deletes the task 1 predictions before continuing to task 2. See the following code which is executed both at the start of task 1 and 2 with opts["task"] = 3.

task_path = pathlib.Path(args.submission_path).joinpath(f"task_{opts['task']}")
opts_file = task_path.joinpath("opts.yaml")
predictions_path = task_path.joinpath("predictions")
if task_path.exists():
    shutil.rmtree(task_path.absolute())

As far as I know task 3 is never used, so maybe it is not worth it to update it. Or perhaps you can wait until the competition is over?

Please change

def biou(gt, dt, dilation_ratio=0.02)

to
def biou(dt, gt, dilation_ratio=0.02)

The following code does not work as intended, otherwise:

target = class_wise(gt, c)
if not np.any(target):
    continue

Hi,

Are we allowed to use data from xView2 / Spacenet challenges as additional training datasets?

Thanks, Eugene

We have generated some additional data in our submission which for convenience has been stored in the data folder in MapAI-Competition/. However this does not work well with the current .gitignore. Is it permitted to add additional folders under data/ in the gitignore file?

Hi,
I am trying to train a network for Lidar. There is confusion about Lidar data and I would appreciate if you can shed some light here.

Typically Lidar data is presented as pointclouds, with formats such as .pcd, .ply, .xyz etc. Sometimes it is also presented in .bin formats to save space. The Lidar data seems to be present in .tif file.

Is this file including a projection of Lidar data (e.g. Bird-Eye View) or does it include raw X,Y,Z distance measurements? please provide minimal code on how to extract the Lidar and get the actual point cloud in case this is indeed raw pointcloud data.

The provided function to load lidar data returns an 2d array with 1 channel

l = load_lidar(lidarpath=lidar_paths[2],size=(500,500))
print(l.shape)
>> torch.Size([500, 500])

Hey, in my local machine gdown sometimes complains with the following error;

Access denied with the following error:

        Cannot retrieve the public link of the file. You may need to change
        the permission to 'Anyone with the link', or have had many accesses. 

You may still be able to access the file from the browser:

You can get past this error, see the wkentaro/gdown#43 (comment)

Does the prohibition against LIDAR in task 1 cover all steps of the task, or are we within the rules as long as we do not use lidar during inference for task 1?

We are in the process of verifying the evaluation pipeline, and have noticed that the pipeline produces two sets of scores. The first set of IoU and BIoU is made in the model_task_*.py files which uses resized masks, labels and lidar (their size is opts["imagesize"] * opts["imagesize"]).

The second pair of metrics is made by evaluate.py which uses the default 500 * 500 masks with the predictions found in the submissions folder (consequently, predictions must be resized in model_task_*.py if opts["imagesize"] is not 500).

These sets of metrics are necessarily different since both the predictions and masks differ from resizing. Therefore, I want to know which is considered the "True" score in the competition.

Am I right in guessing that the metrics obtained in the model_task_*.py files are not used in the competition? If so, can we just remove the computation of the metrics from these files altogether, since they only complicate our implementation?

More generally, is there anything we can not change in model_task_*.py, as long as we store the plotted prediction, and resized prediction in the submissions folder?

Hi,

We are a small company, and we would like to participate, to some degree, let me explain.
So if I understand it correctly, the training and inference codes should be submitted
with the merge request along with a binary model (pytorch, onnx, something) which
can make predictions, and this would be used to evaluate the hidden test set.

We have the following issue:

• we would be happy to submit technical description
• and trained onnx or pytorch model,

But we have quite a large training infrastructure.
This includes hyperparameter search, training codes, etc.
This code is proprietary, and we cannot share it.

So my questions are:

• is it possible to submit a onnx or similar model which is trained but not trainable?

Our goal is not to take the prize, but to compare our capabilities to the other submissions.
If we get good enough, we would be happy to present our approach,
but without open sourcing our training infrastructure.

In other competitions, it is frequently the case that the non-open source
submissions are not eligible for prizes, etc., but submission is allowed,
and shown on a leaderboard, etc.
so I would like to get some clarification on what would be the case here.

Hi,

I see the another deadline extension. I recommend to add a specific hour with time zone in order to avoid further issues.
E.g. December 5, 4pm Oslo time zone, or 23:59 Oslo time zone, or similar, in order to avoid further misunderstandings.

Is the test data published alongside the test results or will there some delay after the 12th before it is released?

And out of curiosity, is the evaluation going according to the schedule?

We were testing our current model with the evaluation pipeline in our private repository, and encountered an error with exit code 137, meaning we ran out of CPU memory. Do you know if this will be an issue in the competition pipeline, or does it have more memory available than the standard?