mapai-competition's People
Forkers
haukebartsch magedhelmy1 olastalberg atelier-eo jorgej17 sainivedh19pt sfsy1 morteg15 michelegazzea gisgfk zjiangsan tutlabrok denissebastian mahelsem bloodaxe adeelh simenbvikane navjordj andebraa fastfirefly llei66 andreaseike awadelrahman jediofgever ndavid asdspal dondondooooon hvl-ml beytullahsarica yunusserhat lm239 audunborgersen alu042 dvolgyes nora-norwegian-ai-research-consortium aranguren nordic-machine-intelligencemapai-competition's Issues
Metrics are flawed on negative samples
Both the metrics for IoU and BIoU are incapable of evaluating samples with no buildings because True Negatives are not accounted for. This also leads to very unstable metrics for samples with almost exclusively background.
For a sample with no buildings, any prediction, regardless of its accuracy, will give a score of 0. This happens because the metrics are only calculated for the positive class, but not the negative class.
Since True Negatives and False Positives are important measures of performance, I highly encourage you to change how you evaluate predictions. I hope you see the urgency here; a competition with flawed metrics is not fair and can't be taken seriously.
Please implement class-wise metrics following Keras' MeanIOU metric. Its documentation states:
Note that this class first computes IoUs for all individual classes, then returns the mean of these values.
Additionally (in a class-wise implementation), both IoU and BIoU should ignore a class (or default to 1) if its target has no positive examples.
cache directory bug
The user home directories are actually not a good place to store data.
This is configurable in the huggingface interface with something like this:
import datasets
datasets.load_dataset("sjyhne/mapai_training_data", cache_dir="/mnt/experiment-3/huggingface")
However, it seems there is a bug in the mapai_training_data.py, and it can't properly handle nondefault directory.
I would suggest to add an optional cache_dir to the create_dataset
function which is propagated
into the load_dataset, and also fixing the bug in the huggingface datasets.
It is hackable, but especially on long term, after the competition, it would be nice to have a conformant dataset
which could be used in the future easily.
Reproduction:
- execute the above lines
Possible cause:
- very likely in the
_split_genetators
function the current working dir is not the one which is assumed,
therefore the os.makedirs refer to wrong location.
The evaluation pipeline will time out after 6 hours
There is a 6 hour limit for jobs in Github Actions. Since the main step in evaluate.yml iterates through every submission it is practically guaranteed that it will time out before the majority of submissions have been tested.
Error when running main.py task 2
Hello, I am getting an error when running main.py with the following command:
"python main.py --device=cuda --task=2 --submission='submission/' --data-type=validation"
The error is:
Traceback (most recent call last):
File "C:\Users<user>\MapAI-Competition-Private\team_template\src\main.py", line 31, in
evaluate_model_2(args=args)
File "C:\Users<user>\MapAI-Competition-Private\team_template\src\model_task_2.py", line 107, in main
biou_score = biou(label, prediction)
File "C:\Users<user>\MapAI-Competition-Private\venv\lib\site-packages\competition_toolkit\eval_functions.py", line 76, in biou
gt_boundary = _mask_to_boundary(target.astype(np.uint8), dilation_ratio)
File "C:\Users<user>\MapAI-Competition-Private\venv\lib\site-packages\competition_toolkit\eval_functions.py", line 43, in _mask_to_boundary
h, w = mask.shape
ValueError: too many values to unpack (expected 2)
The shape of mask is [4,500,500], and changes based on the batch size and image size. I am using the model as provided in the example in "model_task_2.py", should not this work "out-of-the-box"?
editing files outside team folder
It would be wise to remind people not to commit changes outside their team folder,
especially not in the github workflows or in the competition toolkit...
Are the submission figures mandatory?
Just out of curiosity, are the .png figures from plt a mandatory part of the submission in the submission path?
Main.py task 3 bug
The default task in main.py, task 3 which runs tasks 1+ 2 deletes the task 1 predictions before continuing to task 2. See the following code which is executed both at the start of task 1 and 2 with opts["task"] = 3.
task_path = pathlib.Path(args.submission_path).joinpath(f"task_{opts['task']}")
opts_file = task_path.joinpath("opts.yaml")
predictions_path = task_path.joinpath("predictions")
if task_path.exists():
shutil.rmtree(task_path.absolute())
As far as I know task 3 is never used, so maybe it is not worth it to update it. Or perhaps you can wait until the competition is over?
BIoU is flawed on all samples
Please change
def biou(gt, dt, dilation_ratio=0.02)
to
def biou(dt, gt, dilation_ratio=0.02)
The following code does not work as intended, otherwise:
target = class_wise(gt, c)
if not np.any(target):
continue
Is external data allowed
Hi,
Are we allowed to use data from xView2 / Spacenet challenges as additional training datasets?
Thanks, Eugene
Can we add folders to .gitignore
We have generated some additional data in our submission which for convenience has been stored in the data folder in MapAI-Competition/. However this does not work well with the current .gitignore. Is it permitted to add additional folders under data/ in the gitignore file?
What is the actual meaning of Lidar data ?
Hi,
I am trying to train a network for Lidar. There is confusion about Lidar data and I would appreciate if you can shed some light here.
Typically Lidar data is presented as pointclouds, with formats such as .pcd, .ply, .xyz
etc. Sometimes it is also presented in .bin
formats to save space. The Lidar data seems to be present in .tif
file.
Is this file including a projection of Lidar data (e.g. Bird-Eye View) or does it include raw X,Y,Z distance measurements? please provide minimal code on how to extract the Lidar and get the actual point cloud in case this is indeed raw pointcloud data.
The provided function to load lidar data returns an 2d array with 1 channel
l = load_lidar(lidarpath=lidar_paths[2],size=(500,500))
print(l.shape)
>> torch.Size([500, 500])
Be aware of gdown error
Hey, in my local machine gdown sometimes complains with the following error;
Access denied with the following error:
Cannot retrieve the public link of the file. You may need to change
the permission to 'Anyone with the link', or have had many accesses.
You may still be able to access the file from the browser:
You can get past this error, see the wkentaro/gdown#43 (comment)
Task 1 LIDAR rules
Does the prohibition against LIDAR in task 1 cover all steps of the task, or are we within the rules as long as we do not use lidar during inference for task 1?
Different metrics in evaluation.py and model_task_*.py
We are in the process of verifying the evaluation pipeline, and have noticed that the pipeline produces two sets of scores. The first set of IoU and BIoU is made in the model_task_*.py files which uses resized masks, labels and lidar (their size is opts["imagesize"] * opts["imagesize"]).
The second pair of metrics is made by evaluate.py which uses the default 500 * 500 masks with the predictions found in the submissions folder (consequently, predictions must be resized in model_task_*.py if opts["imagesize"] is not 500).
These sets of metrics are necessarily different since both the predictions and masks differ from resizing. Therefore, I want to know which is considered the "True" score in the competition.
Am I right in guessing that the metrics obtained in the model_task_*.py files are not used in the competition? If so, can we just remove the computation of the metrics from these files altogether, since they only complicate our implementation?
More generally, is there anything we can not change in model_task_*.py, as long as we store the plotted prediction, and resized prediction in the submissions folder?
non-open source submission
Hi,
We are a small company, and we would like to participate, to some degree, let me explain.
So if I understand it correctly, the training and inference codes should be submitted
with the merge request along with a binary model (pytorch, onnx, something) which
can make predictions, and this would be used to evaluate the hidden test set.
We have the following issue:
- we would be happy to submit technical description
- and trained onnx or pytorch model,
But we have quite a large training infrastructure.
This includes hyperparameter search, training codes, etc.
This code is proprietary, and we cannot share it.
So my questions are:
- is it possible to submit a onnx or similar model which is trained but not trainable?
Our goal is not to take the prize, but to compare our capabilities to the other submissions.
If we get good enough, we would be happy to present our approach,
but without open sourcing our training infrastructure.
In other competitions, it is frequently the case that the non-open source
submissions are not eligible for prizes, etc., but submission is allowed,
and shown on a leaderboard, etc.
so I would like to get some clarification on what would be the case here.
submission deadline
Hi,
I see the another deadline extension. I recommend to add a specific hour with time zone in order to avoid further issues.
E.g. December 5, 4pm Oslo time zone, or 23:59 Oslo time zone, or similar, in order to avoid further misunderstandings.
When will the test dataset be published?
Is the test data published alongside the test results or will there some delay after the 12th before it is released?
And out of curiosity, is the evaluation going according to the schedule?
Evaluation pipeline exit code 137 (not enough memory)
We were testing our current model with the evaluation pipeline in our private repository, and encountered an error with exit code 137, meaning we ran out of CPU memory. Do you know if this will be an issue in the competition pipeline, or does it have more memory available than the standard?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.