This repository contains code to reproduce the experiments in the preprint "MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning"
Home Page: https://vishalned.github.io/mmearth/
License: Other
I followed INSTALL.MD and it mentioned that MinkowskiEngine only needs to be installed for pre-training. However, when running ./slurm_scripts/slurm_fine_tune_seg.sh, I got the following error:
Traceback (most recent call last): File "/opt/conda/envs/rsfms/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/rsfms/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ubuntu/RSFMs/main_finetune.py", line 34, in <module> import models.convnextv2 as convnextv2 File "/home/ubuntu/RSFMs/models/convnextv2.py", line 12, in <module> from .utils import LayerNorm, GRN File "/home/ubuntu/RSFMs/models/utils.py", line 14, in <module> from MinkowskiEngine import SparseTensor ImportError: cannot import name 'SparseTensor' from 'MinkowskiEngine' (unknown location)
So, I assume it needs to be installed for fine-tuning as well?
Hi, great work with MMEarth!
I am trying to load an MMEarth pretrained state dictionary but am having some issues — including when using the provided example:
I'm using the weights stored in pt-all_mod_atto_1M_128_uncertainty_112-16. The first issue that throws an error (even with strict=False) are shape mismatches between the state dictionary and the initialized model. But I think they can be fixed via:
model_state_dict = {}
for key, value in checkpoint["model"].items():
if ("grn" in key) and ("encoder" in key):
value = value[None, None, ...]
if ("bias" in key) and ("encoder" in key):
value = value.squeeze(0)
model_state_dict[key] = value
With these reshaped weights, torch.load now warns of many missing keys:
_IncompatibleKeys(missing_keys=['encoder.downsample_layers.0.0.weight', 'encoder.downsample_layers.0.0.bias', 'encoder.downsample_layers.0.1.weight', 'encoder.downsample_layers.1.0.weight', 'encoder.downsample_layers.1.0.bias', 'encoder.downsample_layers.1.1.weight', 'encoder.downsample_layers.2.0.weight', 'encoder.downsample_layers.2.0.bias', 'encoder.downsample_layers.2.1.weight', 'encoder.initial_conv.0.weight', 'encoder.initial_conv.1.weight', 'encoder.initial_conv.1.bias', 'encoder.stem.0.weight', 'encoder.stem.1.weight', 'encoder.stem.1.bias', 'encoder.stages.0.0.dwconv.weight', 'encoder.stages.0.0.norm.weight', 'encoder.stages.0.0.norm.bias', 'encoder.stages.0.0.pwconv1.weight', 'encoder.stages.0.0.pwconv1.bias', 'encoder.stages.0.0.pwconv2.weight', 'encoder.stages.0.0.pwconv2.bias', 'encoder.stages.0.1.dwconv.weight', 'encoder.stages.0.1.norm.weight', 'encoder.stages.0.1.norm.bias', 'encoder.stages.0.1.pwconv1.weight', 'encoder.stages.0.1.pwconv1.bias', 'encoder.stages.0.1.pwconv2.weight', 'encoder.stages.0.1.pwconv2.bias', 'encoder.stages.1.0.dwconv.weight', 'encoder.stages.1.0.norm.weight', 'encoder.stages.1.0.norm.bias', 'encoder.stages.1.0.pwconv1.weight', 'encoder.stages.1.0.pwconv1.bias', 'encoder.stages.1.0.pwconv2.weight', 'encoder.stages.1.0.pwconv2.bias', 'encoder.stages.1.1.dwconv.weight', 'encoder.stages.1.1.norm.weight', 'encoder.stages.1.1.norm.bias', 'encoder.stages.1.1.pwconv1.weight', 'encoder.stages.1.1.pwconv1.bias', 'encoder.stages.1.1.pwconv2.weight', 'encoder.stages.1.1.pwconv2.bias', 'encoder.stages.2.0.dwconv.weight', 'encoder.stages.2.0.norm.weight', 'encoder.stages.2.0.norm.bias', 'encoder.stages.2.0.pwconv1.weight', 'encoder.stages.2.0.pwconv1.bias', 'encoder.stages.2.0.pwconv2.weight', 'encoder.stages.2.0.pwconv2.bias', 'encoder.stages.2.1.dwconv.weight', 'encoder.stages.2.1.norm.weight', 'encoder.stages.2.1.norm.bias', 'encoder.stages.2.1.pwconv1.weight', 'encoder.stages.2.1.pwconv1.bias', 'encoder.stages.2.1.pwconv2.weight', 'encoder.stages.2.1.pwconv2.bias', 'encoder.stages.2.2.dwconv.weight', 'encoder.stages.2.2.norm.weight', 'encoder.stages.2.2.norm.bias', 'encoder.stages.2.2.pwconv1.weight', 'encoder.stages.2.2.pwconv1.bias', 'encoder.stages.2.2.pwconv2.weight', 'encoder.stages.2.2.pwconv2.bias', 'encoder.stages.2.3.dwconv.weight', 'encoder.stages.2.3.norm.weight', 'encoder.stages.2.3.norm.bias', 'encoder.stages.2.3.pwconv1.weight', 'encoder.stages.2.3.pwconv1.bias', 'encoder.stages.2.3.pwconv2.weight', 'encoder.stages.2.3.pwconv2.bias', 'encoder.stages.2.4.dwconv.weight', 'encoder.stages.2.4.norm.weight', 'encoder.stages.2.4.norm.bias', 'encoder.stages.2.4.pwconv1.weight', 'encoder.stages.2.4.pwconv1.bias', 'encoder.stages.2.4.pwconv2.weight', 'encoder.stages.2.4.pwconv2.bias', 'encoder.stages.2.5.dwconv.weight', 'encoder.stages.2.5.norm.weight', 'encoder.stages.2.5.norm.bias', 'encoder.stages.2.5.pwconv1.weight', 'encoder.stages.2.5.pwconv1.bias', 'encoder.stages.2.5.pwconv2.weight', 'encoder.stages.2.5.pwconv2.bias', 'encoder.stages.3.0.dwconv.weight', 'encoder.stages.3.0.norm.weight', 'encoder.stages.3.0.norm.bias', 'encoder.stages.3.0.pwconv1.weight', 'encoder.stages.3.0.pwconv1.bias', 'encoder.stages.3.0.pwconv2.weight', 'encoder.stages.3.0.pwconv2.bias', 'encoder.stages.3.1.dwconv.weight', 'encoder.stages.3.1.norm.weight', 'encoder.stages.3.1.norm.bias', 'encoder.stages.3.1.pwconv1.weight', 'encoder.stages.3.1.pwconv1.bias', 'encoder.stages.3.1.pwconv2.weight', 'encoder.stages.3.1.pwconv2.bias', 'encoder.norm.weight', 'encoder.norm.bias', 'encoder.head.weight', 'encoder.head.bias'], unexpected_keys=['loss_fn.log_vars', 'encoder.downsample_layers.0.0.ln.weight', 'encoder.downsample_layers.0.0.ln.bias', 'encoder.downsample_layers.0.1.kernel', 'encoder.downsample_layers.1.0.ln.weight', 'encoder.downsample_layers.1.0.ln.bias', 'encoder.downsample_layers.1.1.kernel', 'encoder.downsample_layers.2.0.ln.weight', 'encoder.downsample_layers.2.0.ln.bias', 'encoder.downsample_layers.2.1.kernel', 'encoder.initial_conv.0.kernel', 'encoder.initial_conv.1.ln.weight', 'encoder.initial_conv.1.ln.bias', 'encoder.stem.0.kernel', 'encoder.stem.1.ln.weight', 'encoder.stem.1.ln.bias', 'encoder.stages.0.0.dwconv.kernel', 'encoder.stages.0.0.norm.ln.weight', 'encoder.stages.0.0.norm.ln.bias', 'encoder.stages.0.0.pwconv1.linear.weight', 'encoder.stages.0.0.pwconv1.linear.bias', 'encoder.stages.0.0.pwconv2.linear.weight', 'encoder.stages.0.0.pwconv2.linear.bias', 'encoder.stages.0.1.dwconv.kernel', 'encoder.stages.0.1.norm.ln.weight', 'encoder.stages.0.1.norm.ln.bias', 'encoder.stages.0.1.pwconv1.linear.weight', 'encoder.stages.0.1.pwconv1.linear.bias', 'encoder.stages.0.1.pwconv2.linear.weight', 'encoder.stages.0.1.pwconv2.linear.bias', 'encoder.stages.1.0.dwconv.kernel', 'encoder.stages.1.0.norm.ln.weight', 'encoder.stages.1.0.norm.ln.bias', 'encoder.stages.1.0.pwconv1.linear.weight', 'encoder.stages.1.0.pwconv1.linear.bias', 'encoder.stages.1.0.pwconv2.linear.weight', 'encoder.stages.1.0.pwconv2.linear.bias', 'encoder.stages.1.1.dwconv.kernel', 'encoder.stages.1.1.norm.ln.weight', 'encoder.stages.1.1.norm.ln.bias', 'encoder.stages.1.1.pwconv1.linear.weight', 'encoder.stages.1.1.pwconv1.linear.bias', 'encoder.stages.1.1.pwconv2.linear.weight', 'encoder.stages.1.1.pwconv2.linear.bias', 'encoder.stages.2.0.dwconv.kernel', 'encoder.stages.2.0.norm.ln.weight', 'encoder.stages.2.0.norm.ln.bias', 'encoder.stages.2.0.pwconv1.linear.weight', 'encoder.stages.2.0.pwconv1.linear.bias', 'encoder.stages.2.0.pwconv2.linear.weight', 'encoder.stages.2.0.pwconv2.linear.bias', 'encoder.stages.2.1.dwconv.kernel', 'encoder.stages.2.1.norm.ln.weight', 'encoder.stages.2.1.norm.ln.bias', 'encoder.stages.2.1.pwconv1.linear.weight', 'encoder.stages.2.1.pwconv1.linear.bias', 'encoder.stages.2.1.pwconv2.linear.weight', 'encoder.stages.2.1.pwconv2.linear.bias', 'encoder.stages.2.2.dwconv.kernel', 'encoder.stages.2.2.norm.ln.weight', 'encoder.stages.2.2.norm.ln.bias', 'encoder.stages.2.2.pwconv1.linear.weight', 'encoder.stages.2.2.pwconv1.linear.bias', 'encoder.stages.2.2.pwconv2.linear.weight', 'encoder.stages.2.2.pwconv2.linear.bias', 'encoder.stages.2.3.dwconv.kernel', 'encoder.stages.2.3.norm.ln.weight', 'encoder.stages.2.3.norm.ln.bias', 'encoder.stages.2.3.pwconv1.linear.weight', 'encoder.stages.2.3.pwconv1.linear.bias', 'encoder.stages.2.3.pwconv2.linear.weight', 'encoder.stages.2.3.pwconv2.linear.bias', 'encoder.stages.2.4.dwconv.kernel', 'encoder.stages.2.4.norm.ln.weight', 'encoder.stages.2.4.norm.ln.bias', 'encoder.stages.2.4.pwconv1.linear.weight', 'encoder.stages.2.4.pwconv1.linear.bias', 'encoder.stages.2.4.pwconv2.linear.weight', 'encoder.stages.2.4.pwconv2.linear.bias', 'encoder.stages.2.5.dwconv.kernel', 'encoder.stages.2.5.norm.ln.weight', 'encoder.stages.2.5.norm.ln.bias', 'encoder.stages.2.5.pwconv1.linear.weight', 'encoder.stages.2.5.pwconv1.linear.bias', 'encoder.stages.2.5.pwconv2.linear.weight', 'encoder.stages.2.5.pwconv2.linear.bias', 'encoder.stages.3.0.dwconv.kernel', 'encoder.stages.3.0.norm.ln.weight', 'encoder.stages.3.0.norm.ln.bias', 'encoder.stages.3.0.pwconv1.linear.weight', 'encoder.stages.3.0.pwconv1.linear.bias', 'encoder.stages.3.0.pwconv2.linear.weight', 'encoder.stages.3.0.pwconv2.linear.bias', 'encoder.stages.3.1.dwconv.kernel', 'encoder.stages.3.1.norm.ln.weight', 'encoder.stages.3.1.norm.ln.bias', 'encoder.stages.3.1.pwconv1.linear.weight', 'encoder.stages.3.1.pwconv1.linear.bias', 'encoder.stages.3.1.pwconv2.linear.weight', 'encoder.stages.3.1.pwconv2.linear.bias'])
Any help would be much appreciated! And I apologize if I'm doing something dumb :)
Thanks again,
Anthony
I have installed GEO-Bench via pip (and ran geobench-test which passed). It downloads benchmark segmentation_v1.0 as default, so I changed:
segmentation_v0.9.1 to segmentation_v1.0 everywhere it was hardcoded.m-cashew-plantation to m-cashew-plant as it seems like they changed the key.However, my folder ../dataset/segmentation_v1.0/m-cashew-plant does not contain label_map.json nor label_stats.json which class geobench_dataset(Dataset) in custom_dataset.py tries to retrieve.
Can you confirm whether this is an issue that has arisen due to the update of the segmentation benchmark or if there might be something else going wrong on my side? Thanks!
Hi, is this a possible typo and should be "sentinel2_cloudprob" instead of "sentinel2_cloudprod"?
Edit:
When running ./slurm_scripts/slurm_fine_tune_seg.sh, I ran into the following error:
usage: FCMAE fine-tuning [-h] [--batch_size BATCH_SIZE] [--epochs EPOCHS] [--update_freq UPDATE_FREQ] [--model MODEL] [--input_size INPUT_SIZE] [--drop_path PCT] [--layer_decay_type {single,group}] [--patch_size PATCH_SIZE] [--baseline BASELINE] [--model_ema MODEL_EMA] [--model_ema_decay MODEL_EMA_DECAY] [--model_ema_force_cpu MODEL_EMA_FORCE_CPU] [--model_ema_eval MODEL_EMA_EVAL] [--clip_grad NORM] [--weight_decay WEIGHT_DECAY] [--lr LR] [--blr LR] [--layer_decay LAYER_DECAY] [--min_lr LR] [--warmup_epochs N] [--warmup_steps N] [--opt OPTIMIZER] [--opt_eps EPSILON] [--opt_betas BETA [BETA ...]] [--momentum M] [--weight_decay_end WEIGHT_DECAY_END] [--color_jitter PCT] [--aa NAME] [--smoothing SMOOTHING] [--train_interpolation TRAIN_INTERPOLATION] [--reprob PCT] [--remode REMODE] [--recount RECOUNT] [--resplit RESPLIT] [--mixup MIXUP] [--cutmix CUTMIX] [--cutmix_minmax CUTMIX_MINMAX [CUTMIX_MINMAX ...]] [--mixup_prob MIXUP_PROB] [--mixup_switch_prob MIXUP_SWITCH_PROB] [--mixup_mode MIXUP_MODE] [--finetune FINETUNE] [--head_init_scale HEAD_INIT_SCALE] [--model_key MODEL_KEY] [--model_prefix MODEL_PREFIX] [--linear_probe LINEAR_PROBE] [--data_path DATA_PATH] [--nb_classes NB_CLASSES] [--output_dir OUTPUT_DIR] [--log_dir LOG_DIR] [--device DEVICE] [--seed SEED] [--resume RESUME] [--eval_data_path EVAL_DATA_PATH] [--imagenet_default_mean_and_std IMAGENET_DEFAULT_MEAN_AND_STD] [--data_set {CIFAR,IMNET,image_folder,geobench.m-eurosat,geobench.m-so2sat,geobench.m-bigearthnet,geobench.m-brick-kiln,geobench.m-cashew-plantation,geobench.m-SA-crop-type}] [--auto_resume AUTO_RESUME] [--save_ckpt SAVE_CKPT] [--save_ckpt_freq SAVE_CKPT_FREQ] [--save_ckpt_num SAVE_CKPT_NUM] [--start_epoch N] [--eval EVAL] [--dist_eval DIST_EVAL] [--num_workers NUM_WORKERS] [--pin_mem PIN_MEM] [--crop_pct CROP_PCT] [--world_size WORLD_SIZE] [--local_rank LOCAL_RANK] [--dist_on_itp DIST_ON_ITP] [--dist_url DIST_URL] [--use_amp USE_AMP] [--wandb WANDB] [--wandb_project WANDB_PROJECT] [--wandb_run_name WANDB_RUN_NAME] [--pretraining PRETRAINING] [--use_orig_stem USE_ORIG_STEM] [--run_on_test RUN_ON_TEST] [--percent PERCENT] [--num_samples NUM_SAMPLES] [--test_scores_dir TEST_SCORES_DIR] FCMAE fine-tuning: error: unrecognized arguments: --visualize False
I have removed the flag from the script and that seemed to work.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.