π¦Έπ»ββοΈ DIGA - Dynamically Instance-Guided Adaptation: A Backward-free Approach for Test-Time Domain Adaptive Semantic Segmentation
We are still working on this repo now.
Before everything start, please make sure the following environment variables(UDATADIR, UPRJDIR, UOUTDIR
) are setup. The following operation would only modifying the above three folders on your devices.
example:
# e.g.
export UDATADIR=~/data # dir for dataset
export UPRJDIR=~/code # dir for code
export UOUTDIR=~/output # dir for output such as logs
export WANDB_API_KEY="xxx360492802218be41f76xxxx" # your wandb key
export NUM_WORKERS=0 # number of works used
mkdir -p $UDATADIR $UDATADIR $UOUTDIR # create dir if it does not exist
ps. Wandb is a wonderful tool for visualization similar to Tensorboard but offer more functions (link).
- Copy or clone this project.
- Organize the files in the following structure.
Code for this project should be put in $UPRJDIR
with the following structure:
$UPRJDIR
βββ DIGA
βββ ...
βββ Readme.md
βββ ...
Then you can use cd $UPRJDIR/DIGA
to get in the project directory.
- Init file structure by running
cp -r $UPRJDIR/DIGA/src/utils/advent_list_lib/* $UDATADIR/
- download dataset
Please read the following before get start.
The environment variable $UDATADIR
setted in last step would be used to locate the dataset files when program is running. Each dataset is contained in one folder, the structure in $UDATADIR
should be like this:
$UDATADIR
βββ GTA5 # (GTA5 in paper)
βββ synthia # (Synthia in paper)
βββ Cityscapes # (CityScapes in paper)
βββ BDD # (BDD100K in paper)
βββ IDD # (IDD in paper)
βββ NTHU # (CrossCity in paper)
βββ Mapillary # (Mapillary in paper)
In most dataset folders, there is one folder called βadvent_listβ, which contains the list for training, validation, testing set [following implementation in ADVENT].
We provide the lists file in this project, the folder is $UPRJDIR/DIGA/src/utils/advent_list_lib
Running the follows would set up the folders for you automatically:
# copy DIGA/src/utils/advent_list_lib/* to $UDATADIR/
cp -r $UPRJDIR/DIGA/src/utils/advent_list_lib/* $UDATADIR/
Now, the structure should look like this:
$UDATADIR
βββ GTA5 # (GTA5 in paper)
β βββ advent_list
β βββ ...
βββ synthia # (Synthia in paper)
β βββ ...
β βββ ...
βββ Cityscapes # (CityScapes in paper)
β βββ advent_list
β βββ ...
βββ BDD # (BDD100K in paper)
β βββ advent_list
β βββ ...
βββ IDD # (IDD in paper)
β βββ ...
β βββ ...
βββ NTHU # (CrossCity in paper)
β βββ advent_list
β βββ ...
βββ Mapillary # (Mapillary in paper)
β βββ advent_list
β βββ ...
The next step is downloading each dataset. It would cost too much space to offer instructions about downloading and there are many tutorial online. Instead, we offer an illustration as follows about the final dataset structure. You can download the datasets and organize them like follows. Some instructions can be find here (link1, link2)
$UDATADIR
βββ BDD
β βββ advent_list
β βββ images
β βββ labels
βββ Cityscapes
β βββ README
β βββ advent_list
β βββ gtFine
β βββ leftImg8bit
βββ GTA5
β βββ advent_list
β βββ images
β βββ labels
βββ IDD
β βββ gtFine
β βββ iddscripts
β βββ leftImg8bit
βββ Mapillary
β βββ LICENSE
β βββ README
β βββ advent_list
β βββ config.json
β βββ small
β βββ testing
β βββ training
β βββ validation
βββ NTHU
β βββ Rio
β βββ Rome
β βββ Taipei
β βββ Tokyo
β βββ advent_list
βββ synthia
β βββ Depth
β βββ GT
β βββ RGB
- Download pre-trained models
- Organize the files in the following structure.
Source models are required to be put in specific directory for running. The pre-trained models are available here (link).
ps. We use consistent models from previous work of this repo link.
$UDATADIR
βββ models # create a new folder called "models" under $UDATADIR
βββ DA_Seg_models # downloaded from above link
βββ GTA5
βββ GTA5_baseline.pth
βββ SYNTHIA
βββ SYNTHIA_source.pth
βββ ...
βββ ...
βββ BDD
The following cmd would do this automatically:
mkdir -p $UDATADIR/models
cd $UDATADIR/models
wget -O download.zip https://www.dropbox.com/s/gpzm15ipyt01mis/DA_Seg_models.zip?dl=0https://www.dropbox.com/s/gpzm15ipyt01mis/DA_Seg_models.zip?dl=0
unzip download.zip -d ./
ps. adding, deleting, editing path could be done at configs/model/net/gta5_source.yaml
We provided 3 ways to setup the environments.
- Using Develop Environment in VS Code (most Recommended)
- Using Docker (Recommended)
- Using Pip/Conda
If you are using VS Code, this is the most recommended way. You can set up all the environment for this project in just one step.
- Open this project in VS Code
- Install extension βDev Containersβ
- Press Cmd/Ctrl+Shift+P β Dev Container: Rebuild and Reopen in Container
ps. The config folder .devcontainer has been included in our project, you can edit it as if you would introduce or remove some libraries.
ps. Details and Instructions about Dev Container in VS Code can be found here (link).
Dockerfile is at .devcontainer/Dockerfile
- Install pytorch form official website (link).
pip install \
tensorboard
pandas \
opencv-python \
pytorch-lightning \
hydra-core \
hydra-colorlog \
hydra-optuna-sweeper \
torchmetrics \
pyrootutils \
pre-commit \
pytest \
sh \
omegaconf \
rich \
fiftyone \
jupyter \
wandb \
grad-cam \
tensorboardx \
ipdb \
hydra-joblib-launcher
ps. We use pytorch/pytorch:1.12.1-cuda11.3-cudnn8-devel while other versions should also work.
The following command will use test the performance of a model pre-trained on GTA5
with target as the validation set of Cityscapes
. Hyper-parameters are set as default.
python src/train.py experiment=ttda
Output Example:
wandb: Run summary:
...
wandb: test/acc/dataloaderr13_idx_0 55.01257 # mIoU of 13 class
wandb: test/acc/dataloaderr16_idx_0 49.24151 # mIoU of 16 class
wandb: test/acc/dataloaderr_idx_0 45.81422 # mIoU of 19 class
...
You can use the following to start an experiment with custom source model
, target dataset
and hyper-parameters
.
# e.g.
python src/train.py \
experiment=ttda \
model/net=gta5_source \
datamodule/test_list=cityscapes \
model.cfg.bn_lambda=0.8 \
model.cfg.proto_lambda=0.8 \
model.cfg.fusion_lambda=0.8 \
model.cfg.confidence_threshold=0.9 \
model.cfg.proto_rho=0.1
The available choices are listed as following:
-
Source model:
model/net=gta5_source,synthia_source,gta5_synthia_source
-
Target dataset:
datamodule/test_list=idd,crosscity,cityscapes,bdd,mapillary
-
Hyper-Paramerters:
bn_lambda: 0-1 (default 0.8)
proto_lambda: 0-1 (default 0.8)
fusion_lambda: 0-1 (default 0.8)
confidence_threshold: 0-1 (default 0.9)
proto_rho: 0-1 (default 0.1)
ps. File based customization is also supported by modifying configs/model/diga.yaml
. Note that the cmd line has higher priority if there are conflicting options.