grseb9s / 3d-semantic-segmentation-of-modular-furniture Goto Github PK

The code is tested on Ubuntu 14.04.

Download RGB-D furniture models and their annotations from here.

eigen, PCL, Gurobi, OpenCV

Following two commands to be run in the terminal from the build directory:

cmake -DCMAKE_BUILD_TYPE=Release ..
make

Note: change the required directory path in CMakeList file.

./bin/cli train ../data/depth/160/crossValidate/set4/train/

All the training models must be stored in the build directory before testing Each training can be separately enabled in parser.cpp

1. trainEdgeDetector
2. trainPartPrior
3. trainAppearanceCodeBook

for example to parse an image named '728.JPG' in the folder ../data/depth/160/crossValidate/set4/test

./bin/cli parse ../data/depth/160/crossValidate/set4/test/ 728

the resulting image will be stored with the name 'last_result.png' in the build directory.

'./bin/cli test ../data/depth/160/crossValidate/set4/test/'

The results will be saved in 'results' folder and the quantitative results in 'results.txt' in the build directory.

RDT(rectangleDetectionThreshold) and maxIOU are critical parameters for the segmentation.

• RDT = 6 for WACV 17 synthetic dataset
• RDT = 1 for WACV 17 real kinect dataset
• RDT = 3 for WACV 16 dataset
• maxIOU = 0.95

Note: RDT can be adjusted in parser.cpp and maxIOU can be adjusted in detector.h
Recommended: RDT = 1 for real images

• DEBUG_MODE_ON: if 1, displays the debugging outputs in the command prompt

• INCLUDE_DEPTH: if 1, includes depth information in the whole pipeline

• DUMMY_MCMC_LOGIC: if 1, enables the 2 phase architecture and deactivtes the default architecture

• ROI_CORRUPTION: if 1 enables corrpting the Region of interest bounding box, the extent of corruption (in pixels) can be specified using CORRUPT_PIXELS

• SPLIT_MERGE_AUGMENT: if 1, enables proposal pool (split and merge) augmentation The split augment parameter can be specified using PROJ_PROF_THRESH and merging allowance (in pixels) can be specified using MERGE_ALLOWANCE

1. INIT_PROB_BIRTH
2. INIT_PROB_DEATH
3. INIT_PROB_SPLIT
4. INIT_PROB_MERGE
5. INIT_PROB_LABEL_DIFFUSE: switch move
6. INIT_PROB_EXCHANGE_DATADRIVEN: data driven exchange
7. INIT_PROB_UPDATE_CENTER
8. INIT_PROB_UPDATE_WIDTH
9. INIT_PROB_UPDATE_HEIGHT
10. INIT_PROB_EXCHANGE_RANDOM: random exchange

1. MAX_TEMP // Starting temperature
2. MIN_TEMP // ending temperature
3. ALPHA // Geometric cooling parameter
4. MAX_UPDATE_ITER // To check convergence
5. NUM_INNER_LOOPS // iterations at each temperature

1. MAX_TEMP_WARMUP
2. MIN_TEMP_WARMUP
3. ALPHA_WARMUP
4. MAX_UPDATE_ITER_WARMUP
5. NUM_INNER_LOOPS_WARMUP

1. MAX_TEMP_MASTER
2. MIN_TEMP_MASTER
3. ALPHA_MASTER
4. MAX_UPDATE_ITER_MASTER
5. NUM_INNER_LOOPS_MASTER

1. EXCHANGE_MOVE_IDX
2. BIRTH_MOVE_IDX
3. DEATH_MOVE_IDX
4. SPLIT_MOVE_IDX
5. MERGE_MOVE_IDX
6. LABEL_DIFFUSE_MOVE_IDX
7. EXCHANGE_DD_MOVE_IDX
8. UPDATE_CENTER_MOVE_IDX
9. UPDATE_WIDTH_MOVE_IDX
10. UPDATE_HEIGHT_MOVE_IDX

1. WH_DIFFUSE_FACTOR : for diffusing width and height
2. CENTER_DIFFUSE_FACTOR : for diffusing center location

• ROULETTE_EXCHANGE: If 1, weighted sampling (russian roulette) is enabled

• CLUSTER_MAX_IOU // measure to check similarity of rectangles in redundancy removal (proposal pool augmentation)

• numPartsCBUpperBound // Only if running in low specification systems

• MAX_OVERLAP

• APPEARANCE_AUGMENT: If this is true, then the parser generates debug output during the parsing process

• VERBOSE_MODE: This is the size of the patches that are used for the edge detector
• PATCH_SIZE: Steepness of Probabilistic SVM Sigmoid

Note on shape prior: Form factor was excluded in the evaluation of Proposed approach in 3d as 7 dimensions were enough. But for 2d, to strenthen the shape prior it was includedin the trained models.

Note on testing on real images: The current depth rectification is done using bilinear interpolation which works perfectly for synthetic images but not the best for real images especially for the ones with large amount of transformation. The authentic rectification is to implement the incomplete module using normal computation, clustering, gram schmidt orthonormalization and then cancelling the transformation based on the computed co-ordinate system. This section is incomplete.

To add noise to the synthetic images use use addKinectNoise.m in scripts folder To get real png images (RGB and depth) use oni2PNG.cpp