The Importance of Pretrained Model Weight. about vision_transformer HOT 4 OPEN

@Zhu-haow

When I drop part of the pretrained model weight., for example, the projection 'embedding' layer(between the R50 and Transformer Encoder)

Actually weights depend on arcitecture, so if you remove any intermediate layer then the rest of the network becomes "useless" and needs to be trained again

from vision_transformer.

In fact, I keep all layers. I just remove a single conv pretrained weight and random initialize this conv again. Nothing changes except this. But when I train this model, I got significant lower accuracy (10 percent).

@Zhu-haow

When I drop part of the pretrained model weight., for example, the projection 'embedding' layer(between the R50 and Transformer Encoder)

Actually weights depend on arcitecture, so if you remove any intermediate layer then the rest of the network becomes "useless" and needs to be trained again

from vision_transformer.

@Zhu-haow Hi Zhu, Im new to flax and jax, and am having trouble fine tuning the pre-trained model on self-defined dataset, would you mind sharing how to build self-defined dataset as well as how to do the fine-tuning...

I have tried the command in main page README but the job got killed all the time.. I am kinda stuck now... Would be great to learn from you. Many thanks!

from vision_transformer.

@Zhu-haow Hi Zhu, Im new to flax and jax, and am having trouble fine tuning the pre-trained model on self-defined dataset, would you mind sharing how to build self-defined dataset as well as how to do the fine-tuning...

I have tried the command in main page README but the job got killed all the time.. I am kinda stuck now... Would be great to learn from you. Many thanks!

Hope can be helpful. you can use below code to get data

`    all_image_paths = list(data_path.glob('*/*'))
    all_image_paths = [str(path) for path in all_image_paths]  # 所有图片路径的列表
    label_names = sorted(item.name for item in data_path.glob('*/') if item.is_dir())
    label_to_index = dict((name, index) for index, name in enumerate(label_names))
    all_image_labels = [label_to_index[pathlib.Path(path).parent.name] for path in all_image_paths]
    ds = tf.data.Dataset.from_tensor_slices((all_image_paths, all_image_labels))`

then use
data = data.map(_pp, tf.data.experimental.AUTOTUNE)
to map your data to preprocess function of _pp

from vision_transformer.