CLIP_CN_Tutorial 微调卡住了，也不报错 about modelscope HOT 8 CLOSED

尝试将tain和evaluation "workers_per_gpu"设置为0。

from modelscope.

2024-04-12 11:13:07,235 - modelscope - INFO - ==========================Training Config Start==========================
2024-04-12 11:13:07,235 - modelscope - INFO - {
"framework": "pytorch",
"task": "multi-modal-embedding",
"pipeline": {
"type": "multi-modal-embedding"
},
"pretrained_model": {
"model_name": "damo/multi-modal_clip-vit-base-patch16_zh"
},
"dataset": {
"column_map": {
"img": "image",
"text": "query"
}
},
"train": {
"work_dir": "./workspace/ckpts/clip",
"max_epochs": 1,
"use_fp16": true,
"dataloader": {
"batch_size_per_gpu": 180,
"workers_per_gpu": 16,
"shuffle": true,
"drop_last": true
},
"lr_scheduler": {
"warmup_proportion": 0.1
},
"lr_scheduler_hook": {
"type": "LrSchedulerHook",
"by_epoch": false
},
"optimizer": {
"type": "AdamW"
},
"optimizer_hparams": {
"lr": 2.5e-05,
"weight_decay": 0.001,
"beta1": 0.9,
"beta2": 0.999,
"eps": 1e-08
},
"optimizer_hook": {
"type": "TorchAMPOptimizerHook",
"cumulative_iters": 1,
"loss_keys": "loss"
},
"loss_cfg": {
"aggregate": true
},
"hooks": [
{
"type": "IterTimerHook"
},
{
"type": "ClipClampLogitScaleHook"
}
],
"logging": {
"interval": 1,
"out_dir": "./workspace/ckpts/clip"
},
"checkpoint": {
"best": {
"metric_key": "inbatch_t2i_recall_at_1",
"by_epoch": false,
"interval": 200,
"save_dir": "./workspace/ckpts/clip"
},
"period": {
"interval": 1,
"save_dir": "./workspace/ckpts/clip"
}
}
},
"evaluation": {
"dataloader": {
"batch_size_per_gpu": 128,
"workers_per_gpu": 16,
"shuffle": false,
"drop_last": true
},
"metrics": [
{
"type": "inbatch_recall"
}
],
"period": {
"by_epoch": true,
"interval": 1
}
},
"preprocessor": []
}
2024-04-12 11:13:07,236 - modelscope - INFO - ===========================Training Config End===========================
2024-04-12 11:13:07,239 - modelscope - INFO - Stage: before_run:
(ABOVE_NORMAL) OptimizerHook
(LOW ) LrSchedulerHook
(LOW ) BestCkptSaverHook
(LOW ) CheckpointHook
(VERY_LOW ) TextLoggerHook

Stage: before_train_epoch:
(LOW ) LrSchedulerHook

Stage: before_train_iter:
(ABOVE_NORMAL) OptimizerHook

Stage: after_train_iter:
(ABOVE_NORMAL) OptimizerHook
(NORMAL ) EvaluationHook
(NORMAL ) ClipClampLogitScaleHook
(LOW ) LrSchedulerHook
(LOW ) BestCkptSaverHook
(LOW ) CheckpointHook
(VERY_LOW ) TextLoggerHook

Stage: after_train_epoch:
(NORMAL ) EvaluationHook
(LOW ) LrSchedulerHook
(LOW ) BestCkptSaverHook
(LOW ) CheckpointHook
(VERY_LOW ) TextLoggerHook

Stage: after_val_epoch:
(VERY_LOW ) TextLoggerHook

Stage: after_run:
(LOW ) BestCkptSaverHook
(LOW ) CheckpointHook

2024-04-12 11:13:07,251 - modelscope - INFO - Checkpoints will be saved to ./workspace/ckpts/clip
2024-04-12 11:13:07,259 - modelscope - INFO - Checkpoints will be saved to ./workspace/ckpts/clip
2024-04-12 11:13:07,259 - modelscope - INFO - Text logs will be saved to ./workspace/ckpts/clip

from modelscope.

pip install pystack-debugger
pstack $pid

请通过这个命令查看下是卡在了哪段代码里

from modelscope.

@wenmengzhou

from modelscope.

似乎命令没成功执行，我们复现下这个错误看看

from modelscope.

@wenmengzhou 应该也不是所有机器都会这样，不然也不可能发布。
但卡住也无法调试，完全没法下一步。

from modelscope.

@slin000111 it just worked. thanks.
and why?

from modelscope.

可能是某些操作在子进程中会hang住导致

from modelscope.