Hi Zexuan, I tried to reproduct the NER results for SciERC and ran t

Non-deterministic NER results about pure HOT 2 CLOSED

princeton-nlp commented on May 30, 2024

Non-deterministic NER results

from pure.

Comments (2)

a3616001 commented on May 30, 2024

Hi Yaobin! How many GPUs are you using for each run? I suspect that this is because the model behavior might be slightly different with different #GPUs. (My experiments were based on one single 2080 GPU).

from pure.

isLouisHsu commented on May 30, 2024

Hi Zexuan,

I use one single GTX3090 GPU. I found the problem is caused by allennlp.nn.utils.batched_index_select which calls torch.index_select. I implemented the function and got deterministic results. However, the function is really time-consuming.

def batched_index_select(target: torch.Tensor, indices: torch.LongTensor) -> torch.Tensor:
    batch_size, sequence_length, hidden_size = target.size()
    _, num_spans = indices.size()
    indexed = []
    for b in range(batch_size):
        target_b = target[b]
        indices_b = indices[b]
        indexed_b = []
        for i in range(num_spans):
            indexed_b.append(target_b[indices_b[i]])
        indexed_b = torch.stack(indexed_b, dim=0)
        indexed.append(indexed_b)
    indexed = torch.stack(indexed, dim=0)
    return indexed