shark-nlp / cont Goto Github PK

As described in the article 2205.14690: "The feature representations come from pooling the output of the encoder (source sequence) or decoder (target sequence) ." However, the Transformer decoders contain cross attention modules, wouldn't this lead to the information leakage of the target sequence feature representation?

Hi,

While playing around, I have found an error in the function torch_bleu which is used to rank batch-negatives and beam-positives.

In model.model.CoNTGenerator.torch_bleu (line 47-70),
there is an severe mistake which results in wrong BLEU scores, certainly when n_gram == 1 and possibly when n_gram >= 2 (rare case where token indices are propotional; i.e. 2-gram [4, 8] and [34, 68]).

Current line 66-67:

sim_matrix = torch.cosine_similarity(input_tensor2_4gram.unsqueeze(3), input_tensor1_4gram.unsqueeze(2),
                                             dim=-1) >= 1.0

Suggestion:

sim_matrix = torch.norm( # Calculate L2 norm to find if N-gram in `sys`` is present in `ref``
        input_tensor2_4gram.unsqueeze(3) - input_tensor1_4gram.unsqueeze(2),
        p=2,
        dim=-1
) == 0.0

您好，我正打算使用CoNT进行文本摘要相关的对比学习，数据格式是CSV，两列内均为str类型文本。经过对代码进行一定修改后，在process_data函数内遇到了一些问题，包括：

• target和source是否为content和label的列名，我对内容进行修改后可以通过运行
• instance显示应该为一个列表，我将文本修改为[文本]的格式，尽管这样可以通过数据处理部分的代码，但是会在训练部分报错，报错内容为fastNLP.core.collators.padders.exceptions.EleDtypeUnsupportedError: Fail to get padder for field:target_outp. TorchNumberPadderonly supports padding python numbers or numpy numbers or torch.Tensor but get<class 'str'>. To view more information please set logger's level to DEBUG.

是否可以提供一下数据格式的说明，以便更好的应用代码？感谢您的贡献。

我看到代码里有

def form_ngram(self, input_tensor, n=2)  
def torch_bleu(self, ref_tensor, sys_tensor, pad_id, n_gram=2):

这俩个函数来计算bleu score.请问为什么不使用nltk这种可以计算bleu的三方库而选择自己实现呢?请问下主要的考量是什么?

Really interesting paper! I was wondering, when will the code be made available for it?

File "xxx/Consistency_model/CoNT/model/model.py", line 192, in generate encoder_feature = self.affine_transformation(encoder_hidden_states, attention_mask) # batch x h File "/xxx/Consistency_model/CoNT/model/model.py", line 103, in affine_transformation trans_tmp = trans_tmp * padding_mask.unsqueeze(-1).float() RuntimeError: The size of tensor a (128) must match the size of tensor b (16) at non-singleton dimension 0
使用的模型为fnlp/cpt-large，在运行训练时，如果在训练过程中需要eval，则会出现如上的错误，请问是什么问题？

您好，很有趣的工作！
我想请问一下，CoNT-main/model/model.py在CoNTGenerator类的forward函数中
distance_mask = (actual_distance < 0.48)
这一行代码是什么意思？0.48是超参数吗？
感谢您的回复！

你好，对您的工作十分感兴趣，可以告知下关于运行配置和训练时长方面的问题吗