tuvuumass / task-transferability Goto Github PK

Hi!
Thank you for open source your code for us to study.
I have the following questions:
1. I don't quite understand the meaning of the variables in formula(2) of your paper, which is the expected covariance of the gradients of the log-likelihood. Could you please explain it to me in more detail? i.e, what is the nabla_thelta?
2. In your code run_taskemb_SL.py or run_taskemb_CR.py, the fisher score was simply computed by model parameter's grad like score = score ** args.pow (which is 2.0 by default) is that right? Does this correspond to formula(2) in the paper? But I think score = score ** args.pow seems to be inconsistent with formula(2)

Looking forward to your reply, thanks!

Hi,
Can you share with us how to plot the text or task embeddings of shape (768,) as in Figure 3a and 3b ?

Thanks,
Fatma

Hi!
Great repository and awesome paper, thanks for making it available!

I'm currently experimenting with your TaskEmb embeddings, however I have some trouble getting results similar to those you reported. Specifically, I have the following issues/ questions:

1. For me, TaskEmb rankings for some target tasks are good (ndcg > 80) but it doesn't seem to work for other tasks (ndcg < 40). Have you encountered similar differences depending on the selected target task?
2. TaskEmbs for QA tasks seem to be off: They are often the lowest ranked source tasks, even when the target is also a QA task. I currently mostly tested CQ and DROP as targets and ndcg scores for both are very low (~30). From you paper, it seems QA tasks should actually work best, so is there anything additional to consider when computing or ranking TaskEmbs for these tasks?

For my experiments, I'm using RoBERTa instead of BERT as base model, I'm working in the "Full -> Limited" setting with transfer across all classes and my set of tasks consists of a subset of your choice of tasks.

Any insights or guidance would be greatly appreciated. Thank you very much!