dfki-nlp / smv Goto Github PK

Code and data for the ACL 2023 NLReasoning Workshop paper "Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methods" (Feldhus et al., 2023)

Python 100.00%

chatgpt explainability explainable-nlp feature-attribution integrated-gradients interpretability saliency-maps saliency-methods text-classification transformers

smv's People

Contributors

Stargazers

Watchers

Forkers

techthiyanes

smv's Issues

The explanations variable (l. 14) needs to be iterated over simultaneously with the texts variable if I see it correctly.
Right now, I'm getting a different text (Tkey) with each loop (l. 15), but the same explanation (first instance) every time.
Maybe we need

for x, text in zip(explanations, texts):

in l. 15?

missing documentation; and outdated docs

update docs in
https://github.com/MNasert/dfki-verbalizer/tree/main/search_methods

documentation is missing or outdated

Remove tokenization artefacts in Summarization mode

https://github.com/MNasert/dfki-verbalizer/blob/22a9afd4518c8ee292877e3156d965dbef37e1c1/src/search_methods/post_searches.py#L101

Indexing the dataset, retrieving a specific instance

Add requirements.txt

Please add a requirements file to the repo.

List may be incomplete:

numba
numpy
transformers

Field and span search return nothing

When calculating metric via "mean: n%" mean is too large for any value

Add typing to all functions, methods?

Issue when getting explanation: getitem out of range

no pointer to code here as issue is very obvious

if no filter or span search result is returned (due to no snippets matching our criterion found) program crashes as we cant access an empty item

Problems keeping track of input-id order

https://github.com/MNasert/dfki-verbalizer/blob/11e0710a9ec0f8cc4ec33b9d2d93a413d61d5250/search_methods/spans.py#L78
https://github.com/MNasert/dfki-verbalizer/blob/11e0710a9ec0f8cc4ec33b9d2d93a413d61d5250/search_methods/filters.py#L128

underlying issue is not in these methods but in
https://github.com/MNasert/dfki-verbalizer/blob/11e0710a9ec0f8cc4ec33b9d2d93a413d61d5250/search_methods/spans.py#L7
https://github.com/MNasert/dfki-verbalizer/blob/11e0710a9ec0f8cc4ec33b9d2d93a413d61d5250/search_methods/filters.py#L7

Both methods directly sort for highest value thus losing track of which sample corresponds to which coherent_word_sum

solutions:
might be solved by using another list-layer

Major issue; broken return shape

https://github.com/MNasert/dfki-verbalizer/blob/ced7c268290fffcf658afecbe251b24be05fb2eb/test.py#L50

When running the code as is, the verbalizations and corresponding index, values dictionary gets fused into one dictionary somehow.
Backtrace stops at dataloader.convolution_search / dataloader.span_search

https://github.com/MNasert/dfki-verbalizer/blob/ced7c268290fffcf658afecbe251b24be05fb2eb/dataloader.py#L207
which backtraces to:
https://github.com/MNasert/dfki-verbalizer/blob/ced7c268290fffcf658afecbe251b24be05fb2eb/dataloader.py#L240

These two methods return a tuple that should be unpacked implicitly but it doesnt resulting in dictionary shape:
({any: tuple({any : list})})

Only code changed was in dataloader.py and test.py
yet the problem already occurs in dataloader.py