Git Product home page Git Product logo

nlg-metricverse's People

Contributors

andreazammarchi3 avatar disi-unibo-nlp avatar giacomofrisoni avatar marcoavagnano98 avatar valentinapieri avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

nlg-metricverse's Issues

Calling scorer gives: TypeError: __call__() takes 1 positional argument but 3 were given

Describe the bug
When I try to follow the tutorial for, I get the error TypeError: __call__() takes 1 positional argument but 3 were given.

To Reproduce

from nlgmetricverse import NLGMetricverse
scorer = NLGMetricverse(metrics=["bertscore"])
predictions = ["test"]
references = ["test"]
scores = scorer(predictions, references) # <-- error happens here

Expected behavior
This should calculate the bertscore.

Exception Traceback (if available)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[28], line 1
----> 1 scores = scorer(predictions, references)
      2 print(scores)

TypeError: __call__() takes 1 positional argument but 3 were given

Environment Information:

  • OS: (e.g. Ubuntu 20.04 LTS)
  • nlgmetricverse version: 0.9.6
  • evaluate version: 0.3.0
  • datasets version: 2.12.0

Remove Retracted Paper Citations

Dear authors,

Thanks for your contribution on this work to the community.

However, this paper has been retracted from COLING 2022. The PDF file of this paper has been watermarked with "RETRACTED" on ACL Anthology. To maintain academic integrity and prevent the dissemination of false claims, please remove any "COLING22" citations from this repository.

Result Inconsistency with e2e-metrics.py

Describe the bug
Result didn't same as e2e-metrics
(Or is there any hyper parameters that I missed?)

To Reproduce
Calculate score with same data

Expected behavior
The results should be the same.

Environment Information:

  • OS: Ubuntu 22.04.3 LTS
  • nlgmetricverse version: 0.9.9

Pip install on mac fails

Describe the bug
Pip installing fails on mac

To Reproduce
pip install nlg-metricverse

Exception Traceback (if available)
`% pip install nlg-metricverse
Looking in indexes: https://pypi.org/simple, https://nlp.circleci:****@artifactory.ops.babylontech.co.uk/artifactory/api/pypi/babylon-pypi/simple
Collecting nlg-metricverse
Using cached nlg-metricverse-0.1.0.tar.gz (81 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [18 lines of output]
Traceback (most recent call last):
File "/Users/francesco.moramarco/gvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in
main()
File "/Users/francesco.moramarco/gvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/francesco.moramarco/gvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/jj/s730cphx5cq73f0xd964gn780000gp/T/pip-build-env-fl5ohxlq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/jj/s730cphx5cq73f0xd964gn780000gp/T/pip-build-env-fl5ohxlq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
self.run_setup()
File "/private/var/folders/jj/s730cphx5cq73f0xd964gn780000gp/T/pip-build-env-fl5ohxlq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 483, in run_setup
super(_BuildMetaLegacyBackend,
File "/private/var/folders/jj/s730cphx5cq73f0xd964gn780000gp/T/pip-build-env-fl5ohxlq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 335, in run_setup
exec(code, locals())
File "", line 153, in
File "", line 71, in get_requirements
FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.`

Environment Information:

  • OS: macOS Monterey (v12.6.1)
  • nlgmetricverse version: latest on pypi

AttributeError: 'DownloadConfig' object has no attribute 'storage_options'

Describe the bug
from nlgmetricverse import NLGMetricverse, load_metric
then the error message occurred:
AttributeError: 'DownloadConfig' object has no attribute 'storage_options'

To Reproduce
Just import the package: from nlgmetricverse import NLGMetricverse, load_metric

Expected behavior
No error message.

Environment Information:
Google Colab Default Environment

Moreover, this bug seems to be the same in obss/jury#125.
It is possible that the bug is caused by Datasets.
After I run pip install datasets==2.9.0, this bug is temporarily fixed.

moverscore fails

Describe the bug
The MoverScore calculation seems to fail when trying to make use of the DistilBert tokenizer.

To Reproduce

>>> import nlgmetricverse
>>> scorer = nlgmetricverse.NLGMetricverse(metrics=['moverscore'])
>>> scorer(predictions=['foo'], references=['bar'])
# Expected score value

Exception Traceback (if available)

>>> import nlgmetricverse
2023-09-19 13:19:51.367801: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> scorer = nlgmetricverse.NLGMetricverse(metrics=['moverscore'])
/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/transformers/generation_utils.py:24: FutureWarning: Importing `GenerationMixin` from `src/transformers/generation_utils.py` is deprecated and will be removed in Transformers v5. Import as `from transformers import GenerationMixin` instead.
  warnings.warn(
/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/transformers/generation_tf_utils.py:24: FutureWarning: Importing `TFGenerationMixin` from `src/transformers/generation_tf_utils.py` is deprecated and will be removed in Transformers v5. Import as `from transformers import TFGenerationMixin` instead.
  warnings.warn(
loading configuration file config.json from cache at /home/mmior/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/6cdc0aad91f5ae2e6712e91bc7b65d1cf5c05411/config.json
Model config DistilBertConfig {
  "activation": "gelu",
  "architectures": [
    "DistilBertForMaskedLM"
  ],
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "output_attentions": true,
  "output_hidden_states": true,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "transformers_version": "4.33.2",
  "vocab_size": 30522
}

loading file vocab.txt from cache at /home/mmior/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/6cdc0aad91f5ae2e6712e91bc7b65d1cf5c05411/vocab.txt
loading file added_tokens.json from cache at None
loading file special_tokens_map.json from cache at None
loading file tokenizer_config.json from cache at /home/mmior/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/6cdc0aad91f5ae2e6712e91bc7b65d1cf5c05411/tokenizer_config.json
loading configuration file config.json from cache at /home/mmior/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/6cdc0aad91f5ae2e6712e91bc7b65d1cf5c05411/config.json
Model config DistilBertConfig {
  "_name_or_path": "distilbert-base-uncased",
  "activation": "gelu",
  "architectures": [
    "DistilBertForMaskedLM"
  ],
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "transformers_version": "4.33.2",
  "vocab_size": 30522
}

loading weights file model.safetensors from cache at /home/mmior/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/6cdc0aad91f5ae2e6712e91bc7b65d1cf5c05411/model.safetensors
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertModel: ['vocab_layer_norm.bias', 'vocab_transform.weight', 'vocab_projector.bias', 'vocab_layer_norm.weight', 'vocab_transform.bias']
- This IS expected if you are initializing DistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of DistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.
If your task is similar to the task the model of the checkpoint was trained on, you can already use DistilBertModel for predictions without further training.
>>> scorer(predictions=['foo'], references=['bar'])
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/mmior/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/mmior/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/moverscore_v2.py", line 30, in process
    a = ["[CLS]"]+truncate(tokenizer.tokenize(a))+["[SEP]"]
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/moverscore_v2.py", line 25, in truncate
    if len(tokens) > tokenizer.max_len - 2:
AttributeError: 'DistilBertTokenizer' object has no attribute 'max_len'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/nlgmetricverse/core.py", line 86, in __call__
    score = self._compute_single_score(inputs)
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/nlgmetricverse/core.py", line 208, in _compute_single_score
    score = metric.compute(predictions=predictions, references=references, reduce_fn=reduce_fn, **kwargs)
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/evaluate/module.py", line 444, in compute
    output = self._compute(**inputs, **compute_kwargs)
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/nlgmetricverse/metrics/_core/base.py", line 362, in _compute
    result = self.evaluate(predictions=predictions, references=references, reduce_fn=reduce_fn, **eval_params)
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/nlgmetricverse/metrics/_core/base.py", line 302, in evaluate
    return eval_fn(predictions=predictions, references=references, **kwargs)
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/nlgmetricverse/metrics/moverscore/moverscore_planet.py", line 132, in _compute_single_pred_single_ref
    idf_dict_ref = moverscore_v2.get_idf_dict(references)  # idf_dict_ref = defaultdict(lambda: 1.)
  File "/home/mmior/.local/share/virtualenvs/annotate-schema-yEyO5xw6/lib/python3.8/site-packages/moverscore_v2.py", line 42, in get_idf_dict
    idf_count.update(chain.from_iterable(p.map(process_partial, arr)))
  File "/home/mmior/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/mmior/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
AttributeError: 'DistilBertTokenizer' object has no attribute 'max_len'

Environment Information:

  • OS: Ubuntu 20.04.6 LTS
  • nlgmetricverse version: 0.9.9
  • evaluate version: 0.4.0
  • datasets version: 2.9.0
  • moverscore version: 1.0.3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.