sciencerio / sciencer-toolkit Goto Github PK

While using the /paper/search entrypoint, the sum of the limit and offset must be less than 10000

The sum of offset and limit must be < 10000 (ref)

The semantic scholar provider crashes when the limit + offset are greater than 10000.
Consider (1) adding a stop condition while fetching the papers and (2) constraining the maximum number of papers to 10000.

I was looking into this and was questioning if the class PaperIDs makes sense existing as a separate class, or if we should just move it into the paper itself.

Originally posted by @RicardoEPRodrigues in #23 (comment)

Consider changing the Paper and PapersIDs models implementation to dataclasses (ref). Only works on python 3.7 or above.

Implementing these models as dataclasses will remove the complex interfaces they currently have.

Currently the CollectByTerms performs 1 request to the API (get_paper_by_terms), which returns N papers according to the query. Then, it performs N requests to get all the information of each paper (get_paper_by_id). Although there is a parameter to choose the output fields of the first query, there is no way to retrieve the ids of references and citations. I believe that is the reason for having those N extra requests inside the CollectByTerms.

I would like to suggest or discuss the possibility of not doing those N extra requests inside the CollectByTerms and, probably, only perform them in the expanders (where they will be needed?!). I think Sciencer was designed having in mind a pipeline that starts with a small set of papers, then it could be expanded and possibly be filtered. However, another possibility is to start with a huge set of papers and then start filtering and possibly expanding later. In those cases doing so many requests takes a huge time and most of those papers will probably be filtered afterwards, which means wasted time.

When using Filters to check paper's validity, the developer should define the default validity when the necessary properties are not available.

Consider adding an optional parameter to the constructor of the adequate Filters:
FilterByAbstract(word='keyword', accept_when_empty=True)

One of the motivators that lead to the development of Sciencer was the necessity to formalize the expansion flowchart of a systematic review. Besides the source code used to define an expansion, a visual representation should also be made available.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) has been widely used to report systematic reviews and meta analyses across several research fields. PRISMA flowchart describes not only the criteria for the systematic review but also the results.

Consider adding a method to generate a flowchart reporting the expansion to Sciencer Toolkit.

By default, a new LogCallbacks is added to Sciencer.iteration.
Instead, the developer should decide whether to use it or not.

As such, the LogCallbacks must not be added by default to the iteration's callbacks.

An error was launched when running the examples/example_1_hello_world.py (I provided an API key, IDK if it is relevant). It seems the response did not have a "data" field.

Traceback (most recent call last):
  File "/home/ricardo/Documents/Projects/sciencer-toolkit/./examples/example_1_hello_world.py", line 77, in <module>
    first_batch = s.iterate(
  File "/home/ricardo/Documents/Projects/sciencer-toolkit/venv/lib/python3.9/site-packages/sciencer/core.py", line 249, in iterate
    paper_after_expansion.update(expander.execute(
  File "/home/ricardo/Documents/Projects/sciencer-toolkit/venv/lib/python3.9/site-packages/sciencer/expanders/expand_by_authors.py", line 32, in execute
    author_papers = provider.get_papers_by_author(author_id)
  File "/home/ricardo/Documents/Projects/sciencer-toolkit/venv/lib/python3.9/site-packages/sciencer/providers/semantic_scholar_provider.py", line 151, in get_papers_by_author
    paper_json) for paper_json in response_json["data"]])
KeyError: 'data'

I would like to see integrated (perhaps in an Utils class) the ability to export the iterations in a format compatible with reference management software, such as RIS, Bibtex, or Endnote.