chiphuyen / lazynlp Goto Github PK

Hello,

I am reaching out regarding your source code file for your Python codes (crawl.py). After running tests using Pylint a few errors present in the source code were found. I felt that it could be something you could fix or look into.

lazynlp/crawl.py:173:0: W0102: Dangerous default value [] as argument (dangerous-default-value)
lazynlp/crawl.py:173:0: W0102: Dangerous default value [] as argument (dangerous-default-value)
lazynlp/crawl.py:222:8: W0105: String statement has no effect (pointless-string-statement)

outputLint.txt

Possible Fix is:

def download_pages(link_file,
                   folder,
                   timeout=30,
                   default_skip=True,
                   extensions=None,
                   domains=None):
    """
    Your function documentation here.
    """
    # Check if extensions and domains are None, and if so, initialize them to empty lists
    if extensions is None:
        extensions = []
    if domains is None:
        domains = []

    # Your function code continues...

This modification ensures that each call to download_pages() gets its own separate empty list for extensions and domains.

I have shown a few errors that I found using Pylint. I also added a link to a text file in which the errors present in all source files were reported using Pylint. Hope this helps.

Regards,
Rebal

Hi,
Thanks for this great tool.

I noticed urllib fails with a Forbidden Request error when I call download_page on some links. You can reproduce the error by trying the code below:

import lazynlp
link = "https://punchng.com/"
page = lazynlp.download_page(link, context=None, timeout=None)

This raises a 403 as shown below.

I've attempted to create a PR that adds headers to the request by default.

Hello,

There are legal problems with code with no license, where I work using code that has no license attached to it is outright banned.

Would you be so kind to add some sort of license in a file?

It would be very nice of you if it were something permissive, like MIT or Apache 2 or BSD too.

Thank you!

Hello,
I am reaching out regarding your Python code. After running tests using Pylint and Pyflakes, there are a few errors considering used variables that are present in the source codes and I felt that it could be something to look into and consider fixing:

Pylint:
lazynlp/cleaner.py:21:4: W0612: Unused variable 'e' (unused-variable)
lazynlp/crawl.py:95:4: W0612: Unused variable 'raw_url' (unused-variable)
lazynlp/crawl.py:118:4: W0612: Unused variable 'e' (unused-variable)
Pyflakes:
lazynlp/crawl.py:95:5: local variable 'raw_url' is assigned to but never used
lazynlp/crawl.py:118:5: local variable 'e' is assigned to but never used
lazynlp/crawl.py:121:5: local variable 'e' is assigned to but never used
lazynlp/crawl.py:152:5: local variable 'e' is assigned to but never used
lazynlp/crawl.py:155:5: local variable 'e' is assigned to but never used

outputFlakes.txt

outputLint.txt

There were a few more present in the remaining source code files, but for sake of not creating too long a message, I have shown a few errors for only variable types that I found using Pyflakes and Pylint. I also added a link to a text file in which the errors/bugs that were present in all source files were reported using Pylint and Pyflakes. Hope this helps.

Regards,
Rebal

I see a "syntax error near unexpected token `sgp.urls,'" on submitting the following command:
lazynlp.download_pages(sgp.urls, text_docs, timeout = 30, default_skip = True, extensions = [], domains = [])

Is there something wrong I am doing? sgp.urls has all the URLs, text_docs is the name of the folder to get the outputs into, the rest of the parameters as default.

One might as well extract structured data from each element of such a dataset.

Linked data.
https://5stardata.info/

Useful features:

• Relations to e.g. https://schema.org/Dataset (s)
• Reified edges to other https://schema.org/ScholarlyArticle (s) indicating whether A seems to confirm or disprove B
• URIs for columns in CSV and CSVW datasets
  • https://www.w3.org/TR/tabular-data-primer/ (CSVW)

Thanks for building this, really nice work!

I was reading through the code and noticed this line

Lmk

All the best

Have you considered adding a metric to assess the text quality of the documents, for example using the frequencies of short frequent words? (http://rolandschaefer.net/?p=78)

Hello,

I am reaching out regarding your source code files for your Python codes. After running tests using Pyflakes and Pylint, there were a few errors present in the source codes and I felt that it could be something you could fix or look into.

lazynlp/analytics.py:200:10: C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
lazynlp/analytics.py:231:21: W0613: Unused argument 'file' (unused-argument)
lazynlp/analytics.py:231:27: W0613: Unused argument 'gran' (unused-argument)
lazynlp/analytics.py:231:40: W0613: Unused argument 'max_n' (unused-argument)

lazynlp/cleaner.py:74:8: R1724: Unnecessary "else" after "continue", remove the "else" and de-indent the code inside it (no-else-continue)
lazynlp/crawl.py:96:4: E0633: Attempting to unpack a non-sequence defined at line 65 of tldextract.tldextract (unpacking-non-sequence)
lazynlp/crawl.py:106:0: R0911: Too many return statements (11/6) (too-many-return-statements)

outputLint.txt

These issues cause unnecessary memory use and can be better formatted. I have shown a few errors that I found using Pylint. I also added a link to a text file in which the errors/bugs that were present in all source files were reported using Pylint. Hope this helps.

Regards,
Rebal

Get important (?!) Images of the webpages in markdown style.