chiphuyen / lazynlp Goto Github PK
View Code? Open in Web Editor NEWLibrary to scrape and clean web pages to create massive datasets.
Library to scrape and clean web pages to create massive datasets.
Hello,
I am reaching out regarding your source code file for your Python codes (crawl.py). After running tests using Pylint a few errors present in the source code were found. I felt that it could be something you could fix or look into.
lazynlp/crawl.py:173:0: W0102: Dangerous default value [] as argument (dangerous-default-value)
lazynlp/crawl.py:173:0: W0102: Dangerous default value [] as argument (dangerous-default-value)
lazynlp/crawl.py:222:8: W0105: String statement has no effect (pointless-string-statement)
Possible Fix is:
def download_pages(link_file,
folder,
timeout=30,
default_skip=True,
extensions=None,
domains=None):
"""
Your function documentation here.
"""
# Check if extensions and domains are None, and if so, initialize them to empty lists
if extensions is None:
extensions = []
if domains is None:
domains = []
# Your function code continues...
This modification ensures that each call to download_pages() gets its own separate empty list for extensions and domains.
I have shown a few errors that I found using Pylint. I also added a link to a text file in which the errors present in all source files were reported using Pylint. Hope this helps.
Regards,
Rebal
Hi,
Thanks for this great tool.
I noticed urllib fails with a Forbidden Request
error when I call download_page
on some links. You can reproduce the error by trying the code below:
import lazynlp
link = "https://punchng.com/"
page = lazynlp.download_page(link, context=None, timeout=None)
This raises a 403 as shown below.
I've attempted to create a PR that adds headers to the request by default.
Hello,
There are legal problems with code with no license, where I work using code that has no license attached to it is outright banned.
Would you be so kind to add some sort of license in a file?
It would be very nice of you if it were something permissive, like MIT or Apache 2 or BSD too.
Thank you!
Hello,
I am reaching out regarding your Python code. After running tests using Pylint and Pyflakes, there are a few errors considering used variables that are present in the source codes and I felt that it could be something to look into and consider fixing:
Pylint:
lazynlp/cleaner.py:21:4: W0612: Unused variable 'e' (unused-variable)
lazynlp/crawl.py:95:4: W0612: Unused variable 'raw_url' (unused-variable)
lazynlp/crawl.py:118:4: W0612: Unused variable 'e' (unused-variable)
Pyflakes:
lazynlp/crawl.py:95:5: local variable 'raw_url' is assigned to but never used
lazynlp/crawl.py:118:5: local variable 'e' is assigned to but never used
lazynlp/crawl.py:121:5: local variable 'e' is assigned to but never used
lazynlp/crawl.py:152:5: local variable 'e' is assigned to but never used
lazynlp/crawl.py:155:5: local variable 'e' is assigned to but never used
There were a few more present in the remaining source code files, but for sake of not creating too long a message, I have shown a few errors for only variable types that I found using Pyflakes and Pylint. I also added a link to a text file in which the errors/bugs that were present in all source files were reported using Pylint and Pyflakes. Hope this helps.
Regards,
Rebal
I see a "syntax error near unexpected token `sgp.urls,'" on submitting the following command:
lazynlp.download_pages(sgp.urls, text_docs, timeout = 30, default_skip = True, extensions = [], domains = [])
Is there something wrong I am doing? sgp.urls has all the URLs, text_docs is the name of the folder to get the outputs into, the rest of the parameters as default.
One might as well extract structured data from each element of such a dataset.
Linked data.
https://5stardata.info/
Useful features:
Thanks for building this, really nice work!
I was reading through the code and noticed this line
Line 56 in 0869697
Lmk
All the best
Have you considered adding a metric to assess the text quality of the documents, for example using the frequencies of short frequent words? (http://rolandschaefer.net/?p=78)
Hello,
I am reaching out regarding your source code files for your Python codes. After running tests using Pyflakes and Pylint, there were a few errors present in the source codes and I felt that it could be something you could fix or look into.
lazynlp/analytics.py:200:10: C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
lazynlp/analytics.py:231:21: W0613: Unused argument 'file' (unused-argument)
lazynlp/analytics.py:231:27: W0613: Unused argument 'gran' (unused-argument)
lazynlp/analytics.py:231:40: W0613: Unused argument 'max_n' (unused-argument)
lazynlp/cleaner.py:74:8: R1724: Unnecessary "else" after "continue", remove the "else" and de-indent the code inside it (no-else-continue)
lazynlp/crawl.py:96:4: E0633: Attempting to unpack a non-sequence defined at line 65 of tldextract.tldextract (unpacking-non-sequence)
lazynlp/crawl.py:106:0: R0911: Too many return statements (11/6) (too-many-return-statements)
These issues cause unnecessary memory use and can be better formatted. I have shown a few errors that I found using Pylint. I also added a link to a text file in which the errors/bugs that were present in all source files were reported using Pylint. Hope this helps.
Regards,
Rebal
Get important (?!) Images of the webpages in markdown style.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.