Python-based text extraction and analysis tool leveraging web scraping and NLP techniques.
git clone https://github.com/kamalkish0r/Text-Analysis.git
python -m venv env
source env/bin/activate # For Windows, use env\Scripts\activate
pip install -r requirements.txt
python main.py
- I started with reading the objective of the assignment.
- Then I explored the Input/Output files.
- Then I went through the given guide to text analysis, which helped me understand how can I compute values of required variables.
- Then one by one, taking the reference from the guide provided I wrote the logic to compute the variable values.
- Saved the extracted text into
data/extracted_text/{url_id}.txt
- Saved the computed values of variable into
data/Output Data Structure.xlsx
in the corresponding row & column. - Difficulties Encoutered :
- Some of the web pages have different html structure, so I needed to handle that in order to scrape the
Title
&Content
of the article. I usedtry except
blocks to identify the errors and fixed them by providing different selectors for these kind of web pages.
- Some of the web pages have different html structure, so I needed to handle that in order to scrape the
Feel free to reachout if you have any queries regarding my solution!