susanou / eecs349finalproject Goto Github PK

Generate the output in the correct format for the submission.

I think we can use some package mentioned here (https://automatetheboringstuff.com/chapter12/) to actually create an XLSX file directly. Which would be nice.

As I mentioned in an email, hackforums or whatever it is requires an account to actually view threads. We will need to learn some things about the website

• What HTTP headers we can use to authenticate (OAuth, login api to get he key)
• Structure of post and threads
• Scraping across pages

This scraper should be scanning the forum for posts about a designated crimeware or CaaS.

Ideally, it will

• search the forums for the keyword in post titles (the crimeware name or CaaS)

• Check if the threads found meet the required discussion threshold (> 30 comments)

• Create the list of thread URLs for the scraping

This is contingent on #1 ,

This scraper should be able to take a URL or some thread ID and download the HTML for the forum (probably using BeautifulSoup).
With the HTML handled, it should be able to distinguish comments from each other and collect all the necessary information (as specified in the T1 template).

However, this is separate from the sentiment analysis (good review, question, etc)