Some python scripts that I used to create a website classifier.
- It reads a file containing URLs
- It trawls the URLs to extract the web content
- Then, it cleans the content and tokenizes the sentences
- It checks if the words correspond to any features (Bag-of-words model)
- The words are weighed using TF-IDF
- The websites are classified using Bayes' probability