right now, doc id has the name of the text file as its value. Change it to the corresponding URL from the JSON.

2 grams before removing stopwords. Ignoring 2 grams with stopwords.

Eliminate filters from the code which we are crawling right now.

The object StemmedTerm looks a little bit haphazard, I think there should be a 'postings' object containing: docId,positions. Can contain a composite 'stemmedTerm' as this posting is for that 'stemmedTerm'. So basically can be 2 objects: stemmedTerm and posting. Need more clarity on this one.
Also, We did not explicitly get the 'docId's of webpages. Our unique field was the 'url' and is still the url. So will mostly will have to generate doc ids for unique web pages.***

***If working with a big url as an id gets cumbersome.

Instead of hard coding subdomain traps, plan is to detect a trap and then come out of it.

1. WAP to give 10 queries to Google and take the results and use them as reference
2. Compute NDCG@5 <I think @5 means for top 5 results>.

3.Put text snippets relevant to query for each document.

1. Anchor Text: to enhance ranking and give a 'title' instead of the url.

Naive implementation: Maintaining a list of visited URLs, but that might go out of memory when we start crawling. I think we need a robust impl on that.