Machine Learning model to rate Search Relevancy for Home Depot.
Visit:Kaggle Home Depot Product Search Relevance for more info and data file (could not upload on GitHub because of size)
Implementation Details: (using cosine similarity and linear regression)
- Starting by reading all the input files into a pandas dataframe in python. The next step is to extract the desired information from product_descriptions and attributes file.
- Then merge the required data from product_description and attributes file into train (dataframe in python)
- The fields used for merging are: product_description (from product_description.csv) and MFG Brand Name, Material, Bullet1, Bullet2 and Bullet3 (from attributes.csv)
- So, the dataframe train has has these fields: id, product_uid, product_title, search_term, relevance, product_description, MFG Brand Name, Material, bullet1, bullet2 and bullet3
- Then, dividing the data for each row in train into three parts: one is search_term (store in one dictionary), second is concatenation of product_description, product_title, MFG Brand Name, Material, bullet1, bullet2 and bullet3 (another dictionary) and third is the relevance field (kept for linear regression, not used at this step)
- Implementing the stemming and removing stopwords (after lowering the tokens) on all the desired fields which are to used in cosine similarity calculation.
- After calculating the cosine similarity between search term and other fields, there are two columns with numerical data: similarity and relevance.
- Implement Linear Regression on these two where x is similarity and y is relevance and theta values are to be calculated.
- So, in the end when test data comes, same merging and stemming processing is performed and cosine similarity is calculated search_term and other fields.
- Then relevance is predicted using theta values from Linear Regression algorithm.
References:
Cosine similarity:
- http://stackoverflow.com/questions/15173225/how-to-calculate-cosine-similarity-given-2-sentence-strings-python
- http://stackoverflow.com/questions/1746501/can-someone-give-an-example-of-cosinesimilarity-in-a-very-simple-graphical-wa
Linear regression:
Panda dataframe functions (merge, rename, lambda):