This project is a part of the Data Science Working Group at Code for San Francisco. Other DSWG projects can be found at the main GitHub repo.

The purpose of this project is to use analytics and topic modelling of search text to improve the user experience at https://data.sfgov.org/

• SF Open Data
• https://datasf.org/opendata/
• Partner contact: Jason Lally, @jasonlally

• Data Analysis
• Descriptive Statistics/Data Visualization
• Natural Language Processing
• Word2Vec Modelling

• R
• Python
  • Pandas, Spacy

Th major goals of the project are as follows:

• Clean and process search terms and categorize search terms by quality
• Utilize Natural Language Processing and Topic Modelling on valid search terms and cluster terms to determine potential demand for data sources
• Provide actionable insights to improve search functionality on the site

• NLP/Topic Modelling Expertise

1. Clone this repo, for help see this tutorial.

2. Data is being kept here

3. Data processing/transformation script is Data Combiner

   • Script that combines raw data from .tsv files into a single .csv file
   • Search Data Processing Jupyter Notebook - Notebook that cleans, processes and categorizes search terms

• Exploration of Search Google Analytics

• Search Data Modelling Juptyer Notebook

  • Notebook with vectorization of search terms using pre-trained word2vec model

Team Lead (Contact): Rocio S Ng (@Rocio)

• If you haven't joined the SF Brigade Slack, you can do that here.
• Our slack channel is #datasci-open-data_src
• Feel free to contact team leads with any questions or if you are interested in contributing!