I got all of the compressed data downloaded, it looks like many of the files are 1/5 the size compressed.

The goals of this check-in are:

1. Done with the first stage of data cleaning

2. Propose an improved analytical question

Data description

1. Relational Diagram of the files (@dhruvalb)
2. MPI run time experiment

Exploratory Analysis

1. Top 15 tags (@tonofshell) - this file OR this file (I'm confused. Are they the same?)
2. Users Activities (@tonofshell) - which users are most active - this file
3. Questions with most answers per year (@tonofshell) - this file
4. Users with gold answer badges locations (@tonofshell) - this file
5. 2-grams of tags that appear together (network of tags) (@tonofshell) - this file

Main Analysis

1. Time series plots of each language

• Please ask @liu431 for the output

The goals of this check-in are:

Hi everyone,

As Adam must be busy working on getting us access to G cloud buckets for us, I thought I would help him get some of the meeting agendas for tomorrow down. Please feel free to add.

Agenda

1. Adam & Nikki briefly report on the uploading and cleaning the datasets

2. Discuss how to implement the main analysis "For each programming language, as it becomes more popular/commonplace, do answer providers become more hostile, meaner, negative towards question askers? The underlying assumption is that answer providers should be more positive/willing to give good and kind answers to questions related to nascent programming languages.

3. Li briefly reports on his sentiment analysis code

4. Dhruval reports on her findings on the datasets

Goal

1. Nail down what columns from which data set we would need to do the analysis

2. How to operationalize it using MapReduce and/or MPI?

Hello Devs,
I want all StackOverflow data in CSVs
like
Post Title | Answer like that

Is there any way ?

User Story:

These can be done in Dataproc.

Please follow these steps:

1. run Questions with highest number of answers per year decrs_max_ans_q.py with Posts.csv

2. run Users who answered/asked questions the most decrs_users_activities.py with Posts.csv

3. Add a column ",badges" to Badges.csv, save it to a new file in the bucket

4. Add a column ",users" to Users.csv, save it to a new file in the bucket

5. Cat Badges.csv and Users.csv, save it to a new file "badges_users.csv"

6. run The locations where users with gold answer badges are from decrs_users_gold_ans.py with "bagdes_users.csv"

7. run 2-grams tags that are usually tagged together decrs_n_grams_tags.py with Posts.csv

I will add more to the list as we have more. Thanks!

This task is due Wednesday, May 8, 2019.

What to do:

• Subset 500 lines of files that you're assigned
• Parse XML to CSV
• Save two CSV files to ./data directory on Github repo

Here's the assignment:

I have found a few issues that will add to the complexity of our analysis that I think we should start thinking about. Feel free to add any questions or problems you find with the data to this issue.

• The data is in XML format NOT raw text files
  • Can we parse XML line by line?
  • Depending on the XML parser, this is likely stored in Python as a list of dictionaries, one dictionary for each row in the XML file.
    • Each row is one user or comment
• Any comment or About Me attribute (basically anything with a sentence or more of text) within each XML row is in HTML format
  • Do we simply drop all HTML tags and how would we do this?
  • Do we process the data and save it to disk in another format to use for our analysis?

Hi guys,

I want to point out that, for some reasons, the sample processed data files are gone. Is that on purpose?

Does anybody have trouble with scp? I could not transfer files from remote to local for some reasons. Is there a better way to do it? As of now, I created a private Git repo where I can push my stuff there and retrieve it on my local, but it is super inconvenient.

test