madgrades / madgrades-extractor Goto Github PK

UW-Madison course and grade distribution data extraction tool.

License: MIT License

Java 100.00%

uw-madison pdf-extractor csv sql java-8 database

madgrades-extractor's Introduction

madgrades-extractor

This project reads UW Madison grade distribution and course report PDF files (published by the UW Madison Office of the Registrar) and converts them into CSV or SQL dump files.

You will find published, update-to-date datasets at Kaggle.

Conversion

The conversion process for a single term is as follows:

Open DIR report for the term.

a. Extract table from PDF (using tabula)

b. Read each row, adding new section per row.

c. Collate section info as necessary (i.e. 2 instructors for 1 single section)

d. Collate courses which appear to be cross-listed (based on similarity between sections offered)
Open grades report for the term.

a. Extract table from PDF

b. Read each row, add add each section grade data to course data added by the DIR report process

Typically all terms are extracted so this process repeats for each term.

Command Line Usage

Build it yourself with mvn clean install or grab a release from the releases page.

Usage: <main class> [options]
  Options:
    -d, -download
      Download the PDF reports instead of extracting data
      Default: false
    -e, -exclude
      Comma-separated list of term codes to exclude (ex. -e 1082)
    -f, -format
      The output format
      Default: CSV
      Possible Values: [CSV, MYSQL]
    -l, -list
      Output list of terms to extract
      Default: false
    -out, -o
      Output directory path for exported files (ex. -o ../data)
      Default: ./
    -t, -terms
      Comma-separated list of term codes to run (ex. -t 1082,1072)

Examples:

java -jar madgrades-final-1.0-SNAPSHOT.jar: will fetch every term and output files to the current directory
java -jar madgrades-final-1.0-SNAPSHOT.jar -t 1082: will fetch just term 1082
java -jar madgrades-final-1.0-SNAPSHOT.jar -o ../ -t 1082,1072: will fetch terms 1072 and 1082, output to ../

Relational Diagram

The CSV or SQL dumps are in the format of a collection of relational entities modeled something like this:

madgrades-extractor's People

Contributors

Stargazers

Watchers

Forkers

bpodonnell1234 odysa dyerai jamesjulich

madgrades-extractor's Issues

Handle term 1124 (Spring 2012)

The only term known that generated the final dir report differently from all the rest.

Compare 1124 to any other final DIR.

Differences:

column order
table layout/sizing
fonts/colors

0% A Rate for Spring 2022 Courses

Courses taught in Spring 2022 appear to have a 0.0% A rate, despite the grade report from the registrar saying otherwise.

For instance, MATH 340 taught by Peter Juhasz (Lec 002 and Lec 003) shows 0 As being granted for the Spring 2022 semester. The actual A rate should be ~30%.

Philosophy 101 taught by Henry Southgate in Spring 2022 is another class that is having the same issue.

These are just two classes that I have noticed that have the issue, but it is likely a much more widespread issue. I'm going to run this program on my system to figure out whether this was an issue with the extractor or an issue updating the databases. I'll update this issue thread if I find anything.

Handle multiple instructor names

There are some cases where instructors have different name entries per semester.

Check on courses with differing subject, but same subject codes

https://github.com/thekeenant/madgrades/blob/997b9a7f735129254eb5291a4c52c25d0b6a1671/src/main/java/com/keenant/madgrades/data/Course.java#L45

Maybe use Map<String, Subject> instead?

Add optional delete statement to SQL dumps

Handy instead of manually dropping tables/deleting.

Year formatting not up to date

I tried to clone the project and gave the '-d' argument to the problem with the following error:

Scraping for subjects and report URLs...
Exception in thread "main" java.lang.NumberFormatException: For input string: "|"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:580)
	at java.lang.Integer.parseInt(Integer.java:615)
	at com.keenant.madgrades.tools.Scrapers.toTermCode(Scrapers.java:101)
	at com.keenant.madgrades.tools.Scrapers.scrapeGradeReports(Scrapers.java:66)
	at com.keenant.madgrades.CommandLineApp.main(CommandLineApp.java:78)

It seems that it is caused by the code on line 101 in Scrapers.java:

int year = Integer.parseInt(termName.split(" ")[1]);

I suspect that the format has changed and the algorithm here need to be updated.

Explore other options for PDF parsing

e.g. https://github.com/facebookresearch/nougat
https://github.com/microsoft/table-transformer

CS/ECE 561 is cross listed incorrectly

Reported by a user:

CS/ECE 561 in the system is also cross-listed with N E and Physics, which is not the case, and has the wrong title "Introduction to Charged Particle Accelerators"The correct title should be "Probability and Information Theory in Machine Learning" and is only cross-listed between CS and ECE.

The extractor seems to get some data from the reports that points to these being the same course when they clearly are not.

Use latest names for instructors

If an instructor changes their name we currently do not honor the name change, even though that is reflected in the data published by the university.

Extract subject areas/codes

https://registrar.wisc.edu/subjectarea/