MIDS W261: Machine Learning at Scale | Fall 2021 | Final Project

Flight delays are a widespread problem in the United States and across the globe. The impact of flight delay can be a risk that represents financial losses, the dissatisfaction of passengers, time losses, loss of reputation and bad business relations. The ability to predict flight delays will help to reduce strain on the air travel system and result in significant financial savings for airlines and passengers.

Delayed flights are defined by the Federal Aviation Administration as a flight which arrives or departs more than 15  minutes later than scheduled. In 2019, the arrival delay rate was 19.2% and the departure delay rate was 18.18% in the United States[1]. The aim of our study is to predict departure delay as a binary yes/no feature, meaning flights that depart 15 minutes or more later than the schedule departure time (referred to as DEP_DEL15 in the flights dataset). As the flight departure time approaches the estimates for delays will increase in accuracy, but be less valuable for airlines and passengers. Given the tradeoff between accuracy and usefulness of delay prediction we have opted to predict the departure delay 2 hours ahead of scheduled departure time.

• Final Report: W261_AU21_FINAL_PROJECT.ipynb - An end-to-end pipeline overview of Question Formulation, Exploratory Data Analysis, Preprocessing, Feature Engineering, Algorithm Implementation of our solution.
• Presentation Slides - slide deck for final presentation with findings

• Jupyter notebooks detailing each stage of the pipeline.

1. EDA Notebook
2. Full Data Pipeline
3. CV GridSearch
4. Model Selection and Ensemble

• executable Python files to run the full pipeline from start to finish.

5. Pipeline: Full_Data_Pipeline.py - Data engineering pipeline to clean, transform, and augment the raw data to produce final train/test datasets suitable for supervised machine learning.
6. Pipeline: Model Selection and Ensemble.py - Logistic Regression, Random Forest and Gradient Boosted Trees models with ensemble of the best machine learning models found in the grid-searches to make final predictions and evaluate performance on the test dataset.
7. Pipeline: gridsearch_cv.py - Custom class to grid-search parameters for different models in a cross-validation method designed for sequential time-series data.