Trying out new technologies

Details

1. Build a Postgres Docker image that creates a user and a database at startup
2. Populate a table using College Scorecard data from US Government

1. Download dotnet core and JDK 11
2. Download PostgresSQL client
3. Use the sample Powershell script to set up dotnet core, JDK and Postgres db server access

• A simple console app
• An xUnit test project
• Cake or C# Make: writing code inside cake scripts

• Single-source feature test
• A Maven test project targeting JDK 11

• A simple Postgressql db access Go project

• A faust: Python Kafka Stream Processing test project

• A Golang sample project
• A dotnet core 2.1 sample project
• A Java project

If you are behind a firewall, these sample projects will fail. I documented my workaround to run tests successfully in local dev environment behind a proxy server.

Each Workflow, or a DAG, is written in Python code. Each task is still executed by a worker in a separate process, that can be any OS command. It does not run on Windows natively. It can run in a Linux subsystem on Windows.

• A test DAG
• A test plugin, or the code in a separate module so that a DAG is more readable

A test project that does:

• use sqlalchemy to generate select and insert queries
• use batches to improve performance

Check out my blog post about how to deal with big data sets.

Using poetry to manage dependencies and virtualenv