Progressing our Agentic engineering abilities by building a 1 billion row challenge app with Electron, DuckDB, Prompts, Cursor and Aider

• Youtube

• brew install duckdb (mac see docs for other platforms)
• yarn install
• cp .env.sample .env - Update the .env file with your openai key
• yarn run generate <number of rows> - Generate the data (optional)
  • 'The git committed data/measurements.txt contains 1 million rows you can use without running this command'
  • 'I recommend generating 1mil rows'
  • 'I wrote the generation script (scripts/createMeasurements.ts) based off the original implementation, to be memory efficient for node, but it still takes a while to generate the data since node is (relatively) hella slow.'
  • 'My M2 64gb ram 1 billion rows took ~8 minutes to generate'. Adjust accordingly.
• yarn run agent - Run turbo4 agent to generate the knowledge base and create the data/agentOutput/* files (optional)
  • The git commited data/agentOutput/* contains the knowledge base you can use without running this command
• duckdb ./data/db.duckdb < ./data/agentOutput/generate-table.sql - Create the tables based on data/measurements.txt
• yarn run dev - Start the app

• src/main/turbo4.ts - OpenAI Assistant Wrapper with Knowledge Base Build/Consume support
• src/main/main.ts - Electron Main Process
• src/renderer/App.vue - Renderer Process (Vue App)
• scripts/createMeasurements.ts - Generate the measurements data
• data/agentOutput/* - Contains files generated by the agents
• data/agentOutputBackup/* - Contains files generated by original 1brc electron app in youtube AI Devlog
  • If you want overwrite data/agentOutput/* and want to use the original data, copy the contents of this folder to data/agentOutput/

• 1 Billion Row Challenge Original
• 1 Billion Row Challenge DuckDB
• Aider
• Cursor
• DuckDB Docs
• Electron Vite Vue Typescript Starter
• Vuetify Server Table
• Vuetify Pagination
• Electron
• LLM In CLI

• Prove our agents did their jobs
  • duckdb ./data/db.duckdb < ./data/agentOutput/generate-table.sql
  • duckdb ./data/db.duckdb .tables
  • duckdb ./data/db.duckdb "from measurements limit 5"
  • duckdb ./data/db.duckdb "from brc limit 5"

• duckdb - Start DuckDB Shell
• In the shell
  • .open <database> - Open a database
  • .tables - List all tables
  • .schema <table> - Show schema of a table
  • .quit - Quit the shell
  • .help - Show help
• duckdb <path/to/database.duckdb> - Start Shell with database
• duckdb <path/to/database.duckdb> <sql statement> - Run SQL statement on database
• duckdb <path/to/database.duckdb> <duckdb command> - Run DuckDb statement on database

flowchart LR

    subgraph database

    end

    subgraph electron:main

    end

    subgraph electron:renderer

    end

     database <--> electron:main <--> electron:renderer 

flowchart LR

    subgraph database
    subgraph duckdb tables
        d_a["brc: {station, min, max, mean}"]
        d_b["measurements: {station, measurement}"]
    end
        
    end

    subgraph electron:main
        em_a[src/main/turbo4.ts]
        em_b[scripts/createMeasurements.ts]
        em_c[src/main/main.ts]
    end

    subgraph electron:renderer
        er_a[src/renderer/App.vue]
    end

     database <--CRUD--> electron:main <--Communication vs Preload.ts--> electron:renderer 

flowchart LR

    subgraph database
    subgraph duckdb tables
        d_a["brc: {station, min, max, mean}"]
        d_b["measurements: {station, measurement}"]
    end
        
    end

    subgraph electron:main
        em_a[src/main/turbo4.ts]
        em_b[scripts/createMeasurements.ts]
        em_c[src/main/main.ts]
    end

    subgraph electron:renderer
        er_a[src/renderer/App.vue]
    end

    subgraph knowledge_base
        kb_a[1brc OG Gunnar post]
        kb_b[1brc DuckDB post]
        kb_c[DuckDB Documentation]
    end

     database <--CRUD--> electron:main <--Communication Via Preload.ts--> electron:renderer 
     em_a <--build kb --> knowledge_base