514-labs / moose Goto Github PK

View Code? Open in Web Editor NEW

31.0 7.0 6.0 73.84 MB

The developer framework for your data & analytics stack

Home Page: https://www.moosejs.com

License: MIT License

JavaScript 4.18% CSS 0.63% TypeScript 33.17% Shell 1.04% Rust 46.93% MDX 12.52% Python 1.53%

data dataengineering deployment framework insights metrics analytics python rust typescript

moose's People

Contributors

Stargazers

Watchers

Forkers

nic mcginleyr1 georgevanderson oatsandsugar onelesd armanjindal

moose's Issues

[POL-84] Run topic synchronization script using igloo dev

_{From SyncLinear.com | POL-84}

[514-266] [514-268] [514-272] Testing module

null

514-281

From SyncLinear.com | 514-278

From SyncLinear.com | 514-275

From SyncLinear.com | 514-272

From SyncLinear.com | 514-268

_{From SyncLinear.com | 514-266}

[POL-80] Run SQL migration for clickhouse using igloo update command

_{From SyncLinear.com | POL-80}

Connection to Clickhouse on start up is brittle

When I spin up dev I get 1/2 times:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Network(hyper::Error(Io, Os { code: 54, kind: ConnectionReset, message: "Connection reset by peer" }))', src/infrastructure/console.rs:38:81
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[1]    99364 abort      ../igloo-stack/apps/igloo-kit-cli/target/debug/igloo-cli dev

Looks like we need to be more tolerant to network resets

[POL-52] Add color message type to CLI messages

I'd use indicatif and console to accomplish this

_{From SyncLinear.com | POL-52}

[POL-81] Create Topic migration script for redpanda

_{From SyncLinear.com | POL-81}

Flow module

null

514-277

From SyncLinear.com | 514-274

_{From SyncLinear.com | 514-270}

[POL-110] [POL-109] Enable persistent deployment to production (copy)

We need to be able to use the framework to instrument igloo itself and iterate on the data collected

_{From SyncLinear.com | POL-109}

_POL-110

[POL-105] Refactor the codebase for easy navigation and feedback

_{From SyncLinear.com | POL-105}

[514-276] [514-279] [514-282] Migration Module

null

514-282

From SyncLinear.com | 514-279

_{From SyncLinear.com | 514-276}

[POL-79] Create SQL migration template for clickhouse

_{From SyncLinear.com | POL-79}

Fix framework console production build

./src/app/constants.ts:1:1
#21 19.10 moose-console:build: Type error: 'constants.ts' cannot be compiled under '--isolatedModules' because it is considered a global script file. Add an import, export, or an empty 'export {}' statement to make it a module.
#21 19.10 moose-console:build: 
#21 19.10 moose-console:build: > 1 |
#21 19.10 moose-console:build:     | ^
#21 19.15 moose-console:build:  ELIFECYCLE  Command failed with exit code 1.
#21 19.17 moose-console:build: ERROR: command finished with error: command (/app/apps/moose-console) /usr/local/bin/pnpm run build exited (1)
#21 19.18 moose-console#build: command (/app/apps/moose-console) /usr/local/bin/pnpm run build exited (1)

And Add some CI to make sure we don't break this inadventendly

[POL-73] [POL-72] Figure out "migrations" for topics

This should probably include defining a schema for the red panda schema registry to enable data quality enforcement. As I've dug into this it seems like protobufs will be a requirement to be able to enforce schemas in the registry. This is something we've talked about in the past. Given that we're hoping for multi language support, it may make sense to leverage ~~~~~~protobufs~~~~~~ avro as the cross-language serialization it may make sense to make avro easy to work with.

_{From SyncLinear.com | POL-72}

[POL-78] Support interface based ingestion abstraction

We've explored leveraging an ORM to create the needed abstraction for the ingestion point. We believe that an ORM DSL based approach, like Prisma Schema Language is the best past forward where the ingestion point would take in a ORM generated type as it's input and enable the user to easily create an ingestion point.

With that in mind, though, we'd like to enable users to create ingestion points quickly. To do so, we will support creating an ingestion point and passing it a typescript interface as a generic type or function parameter (approach TBD).

The hope is for the user to be able to run this command and then have the ingestion point configured for them. We need to support:

creating the table in the database;
creating the topic in red panda
creating the sql statement to ensure that the topic is constantly pushing data to the table
creating the typescript code for the developer to add parameters

$ igloo add ingestion-point --name=user-event -a=timestamp:utc -a=packet:json -a=type:enum

The hardest problem will be enabling the developer to write code against typescript files and automatically update the migration. Alternatively, we can force the developer to define their parameters through the CLI or to manually update their migration scripts if they decide to write parameters manually.

_{From SyncLinear.com | POL-78}

[POL-61] [POL-51] Abstract CLI messages

Create a function to create readable and consistent CLI messages as opposed to one-off print statements

_{From SyncLinear.com | POL-51}

[514-274] Flow module

null

514-277

_{From SyncLinear.com | 514-274}

[POL-55] Flow abstraction

_{From SyncLinear.com | POL-55}

Flow module

[POL-76] Support Prisma as our OLAP ORM

I'm trying to flesh out whether we should create an ORM as part of the igloo framework. It seems like there would benefits both for the developer experience when using the framework but could also serve as an abstraction layer over different databases and stream systems.

It could then integrate nicely into the broader flows paradigms and possibly be used in conjunction with orchestration frameworks.

After exploring this topic we're concluding that contibuting a clickhouse connector to Primsa is like the best path forward. We're excited about the way they do migrations. We'll need to investigate more to understand the LOE here but from the preliminary investigation it seems feasible.

From what I can gather, we'll need DMMF support, type mapping for clickhouse supported types and likely some tweaks to the migrations given that alter statements are different in clickhouse than they are in other systems

_{From SyncLinear.com | POL-76}

[514-192] Give users visual feedback

_{From SyncLinear.com | 514-192}

[POL-57] Metric abstraction

_{From SyncLinear.com | POL-57}

[POL-54] Ingestion point abstraction

_{From SyncLinear.com | POL-54}

Use docker-compose instead of manually stitching docker run together

Right now we have a series of docker run command with a network setup in between.

We are managing all that and making the glue between the different element when we could generate a docker-compose.yaml file in the project directory and use all the optimizations from there when spinning up containers.

docker-compose is shipped with docker, so that should not add another dependency.

That should simplify our code and reduce the number of bugs.

Latest bug I spotted is that if the config of a container changes through a new version of the CLI, the containers fail to boot because we are only creating containers once and then running them. But if the configuration changes, we are not updating it. This is logic that is embedded in docker-compose

sfully validated clickhouse docker container
        Failed to run console container: docker: Error response from daemon: Conflict. The container name "/console-1" is already in use by container "b2cdf08f387864fb94d8546a5767391ef08dd89f7b5813afc51e685d1ff129f6". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

        Failed to run console container: docker: Error response from daemon: Conflict. The container name "/console-1" is already in use by container "b2cdf08f387864fb94d8546a5767391ef08dd89f7b5813afc51e685d1ff129f6". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

        Failed to validate red panda cluster