argoproj-labs / old-argo-dataflow Goto Github PK

View Code? Open in Web Editor NEW

267.0 12.0 31.0 4.87 MB

Dataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines.

License: Apache License 2.0

Dockerfile 0.42% Makefile 1.04% Go 93.70% Java 0.21% Shell 0.12% Python 4.28% JavaScript 0.21% Procfile 0.03%

kubernetes pipeline kafka jetstream data

old-argo-dataflow's Introduction

Dataflow

NOTICE

Argo Dataflow has been reimplemented in the scope of a broader project focussed on real-time data processing and analytics. Please checkout the new numaflow project.

Summary

Dataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines.

Each pipeline is specified as a Kubernetes custom resource which consists of one or more steps which source and sink messages from data sources such Kafka, NATS Streaming, or HTTP services.

Each step runs zero or more pods, and can scale horizontally using HPA or based on queue length using built-in scaling rules. Steps can be scaled-to-zero, in which case they periodically briefly scale-to-one to measure queue length so they can scale a back up.

Learn more about features.

Use Cases

Real-time "click" analytics
Anomaly detection
Fraud detection
Operational (including IoT) analytics

Screenshot

Example

pip install git+https://github.com/argoproj-labs/argo-dataflow#subdirectory=dsls/python

from argo_dataflow import cron, pipeline

if __name__ == '__main__':
    (pipeline('hello')
     .namespace('argo-dataflow-system')
     .step(
        (cron('*/3 * * * * *')
         .cat()
         .log())
    )
     .run())

Documentation

Read in order:

Beginner:

Intermediate:

Advanced

Architecture Diagram

old-argo-dataflow's People

Contributors

Stargazers

Watchers

old-argo-dataflow's Issues

Investigate Camel K

https://camel.apache.org/camel-k/latest/

Golden metrics

Scheduling pipelines

As a user I want my pipeline to run at a specific time of day.

Report retry counter Prometheus metric

As a user, I want to retry errored messages, or maybe ignore them.

what happens when a pipeline changes?

In workflows, we treat spec as immutable. I think that'd be the case here too?

Kafka TLS support

Amazon Kinesis

Kubernetes event source

It might be useful to directly consume Kubernetes events as a source, saving any need to write any code.

sources:
- name: events 
  kubernetesEvents:
    namespace: All

Many nodes in one pod

Custom metrics

Pipelines will want to report custom metrics. These would be emitted by Prometheus and sent to Wavefront.

Make sure we're not implementing K8S Serverless (Kubeless, OpenFASS )

OpenFaaS 20k stars
Kubeless 6k
Fission 6k
OpenWhisk 5k
KNative 4k

https://www.infoworld.com/article/3510472/5-ways-to-do-serverless-on-kubernetes.html

There are MANY good K8S serverless solutions.

Termination - how does a step indicate the pipeline is done and should be terminated?

Document starting a pipeline using Argo Events

go.mod scanning with Snyk

expose Prometheus metrics from pod

We can look to Argo Events for this.

Provide production-ready NATS manifests

As an operator, I do not want to have to think about how to set this up. Please provide some production-ready set-up.

Persistence
TLS (hard)
Client-auth

Pod-affinity

As a user I may want to use affinity to determine where pods run.

Set pod security policy

suspend pipeline

add the ability to suspend and resume a pipeline

suspend would just be scale-to-zero

What about light-weight processors (e.g. using Common Expression Language)

Google PubSub

Speak to Doug Davis a IBM

Custom health check

I think I'd like to be able to write a custom health check for a pipeline. For example, if a certain percentage of errors occurred, it should be marked as "status.phase=Error".

NATS token based auth

Don't cannibalise Argo Workflows or Events

Make sure we are not re-inventing Knative

Document Argo Workflow interop

Scale-to-zero

Windows support

Named pipes is one problem:
https://github.com/Microsoft/go-winio

Implement stdin/stdout connectors

We need to use something similar to Emissary pattern to do this:

Init container copies binary to volume.
Controller update main container command.

How to monitor messages flowing in the system?

When working on this, and also from using Argo Events, messages on the bus are opaque.

This makes understanding what is going on really hard.

We should build some capability for this, maybe similar to various

https://github.com/KualiCo/nats-streaming-console

Contra: that seems to have been abandoned since 2018? Why? Was it not useful?

Could we just provide some manifests?

extract NATS installer to new repo (pkg?)

Rollout/restart

It should be possible to have each step in the pipeline be restarted, much like kubectl rollout restart deploy/my-deploy

Pipeline status

How do we report the status of a pipeline?

Should we support expression-based filtering of data when we source/sink it?

      sinks:
        - filter: 'toJson(message).type == "cat"'
          bus:
            subject: animals

sources:
- s3: 
    bucket: my-bucket
    filename: *.log

Each item in the bucket is a single data item. Any new items appear, these are automatically found and sent.

Sinks are a bit simpler, but we need to determine the filename:

sinks:
- s3: 
    bucket: my-bucket
    filename: meta.filename // an expression from the message context? just a random GUID?