Reference: https://hub.docker.com/_/flink
You can run a JobManager (master).
$ docker run --name flink_jobmanager -d -t flink jobmanager
NB: if you have already the container created, the above command will lead to an error like below:
docker: Error response from daemon: Conflict. The container name "/flink_jobmanager" is already in use by container "32523f311c747cf6b0c0744d9edda2788af2213da189cd7926f60a4192073ce8". You have to remove (or rename) that container to be able to reuse that name.
In this case, run
docker start flink_jobmanager
You can also run a TaskManager (worker). Notice that workers need to register with the JobManager directly or via ZooKeeper so the master starts to send them tasks to execute.
$ docker run --name flink_taskmanager -d -t flink taskmanager
With Docker Compose you can create a Flink cluster:
version: "2.1"
services:
jobmanager:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
expose:
- "6123"
ports:
- "8081:8081"
command: jobmanager
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
and just run docker-compose -f docker-compose-flink.yml up --no-recreate
.
Scale the cluster up or down to N TaskManagers:
docker-compose -f docker-compose-flink.yml up --scale taskmanager=<N> --no-recreate
E.g.,
docker-compose -f docker-compose-flink.yml up --scale taskmanager=3 --no-recreate
$ flink run -c io.github.streamingwithflink.chapter1.AverageSensorReadings examples-scala_2.12-1.0.jar
- open http://localhost:8081/#/submit
- upload the jar file
examples-scala_2.12-1.0.jar
- specify the Entry Class io.github.streamingwithflink.chapter1.AverageSensorReadings
- Submit
https://hub.docker.com/r/bitnami/kafka/
$ docker-compose -f docker-compose-kafka.yml up --no-recreate
https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html