Have Java installed. We can do this with a tool called SDK Man.

curl -s "https://get.sdkman.io" | bash
source "$HOME/.sdkman/bin/sdkman-init.sh"

sdk install java 11.0.23-zulu

Run the Java program.

git clone [email protected]:ericxiao251/flink-http-api-example.git
cd flink-http-api-example/flink-http-api-example/

./gradlew run 

Java repo setup.

Flink Dependencies: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#gradle-build-script.

Other Dependencies:

• Http client.
• JSON to Java Object (POJO) toolkit.

A simple Flink application (Main.java).

Example Main: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/overview/#example-program. Example Flink Async I/O function: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/#async-io-api.

An class that implements the Async I/O abstract class in Flink.

3 main abstract functions to implement:

1. public void open(Configuration parameters). This is what I would call the "initializer" of the operator, where you can start the HTTP client.
2. public void close(). This function is used to gracefully shutdown everything when your Flink application is turned off.
3. public void asyncInvoke(String s, ResultFuture<String> resultFuture).

(Optional) A Java "POJO" to represent the HTTP request response body, the Java way.

Async vs Synchronous

Sometimes HTTP Requests can take a lot longer, we do not want to block and create a bottleneck in our system.

Delivery Guarentees

Authentication

Retries

• Flink has some built in support for retries: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/#retry-support.

Monitoring

• Request duration
• Request response code
• Request volume/rate

Some talks on how to do this in Flink: https://www.youtube.com/watch?v=XgumbKHE2Zw.

Batching