dimakuv / connect-app-state-demos Goto Github PK
View Code? Open in Web Editor NEWDemos for applications checkpointing their own state (as reaction to SIGTERM) and then restoring it
License: BSD 3-Clause "New" or "Revised" License
Demos for applications checkpointing their own state (as reaction to SIGTERM) and then restoring it
License: BSD 3-Clause "New" or "Revised" License
[ Copy-pasted from https://github.com/gramineproject/gramine/discussions/1709 ]
I recently tried Apache Kafka with Kafka Python (confluent-kafka
Python package).
Latest Gramine (v1.6) works without any problems, at least on the example Kafka Producer and Kafka Consumer Python scripts that I was given.
Here I'll describe how to adapt our existing Python CI example to Kafka Python. Unfortunately I was given a private Kafka instance, with closed-source Python scripts. So I anonymize these parts.
I tried on Ubuntu 20.04, with Gramine v1.6. First, install Kafka:
# sudo needed to install into system-wide Python dirs where Gramine Python example looks for files
$ sudo pip3 install confluent-kafka
For Kafka Python example, I copy-pasted the existing Gramine Python example into a new dir:
cd gramine/CI-Examples
cp -r python kafka-python
cd kafka-python
There are the new files I added:
$ gramine/CI-Examples/kafka-python$ tree
├── python.manifest.template # modified, see diff below
├── example.json # kafka-producer.py sends contents of this file to Kafka
├── kafka-ca.pem # Kafka's CA certificate
├── kafka-user.key # my user's private key to log in Kafka
├── kafka-user.pem # my user's certificate to log in Kafka
├── ...
└── scripts
├── kafka-consumer.py
├── kafka-producer.py
├── ...
The kafka-consumer.py
script basically does this:
from confluent_kafka import Consumer
c = Consumer(conf, logger=logger)
c.subscribe(topics)
while True:
msg = c.poll(timeout=1.0)
print(msg.value(), flush=True)
The kafka-producer.py
script basically does this:
from confluent_kafka import Producer
p = Producer(conf)
p.produce(topic, message)
The manifest is modified like this:
diff --git a/python/python.manifest.template b/kafka-python/python.manifest.template
index 2b04d279..931719e7 100644
--- a/python/python.manifest.template
+++ b/kafka-python/python.manifest.template
@@ -52,4 +52,13 @@ sgx.trusted_files = [
{% endfor %}
"file:scripts/",
"file:helper-files/",
+
+ "file:kafka-user.key",
+ "file:kafka-user.pem",
+ "file:kafka-ca.pem",
+ "file:example.json",
+]
+
+sgx.allowed_files = [
+ "file:data.log", # file created by Kafka for logging
]
NOTE: Many parts of the original Python manifest can be removed, as they are not required by Kafka. Left as an exercise for readers.
I ran my example scripts like this:
gramine-sgx python scripts/kafka-producer.py \
--auth kafka-ca.pem kafka-user.pem kafka-user.key <passphrase> \
<kafka-broker-address>:9092 TestTopic example.json
gramine-sgx python scripts/kafka-consumer.py \
--auth kafka-ca.pem kafka-user.pem kafka-user.key <passphrase> \
<kafka-broker-address>:9092 INTEL latest TestTopic
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.