Kafka2MySQL is a Go-based project that demonstrates how to consume messages from an Apache Kafka topic and store them in a MySQL database. It uses several powerful libraries including Watermill for message processing, GORM for ORM (Object-Relational Mapping), and Gin for HTTP server routing.
- Consume messages from a Kafka topic.
- Store consumed messages in a MySQL database.
- Expose a REST API to publish messages to the Kafka topic.
- Automatic schema migration for the MySQL database.
- Go 1.20+
- Docker and Docker Compose
- MySQL
- Apache Kafka
-
Clone the repository:
git clone https://github.com/yourusername/kafka2mysql.git cd kafka2mysql
-
Install dependencies:
go mod tidy
-
Set up MySQL and Kafka using Docker Compose: Ensure you have Docker and Docker Compose installed. Run the following command to start MySQL and Kafka:
docker-compose up -d
-
Run the application:
go run main.go
-
Publish a message to Kafka: Use an HTTP GET request to publish a message to Kafka.
curl "http://localhost:8080/api/v1/kafka2mysql?message=HelloKafka"
-
Check MySQL: The message should be stored in the MySQL database.
kafka2mysql/
├── configs/
│ └── config.go # Configuration loading logic
├── providers/
│ ├── gin.go # Gin server setup
│ ├── mysql.go # MySQL setup
│ └── watermill.go # Watermill router setup
├── subscriber/
│ └── kafka_subscriber.go # Kafka subscriber implementation
├── publisher/
│ └── kafka_publisher.go # Kafka publisher implementation
├── main.go # Main entry point
└── go.mod # Go modules file
- Watermill: A library for building event-driven applications.
- GORM: An ORM library for Go.
- Gin: A web framework for Go.
- Go-JSON: A JSON library for Go, used for marshalling and unmarshalling JSON.
Contributions are welcome! Please submit a pull request or open an issue to discuss what you would like to change.
This project is licensed under the MIT License. See the LICENSE file for details.