Kafka2MySQL is a Go-based project that demonstrates how to consume messages from an Apache Kafka topic and store them in a MySQL database. It uses several powerful libraries including Watermill for message processing, GORM for ORM (Object-Relational Mapping), and Gin for HTTP server routing.

• Kafka2MySQL
  • Table of Contents
  • Features
  • Prerequisites
  • Installation
  • Usage
  • Project Structure
  • Libraries Used
  • Contributing
  • License

• Consume messages from a Kafka topic.
• Store consumed messages in a MySQL database.
• Expose a REST API to publish messages to the Kafka topic.
• Automatic schema migration for the MySQL database.

• Go 1.20+
• Docker and Docker Compose
• MySQL
• Apache Kafka

1. Clone the repository:

   git clone https://github.com/yourusername/kafka2mysql.git
   cd kafka2mysql

2. Install dependencies:

   go mod tidy

3. Set up MySQL and Kafka using Docker Compose: Ensure you have Docker and Docker Compose installed. Run the following command to start MySQL and Kafka:

   docker-compose up -d

1. Run the application:

   go run main.go

2. Publish a message to Kafka: Use an HTTP GET request to publish a message to Kafka.

   curl "http://localhost:8080/api/v1/kafka2mysql?message=HelloKafka"

3. Check MySQL: The message should be stored in the MySQL database.

kafka2mysql/
├── configs/
│   └── config.go            # Configuration loading logic
├── providers/
│   ├── gin.go               # Gin server setup
│   ├── mysql.go             # MySQL setup
│   └── watermill.go         # Watermill router setup
├── subscriber/
│   └── kafka_subscriber.go  # Kafka subscriber implementation
├── publisher/
│   └── kafka_publisher.go   # Kafka publisher implementation
├── main.go                  # Main entry point
└── go.mod                   # Go modules file

• Watermill: A library for building event-driven applications.
• GORM: An ORM library for Go.
• Gin: A web framework for Go.
• Go-JSON: A JSON library for Go, used for marshalling and unmarshalling JSON.

Contributions are welcome! Please submit a pull request or open an issue to discuss what you would like to change.

This project is licensed under the MIT License. See the LICENSE file for details.