Git Product home page Git Product logo

embedded-kafka's Introduction

Overview

Problem: Integration tests using Kafka containers face high memory consumption and long startup times, slowing down the testing process and increasing resource costs.
Solution: Using a native image for EmbeddedKafka with GraalVM.
Status: Completed, article is published, currently using.

Native Embedded kafka in Docker Container

Utilizes org.springframework.kafka.test.EmbeddedKafkaBroker and GraalVM to provide native kafka in docker.

Article

In the article, I want to share my experience with creating a native image for EmbeddedKafka using GraalVM. Utilizing this image in integration tests not only speeds up the execution of test scenarios but also reduces memory consumption. Interestingly, when compared to using confluentinc/cp-kafka in Testcontainers, there is a noticeable difference in both speed and memory usage — and it’s not in favor of the latter.

Article on Habr - Как сократить потребление памяти в интеграционных тестах с Kafka с помощью GraalVM Article on Medium - How to Reduce Memory Consumption in Integration Tests with Kafka Using GraalVM

Usage

Add dependency

implementation 'pw.avvero:emk-testcontainers:1.0.0'

Create a EmbeddedKafkaContainer to use it in your tests:

EmbeddedKafkaContainer kafka = new EmbeddedKafkaContainer("avvero/emk-native:1.0.0"); // OR avvero/emk:latest

Improved Testcontainers Support in Spring Boot 3.1

The new @ServiceConnection annotation can be used on the container instance fields of your tests. See example in tests.

Refer to article in Spring blog to get more information.

Examples for using Kafka in integration tests

One can find examples for using Kafka in integration tests:

  • EmbeddedKafka - module example-spring-embedded-kafka
  • Testcontainers - module example-testcontainers
  • Embedded Kafka Native - module example-embedded-kafka-container

Docker Image

Ready to use Docker image is hosted on Docker Hub and can be pulled using the following command:

docker pull avvero/emk

or native image for linux/arm64

docker pull avvero/emk-native

Build container with java

Run command: make emk-docker-build

Build container with native

Run command: emk-docker-build-native

Native build details

Include Reachability Metadata

  1. Setup graalvm: https://www.graalvm.org/latest/docs/getting-started
  2. Include Reachability Metadata Using the Native Image Gradle Plugin
  3. Run make emk-run-with-agent
  4. Run activity over broker

embedded-kafka's People

Contributors

avvero avatar

Stargazers

avatar avatar avatar

Watchers

avatar

embedded-kafka's Issues

Container failed to load

Version: 1.0.1
GraalVm Version: Community 21.0.1
IDE: intelliJ

I am trying to use the Embedded Kafka Container but it always failed to load with test container don´t really know why Code:

KafkaConfiguration.kt

@Suppress("UtilityClassWithPublicConstructor")
@TestConfiguration
class KafkaTestConfiguration {
    companion object {
        init {
            startContainer()
        }
        private fun startContainer() {
            val kafka = EmbeddedKafkaContainer("avvero/emk:latest").apply {
                this.start()
            }

            println(kafka.bootstrapServers)

            System.setProperty("spring.kafka.bootstrap-servers", kafka.bootstrapServers)
        }
    }
}

Integration Test Annotation

@SpringBootTest
@AutoConfigureMockMvc
@EnableCaching
@Import(
    MongoTestConfiguration::class,
    RedisTestConfiguration::class,
    KafkaTestConfiguration::class,
)
annotation class IntegrationTest

I basically put that annotation on my test and get the following stackTrace

java.lang.IllegalStateException: Failed to load ApplicationContext for ...

Caused by: java.lang.IllegalStateException: Unable to load cache item
	at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:75)
	at org.springframework.cglib.core.internal.LoadingCache.get(LoadingCache.java:34)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData.get(AbstractClassGenerator.java:130)
	at org.springframework.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:317)
	at org.springframework.cglib.proxy.Enhancer.createHelper(Enhancer.java:562)
	at org.springframework.cglib.proxy.Enhancer.createClass(Enhancer.java:407)
	at org.springframework.context.annotation.ConfigurationClassEnhancer.createClass(ConfigurationClassEnhancer.java:138)
	at org.springframework.context.annotation.ConfigurationClassEnhancer.enhance(ConfigurationClassEnhancer.java:109)
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:533)
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanFactory(ConfigurationClassPostProcessor.java:310)
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:363)
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:153)
	at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:789)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:606)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:762)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:464)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:334)
	at org.springframework.boot.test.context.SpringBootContextLoader.lambda$loadContext$3(SpringBootContextLoader.java:137)
	at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:58)
	at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:46)
	at org.springframework.boot.SpringApplication.withHook(SpringApplication.java:1458)
	at org.springframework.boot.test.context.SpringBootContextLoader$ContextLoaderHook.run(SpringBootContextLoader.java:552)
	at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:137)
	at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:108)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:225)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:152)
	... 17 more
Caused by: java.lang.ExceptionInInitializerError
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:534)
	at java.base/java.lang.Class.forName(Class.java:513)
	at org.springframework.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:551)
	at org.springframework.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:371)
	at org.springframework.cglib.proxy.Enhancer.generate(Enhancer.java:575)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData.lambda$new$1(AbstractClassGenerator.java:107)
	at org.springframework.cglib.core.internal.LoadingCache.lambda$createEntry$1(LoadingCache.java:52)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:57)
	... 42 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image avvero/emk-native:latest
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:362)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:333)
	at co.doji.tradeinservice.config.KafkaTestConfiguration$Companion.startContainer(KafkaTestConfiguration.kt:15)
	at co.doji.tradeinservice.config.KafkaTestConfiguration$Companion.access$startContainer(KafkaTestConfiguration.kt:9)
	at co.doji.tradeinservice.config.KafkaTestConfiguration.<clinit>(KafkaTestConfiguration.kt:11)
	... 52 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:347)
	... 56 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:566)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:357)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 57 more
Caused by: org.testcontainers.shaded.org.awaitility.core.ConditionTimeoutException: org.testcontainers.containers.GenericContainer$$Lambda/0x00007af1686d97d8 expected the predicate to return <true> but it returned <false> for input of <InspectContainerResponse(args=[-c, while [ ! -f /testcontainers_start.sh ]; do echo 'Waiting for start kafka'; sleep 0.1; done; /testcontainers_start.sh], config=ContainerConfig(attachStderr=false, attachStdin=false, attachStdout=false, cmd=[/bin/sh, -c, while [ ! -f /testcontainers_start.sh ]; do echo 'Waiting for start kafka'; sleep 0.1; done; /testcontainers_start.sh], domainName=, entrypoint=null, env=[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin], exposedPorts=[2181/tcp, 8080/tcp, 9093/tcp], hostName=b8c3d766be77, image=avvero/emk-native:latest, labels={maintainer=avvero, org.testcontainers=true, org.testcontainers.lang=java, org.testcontainers.sessionId=99b5383b-c806-4d39-8b14-2799357a22c4, org.testcontainers.version=1.19.3}, macAddress=null, networkDisabled=null, onBuild=null, stdinOpen=false, portSpecs=null, stdInOnce=false, tty=false, user=, volumes=null, workingDir=, healthCheck=null), created=2024-03-14T21:12:26.321815917Z, driver=overlay2, execDriver=null, hostConfig=HostConfig(binds=[], blkioWeight=0, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, memorySwappiness=null, nanoCPUs=0, capAdd=null, capDrop=null, containerIDFile=, cpuPeriod=0, cpuRealtimePeriod=0, cpuRealtimeRuntime=0, cpuShares=0, cpuQuota=0, cpusetCpus=, cpusetMems=, devices=null, deviceCgroupRules=null, deviceRequests=null, diskQuota=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=[], groupAdd=null, ipcMode=shareable, cgroup=, links=[], logConfig=LogConfig(type=json-file, config={}), lxcConf=null, memory=0, memorySwap=0, memoryReservation=0, kernelMemory=null, networkMode=default, oomKillDisable=null, init=null, autoRemove=false, oomScoreAdj=0, portBindings={2181/tcp=[Lcom.github.dockerjava.api.model.Ports$Binding;@1221d607, 8080/tcp=[Lcom.github.dockerjava.api.model.Ports$Binding;@4b45c8e4, 9093/tcp=[Lcom.github.dockerjava.api.model.Ports$Binding;@3dc14f80}, privileged=false, publishAllPorts=false, readonlyRootfs=false, restartPolicy=no, ulimits=null, cpuCount=0, cpuPercent=0, ioMaximumIOps=0, ioMaximumBandwidth=0, volumesFrom=[], mounts=null, pidMode=, isolation=null, securityOpts=null, storageOpt=null, cgroupParent=, volumeDriver=, shmSize=67108864, pidsLimit=null, runtime=runc, tmpFs=null, utSMode=, usernsMode=, sysctls=null, consoleSize=[0, 0], cgroupnsMode=private), hostnamePath=/var/lib/docker/containers/b8c3d766be77813bd7fd79b335de55ea5626ddd53319f812ea5e79a3b98d6a18/hostname, hostsPath=/var/lib/docker/containers/b8c3d766be77813bd7fd79b335de55ea5626ddd53319f812ea5e79a3b98d6a18/hosts, logPath=/var/lib/docker/containers/b8c3d766be77813bd7fd79b335de55ea5626ddd53319f812ea5e79a3b98d6a18/b8c3d766be77813bd7fd79b335de55ea5626ddd53319f812ea5e79a3b98d6a18-json.log, id=b8c3d766be77813bd7fd79b335de55ea5626ddd53319f812ea5e79a3b98d6a18, sizeRootFs=null, sizeRw=null, imageId=sha256:8a8cfcba9fe6d94fe8738d0c12c7cc6551581a893882db61cc6b32071e6bccc3, mountLabel=, name=/agitated_easley, restartCount=0, networkSettings=NetworkSettings(bridge=, sandboxId=eabdb82008c217eb777e53176e9cd92967b1dfeedd25d21aeef468c8ce189f9c, hairpinMode=false, linkLocalIPv6Address=, linkLocalIPv6PrefixLen=0, ports={}, sandboxKey=/var/run/docker/netns/eabdb82008c2, secondaryIPAddresses=null, secondaryIPv6Addresses=null, endpointID=, gateway=, portMapping=null, globalIPv6Address=, globalIPv6PrefixLen=0, ipAddress=, ipPrefixLen=0, ipV6Gateway=, macAddress=, networks={bridge=ContainerNetwork(ipamConfig=null, links=[], aliases=null, networkID=ebaf6a86582c52ec338cec181ca38e566f92d618db58372b426bfb6ed63afaae, endpointId=, gateway=, ipAddress=, ipPrefixLen=0, ipV6Gateway=, globalIPv6Address=, globalIPv6PrefixLen=0, macAddress=)}), path=/bin/sh, processLabel=, resolvConfPath=/var/lib/docker/containers/b8c3d766be77813bd7fd79b335de55ea5626ddd53319f812ea5e79a3b98d6a18/resolv.conf, execIds=null, state=InspectContainerResponse.ContainerState(status=exited, running=false, paused=false, restarting=false, oomKilled=false, dead=false, pid=0, exitCode=1, error=, startedAt=2024-03-14T21:12:26.694195329Z, finishedAt=2024-03-14T21:12:26.696235759Z, health=null), volumes=null, volumesRW=null, node=null, mounts=[], graphDriver=GraphDriver(name=overlay2, data=GraphData(rootDir=null, deviceId=null, deviceName=null, deviceSize=null, dir=null)), platform=linux)> within 5 seconds.

Tried with all variations possible for the image name

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.