confluentinc / common-docker Goto Github PK
View Code? Open in Web Editor NEWConfluent Commons with support for building and testing Docker images.
License: Apache License 2.0
Confluent Commons with support for building and testing Docker images.
License: Apache License 2.0
I noticed that we are currently installing both the JDK and the JRE to the image:
common-docker/base/Dockerfile.ubi8
Line 103 in c782a4f
Is there any particular reason why we need both?
Typically the JDK includes everything the JRE does plus additional build tools. It is unclear to me why we need to add both to the image
Kafka 3.7.0 is officially compatible with java 21 , it would be great to update the base of the common image to use the latest LTS java version
Please remove confluent/team from the codeowners file: https://github.com/confluentinc/common-docker/blob/master/.github/CODEOWNERS
there are 763 members of that team who don’t all need to be alerted on every update of every PR
Is there a plan for supporting ARM with these images? I see this issue and this MR in the old repo, but they haven't been active in a while.
Allow cub ensure-topic
cli to check if Topic exists only.
Allow flag to disable validation check on number of partitions
and replicationFactor
and other config.
My use case is this:
I have a number of consumer applications deployed to Kubernetes have init containers using cub ensure-topic
. They just need to check if the topic exists before they can start. They cannot all be changed every time the producer app changes the partitions
and replicationFactor
for the topic.
I have been trying to build cp platform images for ARM64v8 for my work M1. To that end, I'm trying to start with master so that I may make PRS from a known good state.
The master branch does not build.
cd ./kafka && ./gradlew install
cd ./common && mvn install
cd ./common-docker && mvn clean package -Pdocker -DskipTests
mvn clean package -Pdocker -DskipTests
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.version: 12.0
[INFO] os.detected.version.major: 12
[INFO] os.detected.version.minor: 0
[INFO] os.detected.classifier: osx-aarch_64
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] common-docker [pom]
[INFO] utility-belt [jar]
[INFO] docker-utils [jar]
[INFO] cp-base-new [pom]
[INFO] cp-jmxterm [pom]
[INFO]
[INFO] ---------------------< io.confluent:common-docker >---------------------
[INFO] Building common-docker 7.2.0-0 [1/5]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ common-docker ---
[INFO] Deleting /Users/jackcviers/development/common-docker/target
[INFO]
[INFO] --- buildnumber-maven-plugin:1.4:create (default) @ common-docker ---
[INFO] Executing: /bin/sh -c cd '/Users/jackcviers/development/common-docker' && 'git' 'rev-parse' '--verify' 'HEAD'
[INFO] Working directory: /Users/jackcviers/development/common-docker
[INFO] Storing buildNumber: 8c815cf9cb5274ec045f24f192c1aff20d673eb9 at timestamp: 1638986737240
[INFO] Storing buildScmBranch: master
[INFO]
[INFO] --- maven-checkstyle-plugin:3.1.1:check (validate) @ common-docker ---
[INFO] Starting audit...
Audit done.
[INFO] You have 0 Checkstyle violations.
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ common-docker ---
[INFO]
[INFO] --- maven-dependency-plugin:3.2.0:unpack-dependencies (unpack-assembly-plugin-boilerplate) @ common-docker ---
[INFO] Unpacking /Users/jackcviers/.m2/repository/io/confluent/assembly-plugin-boilerplate/7.2.0-112/assembly-plugin-boilerplate-7.2.0-112-resources.zip to /Users/jackcviers/development/common-docker/target/dependency with includes "" and excludes ""
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ common-docker ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.2.0:test-jar (default) @ common-docker ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /Users/jackcviers/development/common-docker/target/common-docker-7.2.0-0-tests.jar
[INFO]
[INFO] --- maven-assembly-plugin:3.3.0:single (make-assembly-for-docker) @ common-docker ---
[INFO] Assemblies have been skipped per configuration of the skipAssembly parameter.
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:java (create-licenses-for-docker) @ common-docker ---
[WARNING] The POM for io.confluent:licenses:jar:7.2.0-4 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for common-docker 7.2.0-0:
[INFO]
[INFO] common-docker ...................................... FAILURE [ 1.602 s]
[INFO] utility-belt ....................................... SKIPPED
[INFO] docker-utils ....................................... SKIPPED
[INFO] cp-base-new ........................................ SKIPPED
[INFO] cp-jmxterm ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.756 s
[INFO] Finished at: 2021-12-08T12:05:38-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (create-licenses-for-docker) on project common-docker: Execution create-licenses-for-docker of goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java failed: Plugin org.codehaus.mojo:exec-maven-plugin:1.5.0 or one of its dependencies could not be resolved: io.confluent:licenses:jar:7.2.0-4 was not found in https://packages.confluent.io/maven/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of confluent has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
Compilation exited abnormally with code 1 at Wed Dec 8 12:05:38
cd <kafka clone dir> && ./gradlew install
cd <common clone dir> && mvn install
mvn clean install
cd <common docker clone dir>/base
mvn package -Pdocker -DskipTests
mvn package -Pdocker -DskipTests
[INFO] Scanning for projects...
Downloading from confluent: http://packages.confluent.io/maven/io/confluent/common/maven-metadata.xml
Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/common/maven-metadata.xml
Downloaded from confluent: http://packages.confluent.io/maven/io/confluent/common/maven-metadata.xml (2.1 kB at 14 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.version: 12.0
[INFO] os.detected.version.major: 12
[INFO] os.detected.version.minor: 0
[INFO] os.detected.classifier: osx-aarch_64
[INFO]
[INFO] ----------------------< io.confluent:cp-base-new >----------------------
[INFO] Building cp-base-new 7.2.0-0
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ cp-base-new ---
[INFO] Deleting /Users/jackcviers/development/common-docker/base/target
[INFO]
[INFO] --- buildnumber-maven-plugin:1.4:create (default) @ cp-base-new ---
[INFO] Executing: /bin/sh -c cd '/Users/jackcviers/development/common-docker/base' && 'git' 'rev-parse' '--verify' 'HEAD'
[INFO] Working directory: /Users/jackcviers/development/common-docker/base
[INFO] Storing buildNumber: 8c815cf at timestamp: 1638987687941
[INFO] Storing buildScmBranch: master
[INFO]
[INFO] --- maven-checkstyle-plugin:3.1.1:check (validate) @ cp-base-new ---
[INFO] Starting audit...
Audit done.
[INFO] You have 0 Checkstyle violations.
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ cp-base-new ---
[INFO]
[INFO] --- maven-dependency-plugin:3.2.0:unpack-dependencies (unpack-assembly-plugin-boilerplate) @ cp-base-new ---
[INFO] Unpacking /Users/jackcviers/.m2/repository/io/confluent/assembly-plugin-boilerplate/7.2.0-112/assembly-plugin-boilerplate-7.2.0-112-resources.zip to /Users/jackcviers/development/common-docker/base/target/dependency with includes "" and excludes ""
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:copy-resources (copy-resources) @ cp-base-new ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ cp-base-new ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.2.0:test-jar (default) @ cp-base-new ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /Users/jackcviers/development/common-docker/base/target/cp-base-new-7.2.0-0-tests.jar
[INFO]
[INFO] --- maven-assembly-plugin:3.3.0:single (make-assembly-for-docker) @ cp-base-new ---
[INFO] Reading assembly descriptor: target/dependency/assembly-plugin-boilerplate-7.2.0-112/common-docker-package.xml
[WARNING] Cannot include project artifact: io.confluent:cp-base-new:pom:7.2.0-0; it doesn't have an associated file or directory.
[INFO] Copying files to /Users/jackcviers/development/common-docker/base/target/cp-base-new-7.2.0-0-package
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:java (create-licenses-for-docker) @ cp-base-new ---
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/licenses/7.2.0-4/licenses-7.2.0-4.pom
Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/licenses/7.2.0-4/licenses-7.2.0-4.pom
[WARNING] The POM for io.confluent:licenses:jar:7.2.0-4 is missing, no dependency information available
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/licenses/7.2.0-4/licenses-7.2.0-4.jar
Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/licenses/7.2.0-4/licenses-7.2.0-4.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.564 s
[INFO] Finished at: 2021-12-08T12:21:30-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (create-licenses-for-docker) on project cp-base-new: Execution create-licenses-for-docker of goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java failed: Plugin org.codehaus.mojo:exec-maven-plugin:1.5.0 or one of its dependencies could not be resolved: Could not find artifact io.confluent:licenses:jar:7.2.0-4 in confluent (https://packages.confluent.io/maven/) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
## Questions
1. What is the build process for these images?
2. What project do I need to clone in order to build/obtain `io.confluent:licenses:jar:7.2.0-4`?
3. `mvn -U <commands>` produces the same result that the pom is missing for `licenses`. So... what is going wrong here?
5.5.x needs to use Java8u231 or later to satisfy vulnerability scans.
https://www.tenable.com/plugins/nessus/130010
I see a cp-base-new has deb9 but cp-zookeeper and cp-kafka aren't based on it
Kafka 3.1.0 is officially compatible with java 17 , it would be great to update the base of the common image to use the latest LTS java version
Hi,
Is https://github.com/confluentinc/confluent-docker-utils moved to this repo ? We are unable to access https://github.com/confluentinc/confluent-docker-utils, throwing 404.
Could you check ?
Thanks,
Lenin
Hi Confluent team,
There is a vulnerability in the 6.2.2 and 6.2.3 images that is preventing us from being able to use them. The vulnerability comes from the UBI base image and it seems like the base image has already been patched: https://github.com/confluentinc/common-docker/blob/6.2.2-post/pom.xml#L35
Would it be possible to rebuild this image so it can take advantage of the patched UBI image?
Thanks!
I am trying to volume mount the kafka data dir back to my centos host. This in my docker-compose.yml:
volumes:
- /var/lib/kafka1:/var/lib/kafka
Unfortunately the files in the container are not making it back to my host:
## files in the container
[root@ip-172-31-41-37 kafka1]# docker exec -it kafka1 ls -lat /var/lib/kafka/data/
total 36
drwxrwxrwx. 131 appuser appuser 8192 Oct 23 20:13 .
-rw-r--r--. 1 appuser appuser 88 Oct 23 20:13 meta.properties
drwxr-xr-x. 2 appuser appuser 178 Oct 23 20:13 _confluent_balancer_broker_samples-30
## files in on the host, non existent
[root@ip-172-31-41-37 kafka1]# ls -lat /var/lib/kafka1/data/
total 0
drwxr-xr-x. 2 12345 root 6 Oct 23 17:36 .
drwxr-xr-x. 3 12345 root 18 Oct 23 17:36 ..
Now according to these docs:
https://docs.confluent.io/current/installation/docker/operations/external-volumes.html#data-volumes-for-kafka-and-zk
We say to make sure the container user should have permissions to write to the directory on the host...
Unfortunately, I do not have a good way of creating the user appuser
with uid = 1000 on my centos machine, because that UID is already taken
## on centos box
[root@ip-172-31-41-37 kafka1]# id centos
uid=1000(centos) gid=1000(centos) groups=1000(centos),4(adm),10(wheel),190(systemd-journal)
## appuser uid
[root@ip-172-31-41-37 kafka1]# docker exec -it kafka1 id
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
I'm wondering if it would be possible to change this line:
https://github.com/confluentinc/common-docker/blob/master/base/Dockerfile.ubi8#L66
to
&& useradd -u 1111 --no-log-init --create-home --shell /bin/bash appuser \
or some arbitrary uid that does not equal 1000
ORR if there is some other way to make my volume mounts easier. I'm all ears
I'm using version 6.0.11 of the image in my project. During testing the "trivy" utility discovered a vulnerability.
$ trivy image confluentinc/cp-kafka:6.0.11
2023-02-06T18:25:22.718+0300 INFO Vulnerability scanning is enabled
2023-02-06T18:25:22.719+0300 INFO Secret scanning is enabled
2023-02-06T18:25:22.719+0300 INFO If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2023-02-06T18:25:22.719+0300 INFO Please see also https://aquasecurity.github.io/trivy/v0.37/docs/secret/scanning/#recommendation for faster secret detection
2023-02-06T18:25:30.121+0300 INFO Detected OS: redhat
2023-02-06T18:25:30.121+0300 INFO Detecting RHEL/CentOS vulnerabilities...
2023-02-06T18:25:30.149+0300 INFO Number of language-specific files: 2
2023-02-06T18:25:30.149+0300 INFO Detecting jar vulnerabilities...
2023-02-06T18:25:30.153+0300 INFO Detecting python-pkg vulnerabilities...
confluentinc/cp-kafka:6.0.11 (redhat 8.7)
Total: 1 (HIGH: 1, CRITICAL: 0)
┌─────────┬────────────────┬──────────┬───────────────────┬───────────────┬─────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├─────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────┤
│ libksba │ CVE-2022-47629 │ HIGH │ 1.3.5-8.el8_6 │ │ libksba: integer overflow to code execution │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-47629 │
└─────────┴────────────────┴──────────┴───────────────────┴───────────────┴─────────────────────────────────────────────┘
2023-02-06T18:25:30.219+0300 INFO Table result includes only package filenames. Use '--format json' option to get the full path to the package file.
Python (python-pkg)
Total: 1 (HIGH: 1, CRITICAL: 0)
┌───────────────────────┬────────────────┬──────────┬───────────────────┬───────────────┬───────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├───────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼───────────────────────────────────────────────────────┤
│ setuptools (METADATA) │ CVE-2022-40897 │ HIGH │ 50.3.2 │ 65.5.1 │ pypa-setuptools: Regular Expression Denial of Service │
│ │ │ │ │ │ (ReDoS) in package_index.py │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-40897 │
└───────────────────────┴────────────────┴──────────┴───────────────────┴───────────────┴───────────────────────────────────────────────────────┘
I need to update python setuptools with the new latest version (67.*). And I want to see my changes in the new image 6.0.12 for fixing the vulnerability. I have prepared changes. Which branch can I pull request it?
Hi
I was trying to use the secret provider for Azure Key Vault from lenses.io
https://github.com/lensesio/secret-provider
https://github.com/lensesio/secret-provider/releases/tag/2.1.6
I added the jar file to the directory used
COPY secret-provider-2.1.6-all.jar /usr/share/java/cp-base-new/
When I start the container (confluentinc/cp-kafka-connect-base:latest) I get the following exception:
===> Check if Kafka is healthy ...
Using log4j config /etc/cp-base-new/log4j.properties
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/connect/errors/ConnectException
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:419)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:408)
at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:577)
at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:521)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:112)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:146)
at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:235)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:144)
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:136)
at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:149)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.errors.ConnectException
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 13 more
When I add connect-api-3.4.0.jar to the directory this exception does not occur.
Maybe this class was moved in the past, would probably be good to include the file by default.
Recent changes to QEMU were released and it now supports hvf accelerator for running AArch64 guests.
This might solve the problem reported at #117.
Source:
QEMU 6.2 Changelog
The way I tested it was by installing QEMU 6.2 through homebrew and changing qemu-system-aarch64 (at /System/Volumes/Data/Users/$USER/Library/Containers/com.docker.docker/Data/tasks/qemu-system-aarch64
) configuration to point to the new version.
Please, could you help verifying if the new release of QEMU can help at #117
Thanks for your attention
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.