confluentinc / common-docker Goto Github PK

Hi,

Is https://github.com/confluentinc/confluent-docker-utils moved to this repo ? We are unable to access https://github.com/confluentinc/confluent-docker-utils, throwing 404.

Could you check ?

Thanks,
Lenin

Allow cub ensure-topic cli to check if Topic exists only.
Allow flag to disable validation check on number of partitions and replicationFactor and other config.

My use case is this:
I have a number of consumer applications deployed to Kubernetes have init containers using cub ensure-topic. They just need to check if the topic exists before they can start. They cannot all be changed every time the producer app changes the partitions and replicationFactor for the topic.

I have been trying to build cp platform images for ARM64v8 for my work M1. To that end, I'm trying to start with master so that I may make PRS from a known good state.

The master branch does not build.

Steps to reproduce

1. Fork and clone confluentinc/kafka
2. cd ./kafka && ./gradlew install
3. Fork and clone confluentinc/common
4. cd ./common && mvn install
5. Fork and clone confluentinc/common-docker
6. cd ./common-docker && mvn clean package -Pdocker -DskipTests

Results

mvn clean package -Pdocker -DskipTests 
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.version: 12.0
[INFO] os.detected.version.major: 12
[INFO] os.detected.version.minor: 0
[INFO] os.detected.classifier: osx-aarch_64
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] common-docker                                                      [pom]
[INFO] utility-belt                                                       [jar]
[INFO] docker-utils                                                       [jar]
[INFO] cp-base-new                                                        [pom]
[INFO] cp-jmxterm                                                         [pom]
[INFO] 
[INFO] ---------------------< io.confluent:common-docker >---------------------
[INFO] Building common-docker 7.2.0-0                                     [1/5]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ common-docker ---
[INFO] Deleting /Users/jackcviers/development/common-docker/target
[INFO] 
[INFO] --- buildnumber-maven-plugin:1.4:create (default) @ common-docker ---
[INFO] Executing: /bin/sh -c cd '/Users/jackcviers/development/common-docker' && 'git' 'rev-parse' '--verify' 'HEAD'
[INFO] Working directory: /Users/jackcviers/development/common-docker
[INFO] Storing buildNumber: 8c815cf9cb5274ec045f24f192c1aff20d673eb9 at timestamp: 1638986737240
[INFO] Storing buildScmBranch: master
[INFO] 
[INFO] --- maven-checkstyle-plugin:3.1.1:check (validate) @ common-docker ---
[INFO] Starting audit...
Audit done.
[INFO] You have 0 Checkstyle violations.
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ common-docker ---
[INFO] 
[INFO] --- maven-dependency-plugin:3.2.0:unpack-dependencies (unpack-assembly-plugin-boilerplate) @ common-docker ---
[INFO] Unpacking /Users/jackcviers/.m2/repository/io/confluent/assembly-plugin-boilerplate/7.2.0-112/assembly-plugin-boilerplate-7.2.0-112-resources.zip to /Users/jackcviers/development/common-docker/target/dependency with includes "" and excludes ""
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ common-docker ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:3.2.0:test-jar (default) @ common-docker ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /Users/jackcviers/development/common-docker/target/common-docker-7.2.0-0-tests.jar
[INFO] 
[INFO] --- maven-assembly-plugin:3.3.0:single (make-assembly-for-docker) @ common-docker ---
[INFO] Assemblies have been skipped per configuration of the skipAssembly parameter.
[INFO] 
[INFO] --- exec-maven-plugin:1.5.0:java (create-licenses-for-docker) @ common-docker ---
[WARNING] The POM for io.confluent:licenses:jar:7.2.0-4 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for common-docker 7.2.0-0:
[INFO] 
[INFO] common-docker ...................................... FAILURE [  1.602 s]
[INFO] utility-belt ....................................... SKIPPED
[INFO] docker-utils ....................................... SKIPPED
[INFO] cp-base-new ........................................ SKIPPED
[INFO] cp-jmxterm ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.756 s
[INFO] Finished at: 2021-12-08T12:05:38-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (create-licenses-for-docker) on project common-docker: Execution create-licenses-for-docker of goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java failed: Plugin org.codehaus.mojo:exec-maven-plugin:1.5.0 or one of its dependencies could not be resolved: io.confluent:licenses:jar:7.2.0-4 was not found in https://packages.confluent.io/maven/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of confluent has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

Compilation exited abnormally with code 1 at Wed Dec  8 12:05:38

Alternative steps attempted

1. Fork and clone confluentinc/kafka
2. cd <kafka clone dir> && ./gradlew install
3. Fork and clone confluentinc/common
4. cd <common clone dir> && mvn install
5. Fork and clone confluentinc/common-docker
6. `cd
7. mvn clean install
8. cd <common docker clone dir>/base
9. mvn package -Pdocker -DskipTests

Alternative Results

mvn package -Pdocker -DskipTests

[INFO] Scanning for projects...
Downloading from confluent: http://packages.confluent.io/maven/io/confluent/common/maven-metadata.xml
Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/common/maven-metadata.xml
Downloaded from confluent: http://packages.confluent.io/maven/io/confluent/common/maven-metadata.xml (2.1 kB at 14 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.version: 12.0
[INFO] os.detected.version.major: 12
[INFO] os.detected.version.minor: 0
[INFO] os.detected.classifier: osx-aarch_64
[INFO]
[INFO] ----------------------< io.confluent:cp-base-new >----------------------
[INFO] Building cp-base-new 7.2.0-0
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ cp-base-new ---
[INFO] Deleting /Users/jackcviers/development/common-docker/base/target
[INFO]
[INFO] --- buildnumber-maven-plugin:1.4:create (default) @ cp-base-new ---
[INFO] Executing: /bin/sh -c cd '/Users/jackcviers/development/common-docker/base' && 'git' 'rev-parse' '--verify' 'HEAD'
[INFO] Working directory: /Users/jackcviers/development/common-docker/base
[INFO] Storing buildNumber: 8c815cf at timestamp: 1638987687941
[INFO] Storing buildScmBranch: master
[INFO]
[INFO] --- maven-checkstyle-plugin:3.1.1:check (validate) @ cp-base-new ---
[INFO] Starting audit...
Audit done.
[INFO] You have 0 Checkstyle violations.
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ cp-base-new ---
[INFO]
[INFO] --- maven-dependency-plugin:3.2.0:unpack-dependencies (unpack-assembly-plugin-boilerplate) @ cp-base-new ---
[INFO] Unpacking /Users/jackcviers/.m2/repository/io/confluent/assembly-plugin-boilerplate/7.2.0-112/assembly-plugin-boilerplate-7.2.0-112-resources.zip to /Users/jackcviers/development/common-docker/base/target/dependency with includes "" and excludes ""
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:copy-resources (copy-resources) @ cp-base-new ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ cp-base-new ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.2.0:test-jar (default) @ cp-base-new ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /Users/jackcviers/development/common-docker/base/target/cp-base-new-7.2.0-0-tests.jar
[INFO]
[INFO] --- maven-assembly-plugin:3.3.0:single (make-assembly-for-docker) @ cp-base-new ---
[INFO] Reading assembly descriptor: target/dependency/assembly-plugin-boilerplate-7.2.0-112/common-docker-package.xml
[WARNING] Cannot include project artifact: io.confluent:cp-base-new:pom:7.2.0-0; it doesn't have an associated file or directory.
[INFO] Copying files to /Users/jackcviers/development/common-docker/base/target/cp-base-new-7.2.0-0-package
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:java (create-licenses-for-docker) @ cp-base-new ---
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/licenses/7.2.0-4/licenses-7.2.0-4.pom
Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/licenses/7.2.0-4/licenses-7.2.0-4.pom
[WARNING] The POM for io.confluent:licenses:jar:7.2.0-4 is missing, no dependency information available
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/licenses/7.2.0-4/licenses-7.2.0-4.jar
Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/licenses/7.2.0-4/licenses-7.2.0-4.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.564 s
[INFO] Finished at: 2021-12-08T12:21:30-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (create-licenses-for-docker) on project cp-base-new: Execution create-licenses-for-docker of goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java failed: Plugin org.codehaus.mojo:exec-maven-plugin:1.5.0 or one of its dependencies could not be resolved: Could not find artifact io.confluent:licenses:jar:7.2.0-4 in confluent (https://packages.confluent.io/maven/) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

## Questions

1. What is the build process for these images?
2. What project do I need to clone in order to build/obtain `io.confluent:licenses:jar:7.2.0-4`?
3. `mvn -U <commands>` produces the same result that the pom is missing for `licenses`. So... what is going wrong here?

I am trying to volume mount the kafka data dir back to my centos host. This in my docker-compose.yml:

    volumes:
    - /var/lib/kafka1:/var/lib/kafka

Unfortunately the files in the container are not making it back to my host:

## files in the container
[root@ip-172-31-41-37 kafka1]# docker exec -it kafka1 ls -lat /var/lib/kafka/data/
total 36
drwxrwxrwx. 131 appuser appuser 8192 Oct 23 20:13 .
-rw-r--r--.   1 appuser appuser   88 Oct 23 20:13 meta.properties
drwxr-xr-x.   2 appuser appuser  178 Oct 23 20:13 _confluent_balancer_broker_samples-30

## files in on the host, non existent
[root@ip-172-31-41-37 kafka1]# ls -lat /var/lib/kafka1/data/
total 0
drwxr-xr-x. 2 12345 root  6 Oct 23 17:36 .
drwxr-xr-x. 3 12345 root 18 Oct 23 17:36 ..

Now according to these docs:
https://docs.confluent.io/current/installation/docker/operations/external-volumes.html#data-volumes-for-kafka-and-zk

We say to make sure the container user should have permissions to write to the directory on the host...

Unfortunately, I do not have a good way of creating the user appuser with uid = 1000 on my centos machine, because that UID is already taken

## on centos box
[root@ip-172-31-41-37 kafka1]# id centos
uid=1000(centos) gid=1000(centos) groups=1000(centos),4(adm),10(wheel),190(systemd-journal)

## appuser uid
[root@ip-172-31-41-37 kafka1]# docker exec -it kafka1 id
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)

I'm wondering if it would be possible to change this line:
https://github.com/confluentinc/common-docker/blob/master/base/Dockerfile.ubi8#L66

to

    && useradd -u 1111 --no-log-init --create-home --shell /bin/bash appuser \

or some arbitrary uid that does not equal 1000

ORR if there is some other way to make my volume mounts easier. I'm all ears

Kafka 3.1.0 is officially compatible with java 17 , it would be great to update the base of the common image to use the latest LTS java version

5.5.x needs to use Java8u231 or later to satisfy vulnerability scans.
https://www.tenable.com/plugins/nessus/130010
I see a cp-base-new has deb9 but cp-zookeeper and cp-kafka aren't based on it

Hi Confluent team,

There is a vulnerability in the 6.2.2 and 6.2.3 images that is preventing us from being able to use them. The vulnerability comes from the UBI base image and it seems like the base image has already been patched: https://github.com/confluentinc/common-docker/blob/6.2.2-post/pom.xml#L35

Would it be possible to rebuild this image so it can take advantage of the patched UBI image?

Thanks!

Is there a plan for supporting ARM with these images? I see this issue and this MR in the old repo, but they haven't been active in a while.

Hi
I was trying to use the secret provider for Azure Key Vault from lenses.io
https://github.com/lensesio/secret-provider
https://github.com/lensesio/secret-provider/releases/tag/2.1.6

I added the jar file to the directory used
COPY secret-provider-2.1.6-all.jar /usr/share/java/cp-base-new/

When I start the container (confluentinc/cp-kafka-connect-base:latest) I get the following exception:

===> Check if Kafka is healthy ...
Using log4j config /etc/cp-base-new/log4j.properties
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/connect/errors/ConnectException
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:398)
        at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:419)
        at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:408)
        at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:577)
        at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:521)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:112)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:146)
        at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:235)
        at org.apache.kafka.clients.admin.Admin.create(Admin.java:144)
        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
        at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:136)
        at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:149)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.errors.ConnectException
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 13 more

When I add connect-api-3.4.0.jar to the directory this exception does not occur.

Maybe this class was moved in the past, would probably be good to include the file by default.

I'm using version 6.0.11 of the image in my project. During testing the "trivy" utility discovered a vulnerability.

$ trivy image confluentinc/cp-kafka:6.0.11
2023-02-06T18:25:22.718+0300	INFO	Vulnerability scanning is enabled
2023-02-06T18:25:22.719+0300	INFO	Secret scanning is enabled
2023-02-06T18:25:22.719+0300	INFO	If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2023-02-06T18:25:22.719+0300	INFO	Please see also https://aquasecurity.github.io/trivy/v0.37/docs/secret/scanning/#recommendation for faster secret detection
2023-02-06T18:25:30.121+0300	INFO	Detected OS: redhat
2023-02-06T18:25:30.121+0300	INFO	Detecting RHEL/CentOS vulnerabilities...
2023-02-06T18:25:30.149+0300	INFO	Number of language-specific files: 2
2023-02-06T18:25:30.149+0300	INFO	Detecting jar vulnerabilities...
2023-02-06T18:25:30.153+0300	INFO	Detecting python-pkg vulnerabilities...

confluentinc/cp-kafka:6.0.11 (redhat 8.7)

Total: 1 (HIGH: 1, CRITICAL: 0)

┌─────────┬────────────────┬──────────┬───────────────────┬───────────────┬─────────────────────────────────────────────┐
│ Library │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                    Title                    │
├─────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────┤
│ libksba │ CVE-2022-47629 │ HIGH     │ 1.3.5-8.el8_6     │               │ libksba: integer overflow to code execution │
│         │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-47629  │
└─────────┴────────────────┴──────────┴───────────────────┴───────────────┴─────────────────────────────────────────────┘
2023-02-06T18:25:30.219+0300	INFO	Table result includes only package filenames. Use '--format json' option to get the full path to the package file.

Python (python-pkg)

Total: 1 (HIGH: 1, CRITICAL: 0)

┌───────────────────────┬────────────────┬──────────┬───────────────────┬───────────────┬───────────────────────────────────────────────────────┐
│        Library        │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                         Title                         │
├───────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼───────────────────────────────────────────────────────┤
│ setuptools (METADATA) │ CVE-2022-40897 │ HIGH     │ 50.3.2            │ 65.5.1        │ pypa-setuptools: Regular Expression Denial of Service │
│                       │                │          │                   │               │ (ReDoS) in package_index.py                           │
│                       │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-40897            │
└───────────────────────┴────────────────┴──────────┴───────────────────┴───────────────┴───────────────────────────────────────────────────────┘

I need to update python setuptools with the new latest version (67.*). And I want to see my changes in the new image 6.0.12 for fixing the vulnerability. I have prepared changes. Which branch can I pull request it?

Please remove confluent/team from the codeowners file: https://github.com/confluentinc/common-docker/blob/master/.github/CODEOWNERS

there are 763 members of that team who don’t all need to be alerted on every update of every PR

Recent changes to QEMU were released and it now supports hvf accelerator for running AArch64 guests.
This might solve the problem reported at #117.

Source:
QEMU 6.2 Changelog

The way I tested it was by installing QEMU 6.2 through homebrew and changing qemu-system-aarch64 (at /System/Volumes/Data/Users/$USER/Library/Containers/com.docker.docker/Data/tasks/qemu-system-aarch64) configuration to point to the new version.

Please, could you help verifying if the new release of QEMU can help at #117

Thanks for your attention