31z4 / storm-docker Goto Github PK
View Code? Open in Web Editor NEWDocker image packaging for Apache Storm
License: MIT License
Docker image packaging for Apache Storm
License: MIT License
Hi there!
Thanks you for creating this project! It was just what I was looking for to test upgrading to storm 1.0.1.
I've copied your docker-compose configuration and it seems to be running, but I am not able to connect to it from outside docker, even though the 6627 port is exposed. I'm running on Ubuntu with the latest docker-engine and docker-compose versions, and when I try a simple storm list
command, I get the below exception. Have you seen this before? Any idea what I am doing wrong?
Thanks for your help!
-Marc
$ which storm
/home/mzbyszynski/Applications/apache-storm-1.0.1/bin/storm
$ docker-compose -f src/docker/docker-compose.yml ps
Name Command State Ports
-------------------------------------------------------------------------------------
docker_nimbus_1 storm nimbus -c storm.zook ... Up 0.0.0.0:6627->6627/tcp
docker_supervisor_1 storm supervisor -c storm. ... Up
docker_zookeeper_1 /docker-entrypoint.sh zkSe ... Up 2181/tcp
$ storm remoteconfvalue nimbus.seeds
nimbus.seeds: [localhost]
$ storm remoteconfvalue nimbus.thrift.port
nimbus.thrift.port: 6627
$ storm localconfvalue nimbus.thrift.port
nimbus.thrift.port: 6627
t$ storm localconfvalue nimbus.seeds
nimbus.seeds: [localhost]
$ storm list
Running: /usr/lib/jvm/java-8-oracle/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/home/mzbyszynski/Applications/apache-storm-1.0.1 -Dstorm.log.dir=/home/mzbyszynski/Applications/apache-storm-1.0.1/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/mzbyszynski/Applications/apache-storm-1.0.1/lib/reflectasm-1.10.1.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/clojure-1.7.0.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/minlog-1.3.0.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/disruptor-3.3.2.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/log4j-slf4j-impl-2.1.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/asm-5.0.3.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/servlet-api-2.5.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/storm-rename-hack-1.0.1.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/log4j-api-2.1.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/log4j-over-slf4j-1.6.6.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/kryo-3.0.3.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/objenesis-2.1.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/storm-core-1.0.1.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/log4j-core-2.1.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/lib/slf4j-api-1.7.7.jar:/home/mzbyszynski/Applications/apache-storm-1.0.1/conf:/home/mzbyszynski/Applications/apache-storm-1.0.1/bin org.apache.storm.command.list
Exception in thread "main" java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.UnknownHostException: 28cb4180e92a
at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:106)
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:78)
at org.apache.storm.command.list$_main.invoke(list.clj:22)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at org.apache.storm.command.list.main(Unknown Source)
Caused by: org.apache.storm.thrift.transport.TTransportException: java.net.UnknownHostException: 28cb4180e92a
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
... 8 more
Caused by: java.net.UnknownHostException: 28cb4180e92a
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221)
... 11 more
Hi,
I tried to use your images with docker-compose you provide in README.
It seems to me that is it not enough to make containers dependent on each other, they must also link each other. Also, I had to change "nimbus.host" and "storm.zookeeper.servers" to the names of containers (which, in my case, are networkname_nimbus_1, networkname_supervisor_1 etc. and not the names of entries from docker-compose). Using docker-compose 1.7.0 and Docker 1.11.1 on Windows 7 host. I wonder if the naming convention depends on Docker version?
I got it working with sth like (docker-compose.yml placed in 'storm' directory):
version: '2'
services:
zookeeper:
image: 31z4/zookeeper:3.4.8
restart: always
nimbus:
image: 31z4/storm:1.0.2
command: storm nimbus -c storm.zookeeper.servers="[\"storm_zookeeper_1\"]" -c nimbus.host="storm_nimbus_1"
links:
- zookeeper
depends_on:
- zookeeper
restart: always
ports:
- 6627:6627
supervisor:
image: 31z4/storm:1.0.2
command: storm supervisor -c storm.zookeeper.servers="[\"storm_zookeeper_1\"]" -c nimbus.host="storm_nimbus_1"
links:
- nimbus
- zookeeper
depends_on:
- nimbus
- zookeeper
restart: always
Those images are needed, I hope they will soon become official ones :)
Edit: just learned that container names can be set with "container_name". That solves the problem with names dependent on parent directory.
Hey @31z4! I think we're probably due for a version bump? It looks like Storm upstream released 2.3.1 and 2.2.2. (Interestingly, no 1.x -- is that still an actively supported release line?)
I believe both of those include log4j version bumps, so maybe important? 🙈
The commit 9eae4ab changed the version of Java to 11, however by default Storm specifies a deprecated JVM option
2021-12-09 14:38:58.264 STDERR Thread-0 [INFO] Unrecognized VM option 'PrintGCDateStamps'
2021-12-09 14:38:58.264 STDERR Thread-1 [INFO] [0.000s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:artifacts/gc.log instead.
2021-12-09 14:38:58.269 STDERR Thread-0 [INFO] Error: Could not create the Java Virtual Machine.
2021-12-09 14:38:58.269 STDERR Thread-0 [INFO] Error: A fatal exception has occurred. Program will exit.
You should either revert the commit above and go back to Java 8 or remove the option from the configuration. This can be done by overriding the value of worker.childopts when calling Storm.
storm supervisor -c worker.childopts=-Xmx%HEAP-MEM%m
I tried bringing up a minimal docker cluster, using following command
List of Commands
docker run -d --restart always --name some-zookeeper zookeeper:3.4
When you go to http://localhst:8080/index.html, i am being greeted with a InternalServerError
Internal Server Error
java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59)
at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51)
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72)
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:69)
at backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:76)
at backtype.storm.ui.core$supervisor_summary.invoke(core.clj:578)
at backtype.storm.ui.core$fn__9652.invoke(core.clj:967)
at org.apache.storm.shade.compojure.core$make_route$fn__1829.invoke(core.clj:93)
at org.apache.storm.shade.compojure.core$if_route$fn__1817.invoke(core.clj:39)
at org.apache.storm.shade.compojure.core$if_method$fn__1810.invoke(core.clj:24)
at org.apache.storm.shade.compojure.core$routing$fn__1835.invoke(core.clj:106)
at clojure.core$some.invoke(core.clj:2515)
at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invoke(core.clj:626)
at org.apache.storm.shade.compojure.core$routes$fn__1839.invoke(core.clj:111)
at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__8930.invoke(json.clj:56)
at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__2889.invoke(multipart_params.clj:103)
at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__3620.invoke(reload.clj:22)
at backtype.storm.ui.core$catch_errors$fn__9723.invoke(core.clj:1061)
at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__2822.invoke(keyword_params.clj:27)
at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__2861.invoke(nested_params.clj:65)
at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__2794.invoke(params.clj:55)
at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__2889.invoke(multipart_params.clj:103)
at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3070.invoke(flash.clj:14)
at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3059.invoke(session.clj:43)
at org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__2990.invoke(cookies.clj:160)
at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__2707.invoke(servlet.clj:127)
at org.apache.storm.shade.ring.util.servlet$servlet$fn__2711.invoke(servlet.clj:136)
at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at backtype.storm.ui.helpers$x_frame_options_filter_handler$fn__3205.invoke(helpers.clj:191)
at backtype.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift7.transport.TSocket.open(TSocket.java:187)
at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48)
... 61 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift7.transport.TSocket.open(TSocket.java:182)
... 64 more
Everything works as expected with storm 1.0.2
I have pulled out the image for the storm and as per the listed steps , I have got the nimbus and storm UI working. But unfortuantely the while running the jar, the situation below is arised.
/usr/local/openjdk-8/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/apache-storm-2.1.0 -Dstorm.log.dir=/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 -Dstorm.conf.file= -cp /apache-storm-2.1.0/:/apache-storm-2.1.0/lib/:/apache-storm-2.1.0/extlib/*:/topology.jar:/conf:/apache-storm-2.1.0/bin: -Dstorm.jar=/topology.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.starter.ExclamationTopology
I am not even able to get access to the localhost:6627 even after deploying.
This is only a question.
Can you explain me why the configuration storm.yaml in /conf works, if the installation is in other path?
All of our topology jars are compiled and expected to run at JAVA-11 runtime, because of which we are not able to use the official docker image from this repo, Request you to upgrade JAVA from 8 to 11
This helps anyone who is using java-11 to fetch the official docker image instead of creating their own custom ones.
Currently, when we use the docker image provided by you, it fails to bring up our topologies with version mismatch errors.
Tell us what happens instead.
Please include as much relevant information as possible.
I followed the procedure listed on the Docker hub for Storm 1.2.1 to set up a minimum Storm cluster. But when I executed the command to submit a topology as the documentation suggests $ docker run --link some-nimbus:nimbus -it --rm -v $(pwd)/topology.jar:/topology.jar storm storm jar /topology.jar org.apache.storm.starter.WordCountTopology topology
. The error occurred in the console:
Error: Could not find or load main class org.apache.storm.starter.ExclamationTopology.
The jar file is compiled from the official master branch with version 1.2.1. I confirmed that the jar file in local current directory does include the org.apache.storm.starter.ExclamationTopology class with the correct path.
We want to deploy apache-storm as cluster in cluster hosts. More precisely we need have supervisor instances in all host-nodes. How we can deploy storm cluster with docker-compose file? We have docker-compose.yml file:
version: "3.6"
services:
zookeeper:
image: zookeeper
networks:
- esnet1
deploy:
mode: global
restart_policy:
condition: on-failure
nimbus:
image: storm
command: storm nimbus
volumes:
- logs:/logs
depends_on:
- zookeeper
networks:
- esnet1
ports:
- 6627:6627
deploy:
restart_policy:
condition: on-failure
supervisor:
image: storm
command: storm supervisor -c storm.log.dir="/logs"
depends_on:
- nimbus
- zookeeper
- elasticsearch
networks:
- esnet1
# volumes:
# - type: bind
# source: /data/storm
# target: /data
# - type: volume
# source: logs
# target: /logs
deploy:
mode: global
# replicas: 1
placement:
constraints:
- node.labels.processor==true
restart_policy:
condition: on-failure
ui:
image: storm
command: storm ui -c storm.log.dir="/logs"
volumes:
- logs:/log
networks:
- esnet1
depends_on:
- nimbus
ports:
- 8081:8080
deploy:
restart_policy:
condition: on-failure
networks:
esnet1:
attachable: true
volumes:
logs:
than we see next log in supervisor service:
Running: /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -server -Ddaemon.name=supervisor -Dstorm.options=storm.log.dir%3D%2Flogs -Dstorm.home=/apache-storm-1.2.1 -Dstorm.log.dir=/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /apache-storm-1.2.1/*:/apache-storm-1.2.1/lib/*:/apache-storm-1.2.1/extlib/*:/apache-storm-1.2.1/extlib-daemon/*:/conf -Xmx256m -Dlogfile.name=supervisor.log -Dlog4j.configurationFile=/apache-storm-1.2.1/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor
standard_init_linux.go:190: exec user process caused "exec format error"
If we made next change in docker-compose.yml:
supervisor:
...
deploy:
# mode: global
replicas: 1
...
than cluster deployed.
How we can deploy Storm cluster through docker-compose?
We deploy a testing environment as a stack of docker services multiple times a day. On Monday afternoon (Feb 22), this deployment broke due to the Java upgrade from 8 to 11 made in the image. We are using the 2.2.0 image. When this latest update went through, the 2.2.0 tag was updated and there is no tag pointing to the previous (working) version. This has actually happened a few times over the last couple years.
Currently I am seeing the following errors in our worker logs. It's not clear to me where these originate from right off hand. We'll have to investigate. But in the meantime, our deployment remains broken. If the previous tag was available, we could get it working again.
Unrecognized VM option 'PrintGCDateStamps'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
When updates are made to this image, the existing images should still be accessible. One way to do this would be to have tags like 2.2.0-1, 2.2.0-2, 2.2.0-3, etc. The 2.2.0 tag could still exist and point to the latest 2.2.0 version. This would allow users to point to 2.2.0-1 for example and not have to worry about an update breaking things; they can upgrade to a more recent tag after testing the changes to verify things still work or making the appropriate modifications.
Another option might be to maintain separate images/tags for major Java versions.
It would also be nice if the previous version (with Java 8) could be made available on Docker Hub again.
When a change is pushed, the existing tags are overwritten and there's no way to get back to a previous working version.
Have a Docker image that is built FROM storm:2.2.0 or a compose file that references storm:2.2.0. When that tag is replaced, this may break something that was previously working.
How to configure the timezone inside the Storm container? Currently the default timezone is UTC.
Do I have to extend the image and set the timezone using tzdata?
something like:
RUN echo "Asia/Kolkata" > /etc/timezone
RUN dpkg-reconfigure -f noninteractive tzdata
Please suggest.
org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts ["nimbus"]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:111)
at org.apache.storm.ui.core$all_topologies_summary.invoke(core.clj:509)
at org.apache.storm.ui.core$fn__9557.invoke(core.clj:1144)
at org.apache.storm.shade.compojure.core$make_route$fn__5962.invoke(core.clj:100)
at org.apache.storm.shade.compojure.core$if_route$fn__5950.invoke(core.clj:46)
at org.apache.storm.shade.compojure.core$if_method$fn__5943.invoke(core.clj:31)
at org.apache.storm.shade.compojure.core$routing$fn__5968.invoke(core.clj:113)
at clojure.core$some.invoke(core.clj:2570)
at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invoke(core.clj:632)
at org.apache.storm.shade.compojure.core$routes$fn__5972.invoke(core.clj:118)
at org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__8877.invoke(cors.clj:149)
at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__8824.invoke(json.clj:56)
at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6604.invoke(multipart_params.clj:118)
at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__7887.invoke(reload.clj:22)
at org.apache.storm.ui.helpers$requests_middleware$fn__6857.invoke(helpers.clj:50)
at org.apache.storm.ui.core$catch_errors$fn__9744.invoke(core.clj:1428)
at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__6524.invoke(keyword_params.clj:35)
at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__6567.invoke(nested_params.clj:84)
at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__6496.invoke(params.clj:64)
at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6604.invoke(multipart_params.clj:118)
at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__6819.invoke(flash.clj:35)
at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__6805.invoke(session.clj:98)
at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__6354.invoke(servlet.clj:127)
at org.apache.storm.shade.ring.util.servlet$servlet$fn__6358.invoke(servlet.clj:136)
at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__6950.invoke(helpers.clj:189)
at org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:748)
It will allow to see the logs from the Storm UI. Both logviewer
and storm
processes must be run under Supervisor
.
Before you file an issue here, please keep in mind that your issue may be not related to the image itself. Please make sure that it is, otherwise report the issue upstream.
Tell us what should happen.
Tell us what happens instead.
Tell us how to reproduce the issue.
Please include as much relevant information as possible.
no matching manifest for linux/arm64/v8 in the manifest list entries
hi,
when I submitted a topology to nimbus remotely using nimbusclient, I got
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.UnknownHostException: 6529caa5230d
ps 6529caa5230d is the docker container Id, I want to know how to set it,
hi~
I used official repositpry storm docker image and docker-compose build a storm cluster.
but used storm ui,I can't visit every nodes log,because all log visit port is 8000,but only one port can mapping on host computer。 so how to configuation storm ui with storm cluster on docker ? please give me some suggestion, thank you !
Hi,
The Storm docker package is building and running fine at my end for ARM64v8 architecture.
Branch used to build is:
Stable: latest: 1.2.2
Please suggest, what needs to be done to raise the request to add support for ARM64v8 arch at official docker library too.
Regards,
The Storm 2.1.0 image is not available in docker-hub because Storm 2.1.0 image multiarch build is failing:
https://doi-janky.infosiftr.net/job/multiarch/view/images/view/storm/job/amd64/job/storm/199/console
+ wget -q http://www.apache.org/dist/storm/apache-storm-2.1.0/apache-storm-2.1.0.tar.gz
Removing intermediate container 9e3c26126923
The command '/bin/sh -c set -eux; wget -q "http://www.apache.org/dist/storm/$DISTRO_NAME/$DISTRO_NAME.tar.gz"; wget -q "http://www.apache.org/dist/storm/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc"; export GNUPGHOME="$(mktemp -d)"; gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" || gpg --keyserver pgp.mit.edu --recv-keys "$GPG_KEY" || gpg --keyserver keyserver.pgp.com --recv-keys "$GPG_KEY"; gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz"; tar -xzf "$DISTRO_NAME.tar.gz"; rm -rf "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc"; chown -R storm:storm "$DISTRO_NAME"' returned a non-zero code: 4
[Pipeline] }
ERROR: script returned exit code 4
As per docker-library/openjdk#322 there is
no more OpenJDK 8 Alpine images (Alpine/musl is not officially supported by the OpenJDK project, so this reflects that -- see "Project Portola" for the Alpine porting efforts which I understand are still in need of help)
So we have to migrate to openjdk:8-jre-slim
and drop architectures to amd64
.
Please see 31z4/zookeeper-docker@01d134a for reference.
This is likely more of a Apache Storm issue but I can't seem to get storm nimbus -c
to function as described in your readme.md.
Example given:
$ docker run -d --restart always --name nimbus 31z4/storm:1.0.3 storm nimbus -c storm.zookeeper.servers="[zookeeper]"
Results:
$ docker logs some-nimbus
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2154)
at clojure.lang.RT.classForName(RT.java:2163)
at clojure.lang.RT.loadClassForName(RT.java:2182)
at clojure.lang.RT.load(RT.java:436)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5671)
at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
at clojure.core$load_lib.doInvoke(core.clj:5710)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$load_libs.doInvoke(core.clj:5753)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:634)
at clojure.core$use.doInvoke(core.clj:5843)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at org.apache.storm.command.config_value$loading__5340__auto____11787.invoke(config_value.clj:16)
at org.apache.storm.command.config_value__init.load(Unknown Source)
at org.apache.storm.command.config_value__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2154)
at clojure.lang.RT.classForName(RT.java:2163)
at clojure.lang.RT.loadClassForName(RT.java:2182)
at clojure.lang.RT.load(RT.java:436)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:379)
at org.apache.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.IllegalArgumentException: Field STORM_ZOOKEEPER_SERVERS must be an Iterable but was a class java.lang.String
at org.apache.storm.validation.ConfigValidationUtils$2.validateField(ConfigValidationUtils.java:125)
at org.apache.storm.validation.ConfigValidationUtils$NestableFieldValidator.validateField(ConfigValidationUtils.java:48)
at org.apache.storm.validation.ConfigValidation$ListEntryTypeValidator.validateField(ConfigValidation.java:304)
at org.apache.storm.validation.ConfigValidation$ListEntryTypeValidator.validateField(ConfigValidation.java:299)
at org.apache.storm.validation.ConfigValidation.validateField(ConfigValidation.java:624)
at org.apache.storm.validation.ConfigValidation.validateFields(ConfigValidation.java:659)
at org.apache.storm.validation.ConfigValidation.validateFields(ConfigValidation.java:638)
at org.apache.storm.config$validate_configs_with_schemas.invoke(config.clj:74)
at org.apache.storm.config$read_storm_config.invoke(config.clj:79)
at org.apache.storm.config$fn__908.invoke(config.clj:100)
at org.apache.storm.config__init.load(Unknown Source)
at org.apache.storm.config__init.<clinit>(Unknown Source)
... 35 more
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2154)
at clojure.lang.RT.classForName(RT.java:2163)
at clojure.lang.RT.loadClassForName(RT.java:2182)
at clojure.lang.RT.load(RT.java:436)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5671)
at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
at clojure.core$load_lib.doInvoke(core.clj:5710)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$load_libs.doInvoke(core.clj:5753)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:634)
at clojure.core$use.doInvoke(core.clj:5843)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at org.apache.storm.command.config_value$loading__5340__auto____11787.invoke(config_value.clj:16)
at org.apache.storm.command.config_value__init.load(Unknown Source)
at org.apache.storm.command.config_value__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2154)
at clojure.lang.RT.classForName(RT.java:2163)
at clojure.lang.RT.loadClassForName(RT.java:2182)
at clojure.lang.RT.load(RT.java:436)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:379)
at org.apache.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.IllegalArgumentException: Field STORM_ZOOKEEPER_SERVERS must be an Iterable but was a class java.lang.String
at org.apache.storm.validation.ConfigValidationUtils$2.validateField(ConfigValidationUtils.java:125)
at org.apache.storm.validation.ConfigValidationUtils$NestableFieldValidator.validateField(ConfigValidationUtils.java:48)
at org.apache.storm.validation.ConfigValidation$ListEntryTypeValidator.validateField(ConfigValidation.java:304)
at org.apache.storm.validation.ConfigValidation$ListEntryTypeValidator.validateField(ConfigValidation.java:299)
at org.apache.storm.validation.ConfigValidation.validateField(ConfigValidation.java:624)
at org.apache.storm.validation.ConfigValidation.validateFields(ConfigValidation.java:659)
at org.apache.storm.validation.ConfigValidation.validateFields(ConfigValidation.java:638)
at org.apache.storm.config$validate_configs_with_schemas.invoke(config.clj:74)
at org.apache.storm.config$read_storm_config.invoke(config.clj:79)
at org.apache.storm.config$fn__908.invoke(config.clj:100)
at org.apache.storm.config__init.load(Unknown Source)
at org.apache.storm.config__init.<clinit>(Unknown Source)
... 35 more
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2154)
at clojure.lang.RT.classForName(RT.java:2163)
at clojure.lang.RT.loadClassForName(RT.java:2182)
at clojure.lang.RT.load(RT.java:436)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5671)
at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
at clojure.core$load_lib.doInvoke(core.clj:5710)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$load_libs.doInvoke(core.clj:5753)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:634)
at clojure.core$use.doInvoke(core.clj:5843)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at org.apache.storm.command.config_value$loading__5340__auto____11787.invoke(config_value.clj:16)
at org.apache.storm.command.config_value__init.load(Unknown Source)
at org.apache.storm.command.config_value__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2154)
at clojure.lang.RT.classForName(RT.java:2163)
at clojure.lang.RT.loadClassForName(RT.java:2182)
at clojure.lang.RT.load(RT.java:436)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:379)
at org.apache.storm.command.config_value.<clinit>(Unknown Source)
Caused by: java.lang.IllegalArgumentException: Field STORM_ZOOKEEPER_SERVERS must be an Iterable but was a class java.lang.String
at org.apache.storm.validation.ConfigValidationUtils$2.validateField(ConfigValidationUtils.java:125)
at org.apache.storm.validation.ConfigValidationUtils$NestableFieldValidator.validateField(ConfigValidationUtils.java:48)
at org.apache.storm.validation.ConfigValidation$ListEntryTypeValidator.validateField(ConfigValidation.java:304)
at org.apache.storm.validation.ConfigValidation$ListEntryTypeValidator.validateField(ConfigValidation.java:299)
at org.apache.storm.validation.ConfigValidation.validateField(ConfigValidation.java:624)
at org.apache.storm.validation.ConfigValidation.validateFields(ConfigValidation.java:659)
at org.apache.storm.validation.ConfigValidation.validateFields(ConfigValidation.java:638)
at org.apache.storm.config$validate_configs_with_schemas.invoke(config.clj:74)
at org.apache.storm.config$read_storm_config.invoke(config.clj:79)
at org.apache.storm.config$fn__908.invoke(config.clj:100)
at org.apache.storm.config__init.load(Unknown Source)
at org.apache.storm.config__init.<clinit>(Unknown Source)
... 35 more
I've tried different formats of entering the config override and it continues to come back with there error Caused by: java.lang.IllegalArgumentException: Field STORM_ZOOKEEPER_SERVERS must be an Iterable but was a class java.lang.String
. Unfortunately Apache Storm documentation around the storm cli tool is very limited.
I'm starting in cluster mode with docker composer, and I have 2 issues
Any help?
Before you file an issue here, please keep in mind that your issue may be not related to the image itself. Please make sure that it is, otherwise report the issue upstream.
Setting up a storm topology
topology starts normally but nothing happens, looking inside the workers' logs it seems as though there a jvm issue.
trying to deploy storm cluster locally via docker compose and submitting topologies to it
Tell us how to reproduce the issue.
Linux version 4.4.0-128-generic (buildd@lcy01-amd64-019) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #154-Ubuntu SMP Fri May 25 14:15:18 UTC 2018
Please include as much relevant information as possible.
Workers Logs:
2018-06-28 21:34:48.036 STDERR Thread-1 [INFO] Unrecognized VM option 'UnlockCommercialFeatures'
2018-06-28 21:34:48.046 STDERR Thread-1 [INFO] Error: Could not create the Java Virtual Machine.
2018-06-28 21:34:48.046 STDERR Thread-1 [INFO] Error: A fatal exception has occurred. Program will exit.
Docker-compose:
version: '2.4'
services:
zookeeper:
image: zookeeper
container_name: zookeeper
restart: always
ports:
- 2181:2181
nimbus:
image: storm
container_name: nimbus
command: storm nimbus -c topology.worker.max.heap.size.mb=65536 -c worker.heap.memory.mb=65536
depends_on:
- zookeeper
links:
- zookeeper
restart: always
ports:
- 6627:6627
volumes:
- ./nimbus/logs:/logs
- ./nimbus/data:/data
supervisor:
image: storm
container_name: supervisor
command: storm supervisor -c worker.heap.memory.mb=65536 -c supervisor.memory.capacity.mb=262144 -c worker.profiler.enabled=true
depends_on:
- nimbus
- zookeeper
links:
- nimbus
- zookeeper
restart: always
volumes:
- ./supervisor/logs:/logs
- ./supervisor/data:/data
ui:
image: storm
container_name: storm-ui
command: storm ui
depends_on:
- nimbus
- zookeeper
links:
- nimbus
- zookeeper
restart: always
ports:
- 8080:8080
This issue follows up on #32. PR #3504 fixes the bug, but, from what I'm seeing, a patch has yet to be released. Therefore, the supervisor process is still failing in the latest image build.
Supervisor ignores unrecognized Java 11 VM options.
Supervisor fails because of deprecated VM options.
2023-02-26 12:11:52.741 STDERR Thread-0 [INFO] Unrecognized VM option 'PrintGCDateStamps'
2023-02-26 12:11:52.741 STDERR Thread-1 [INFO] [0.000s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:artifacts/gc.log instead.
2023-02-26 12:11:52.743 STDERR Thread-0 [INFO] Error: Could not create the Java Virtual Machine.
2023-02-26 12:11:52.743 STDERR Thread-0 [INFO] Error: A fatal exception has occurred. Program will exit.
However, adding the IgnoreUnrecognizedVMOptions
option in /conf/storm.yaml
to override the default worker.childopts
config fixes the issue:
worker.childopts: "-Xmx%HEAP-MEM%m -XX:+IgnoreUnrecognizedVMOptions -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"
Are you guys planning to release a patch soon?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.