Comments (4)
debakkerb
commented on July 22, 2024
When the JobManager is started, I can see this in the logs:
2021-11-30 08:47:07,063 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.10.3, Rev:2b8b670, Date:20.01.2021 @ 11:45:46 CST)
2021-11-30 08:47:07,063 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink
2021-11-30 08:47:07,063 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found>
2021-11-30 08:47:07,063 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.292-b10
2021-11-30 08:47:07,063 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 1820 MiBytes
2021-11-30 08:47:07,063 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/local/openjdk-8
That maximum heap size seems really small.
from flink-on-k8s-operator.
regadas
commented on July 22, 2024
Hi @debakkerb!
Can you provide an example of your FlinkCluster resource? I suspect you are not setting the flinkVersion property.
from flink-on-k8s-operator.
debakkerb
commented on July 22, 2024
Hey @regadas
Apologies, I forgot to update the ticket. I didn't fully realise that you could set the heap size in the flinkProperties-section in the YAML config:
flinkProperties:
taskmanager.numberOfTaskSlots: "1"
jobmanager.memory.heap.size: "4Gi"
Now I get a different error though, which is causing a lot of confusion on our end:
2021-12-03 10:54:50,752 INFO org.apache.flink.api.java.ExecutionEnvironment - The job has 0 registered types and 0 default Kryo serializers
2021-12-03 10:54:50,788 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Unhandled exception.
org.apache.flink.optimizer.CompilerException: Cannot currently handle nodes with more than 64 outputs.
at org.apache.flink.optimizer.dag.OptimizerNode.addOutgoingConnection(OptimizerNode.java:348)
at org.apache.flink.optimizer.dag.SingleInputNode.setInput(SingleInputNode.java:202)
at org.apache.flink.optimizer.traversals.GraphCreatingVisitor.postVisit(GraphCreatingVisitor.java:272)
at org.apache.flink.optimizer.traversals.GraphCreatingVisitor.postVisit(GraphCreatingVisitor.java:82)
I haven't spent a lot of time investigating, but the guys who wrote our code state that they are not using that many outputs.
from flink-on-k8s-operator.
regadas
commented on July 22, 2024
Hi @debakkerb I think we can close this issue as it seems unrelated now. If if you feel it's still relevant please reopen.
from flink-on-k8s-operator.
Related Issues (20)
- Cluster stuck in Updating state if PodDisruptionBudget is set
- Wrong job status after job update.
- Rework examples
- Allow Flink to ignore savepoint on restore if the states of the old and new jobs are incompatible
- Create a new cluster before deleting the old one on the job update
- FlinkCluster stuck in Updating state when PDB is used.
- Add HorizontalPodAutoScaller properties to FlinkCluster spec HOT 1
- QUESTION: how to get sample app WordCount.jar to run with version 1.15.3 and 2 taskmanager replicas HOT 3
- Caused by GSSException: No valid credentials provided(Mechanism level: Failed to find any Kerberos tgt)
- Validation Error `nodeaffinity` rule for the flinkcluster HOT 2
- HPA not creating new pods on scale event HOT 3
- Pod Affinity Feature Causing Flink Pipeline Redeployment to Fail HOT 2
- poddisruptionbudget is not allowing any disruptions HOT 1
- While using application mode, the jobmanager pod is not restarted when killed
- Flink Operator Loses Job Manager Contact during EKS upgrade HOT 10
- Question: is the latest CRD backwards compatible with the CRD from 0.30 HOT 3
- Job Manager is not brought back up HOT 1
- Application mode Job Manager restart can create multiple FlinkJobs
- Streaming Application mode Jobs can sometimes reach completed stage
- If the job submitter fails, the job keeps running
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-on-k8s-operator.