Comments (17)
knaufk
commented on May 23, 2024
First of all, thanks for reaching out. That's odd. I just ran another test with Flink 1.12. and it is running.
- Which Flink Version?
- Are you sure you pulled the latest commit of this repository? I added DataTimeParsing quite late.
- Does it happen immediately on the first row?
from flink-faker.
O1lpunch3r
commented on May 23, 2024
- Flink version 1.11.2
- downloaded jar form releases page (0.1.0) to /lib directory of sql-client
- It happens immediately
Thanks for your quick response 👍
from flink-faker.
O1lpunch3r
commented on May 23, 2024
Here is the stack trace:
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185)
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179)
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503)
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:386)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.time.format.DateTimeParseException: Text 'Mon Dec 14 17:29:11 CET 2020' could not be parsed at index 2
at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949)
at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1777)
at com.github.knaufk.flink.faker.FakerUtils.stringValueToType(FakerUtils.java:47)
at com.github.knaufk.flink.faker.FlinkFakerSourceFunction.generateNextRow(FlinkFakerSourceFunction.java:93)
at com.github.knaufk.flink.faker.FlinkFakerSourceFunction.run(FlinkFakerSourceFunction.java:49)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213)
On my CentOS standalone cluster, the error appears.
But when i execute it in a docker environment like this from Ververica, it is running. That's odd.
from flink-faker.
knaufk
commented on May 23, 2024
It somehow seems to be related to your environment. I only found this SO question [1] that seems related, although the root cause differs. Maybe this helps in debugging. If there is anything I can do to help you debug this, let me know.
from flink-faker.
O1lpunch3r
commented on May 23, 2024
Thanks for the research. This is a really weird problem. Unfortunately, neither the installation of AdoptOpenJDK hotspot (version 8 and 11) nor the downgrade of the rpm package tzdata/tzdata-java did anything. Maybe you can reproduce it in a Centos environment.
from flink-faker.
appleyuchi
commented on May 23, 2024
from flink-faker.
appleyuchi
commented on May 23, 2024
I reproduced it in Flink1.12 with the following code:
CREATE TEMPORARY TABLE timestamp_example (
`timestamp1` TIMESTAMP(3),
`timestamp2` TIMESTAMP(3)
)
WITH (
'connector' = 'faker',
'fields.timestamp1.expression' = '#{date.past ''15'',''SECONDS''}',
'fields.timestamp2.expression' = '#{date.past ''15'',''5'',''SECONDS''}'
);
SELECT * FROM timestamp_example;
from flink-faker.
appleyuchi
commented on May 23, 2024
maybe due to timezone?
from flink-faker.
O1lpunch3r
commented on May 23, 2024
Which operation system / Java JRE do you use for the sql-client and flink cluster env @appleyuchi ?
from flink-faker.
appleyuchi
commented on May 23, 2024
Which operation system / Java JRE do you use for the sql-client and flink cluster env @appleyuchi ?
| Environment | version |
|---|---|
| System | Ubuntu20.04 |
| Flink(HA) | 1.12 |
| JDK | 1.8.0_131 |
| Zookeeper | 3.6.0 |
openjdk is uninstalled(I'm sure about this)
My $FLINK_HOME/lib is as follows:
commons-cli-1.4.jar
flink-connector-hbase-2.2_2.12-1.12.0_jar
flink-connector-hive_2.12-1.12.0.jar
flink-connector-jdbc_2.12-1.12.0.jar
flink-connector-kafka_2.12-1.12.0.jar
flink-connector-redis_2.11-1.1.5.jar
flink-csv-1.12.0.jar
flink-dist_2.12-1.12.0.jar
flink-faker-0.1.0.jar
flink-json-1.12.0.jar
flink-shaded-hadoop-3-uber-3.1.1.7.0.3.0-79-7.0.jar
flink-shaded-zookeeper-3.4.14.jar
flink-sql-connector-hbase-2.2_2.12-1.12.0.jar
flink-sql-connector-hive-3.1.2_2.12-1.12.0.jar
flink-table_2.12-1.12.0.jar
flink-table-blink_2.12-1.12.0.jar
flink-table-planner_2.12-1.12.0.jar
flink-table-planner-blink_2.12-1.12.0.jar
hadoop-yarn-api-3.1.2.jar
hive-common-3.1.2.jar
hive-exec-3.1.2.jar
javax.ws.rs-api-2.0.jar
kafka-clients-2.5.0.jar
libjars
log4j-1.2-api-2.12.1.jar
log4j-api-2.12.1.jar
log4j-core-2.12.1.jar
log4j-slf4j-impl-2.12.1.jar
mysql-connector-java-8.0.22.jar
I have added the following configuration into $FLINK_HOME/conf/flink-conf.yaml
env.java.opts.jobmanager: -Duser.timezone=Etc/GMT
env.java.opts.taskmanager: -Duser.timezone=Etc/GMT
um....
connector faker is down from https://github.com/knaufk/flink-faker
and mvn clean package and then put the jar under $FLINK_HOME/lib
from flink-faker.
appleyuchi
commented on May 23, 2024
If you have time ,you can control my computer via TeamViewer to reproduce it.
from flink-faker.
knaufk
commented on May 23, 2024
Hi @appleyuchi, @O1lpunch3r,
thanks for the continued interested in this issue. I was still not able to reproduce it (neither with Flink 1.12/11 or OpenJDK 8/11). Anyway, I have a guess that https://github.com/knaufk/flink-faker/tree/datetimeparsing might fix it. Could you check out this branch, mvn clean package and re-try your queries?
Thanks,
Konstantin
from flink-faker.
appleyuchi
commented on May 23, 2024
I'll try it right now.
from flink-faker.
appleyuchi
commented on May 23, 2024
Now everything is ok.
This is tested in China.
https://github.com/knaufk/flink-faker/tree/datetimeparsing
However,before you close this issue,
Could you tell us that you'll merge it in to the https://github.com/knaufk/flink-faker or NOT?
Thanks for your help
from flink-faker.
knaufk
commented on May 23, 2024
Great! Thank you for testing.
If this also fixes it for @O1lpunch3r, I would merge to master and include it in the next release.
from flink-faker.
O1lpunch3r
commented on May 23, 2024
Great! It is working now. I have tested the example code of @appleyuchi and mine. Both are working with yout fix. Thank you @knaufk for fixing this. This connector is so great for testing.
from flink-faker.
knaufk
commented on May 23, 2024
Thank you both for your help. I quickly released 0.1.1 with this fix included.
from flink-faker.
Related Issues (17)
- Expose Source/Sink Metrics
- Add Sequence Columns
- Faker has a small probability to generate NULL values for columns with a zero null rate HOT 2
- Demo web application page link broken
- 直接下载jar 放到flink1.14 lib下 报错 java.lang.ClassNotFoundException: com.github.knaufk.flink.faker.FlinkFakerSourceFunction HOT 1
- Migrate to New Source Interface HOT 3
- Move from TravisCI to GHA HOT 1
- Migrate to Datafaker? HOT 5
- Support Null Rate per Field HOT 1
- RowTime field should not be null, please convert it to a non-null long value.
- Cannot use #{address.city} in expression HOT 5
- #{lorem.sentence} and #{lorem.paragraph} are always the same for the same row HOT 4
- Support More Complex Data Types HOT 1
- Interpret Table as Changelog
- why ~ in the end?Thanks HOT 4
- flink-faker 0.5.1 doesn't work with flink 1.14.4 due to api incompatability HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-faker.