Comments (34)
wangzw
commented on June 12, 2024
Please check if there is active load query in database first. And then check loading status with rest api or jmx.
from bireme.
jinxiaoxin
commented on June 12, 2024
@wangzw 第一句话什么意思？我没有理解。。我同步5张表没有问题，表一增加 就会出现上述情况
rest api ip:8080/mysql
result:
rest api ip:8080/
result:
why?
from bireme.
wangzw
commented on June 12, 2024
Which version of bireme are you using?
from bireme.
jinxiaoxin
commented on June 12, 2024
@wangzw version: Release v1.0 你有同步过表数量比较多的实例吗？有问题吗？
from bireme.
wangzw
commented on June 12, 2024
Have you checked if binlog data has been produced into kafka?
from bireme.
jinxiaoxin
commented on June 12, 2024
一直实时监控的，确实生产到kafka了
from bireme.
jinxiaoxin
commented on June 12, 2024
@wangzw 发现我只要是 在bireme里面配置的表多了，就会出现此问题，，例如 kafka里面只有5张表的数据，而bireme里面配置50张表，就会出问题。是不是需要合理调节下参数？
from bireme.
wangzw
commented on June 12, 2024
@jinxiaoxin Thanks for your information.
It would be better to enable TRACE level logging for bireme in log2j.xml.
from bireme.
wangzw
commented on June 12, 2024
@Rucfisher What if the topic we want to subscribe does not exist? Need more investigation.
from bireme.
jinxiaoxin
commented on June 12, 2024
我之前都用log4j，对log2j不熟悉，把这里改成trace级别就可以吧？
from bireme.
RebeccaZxy
commented on June 12, 2024
@jinxiaoxin 是的，level改成trace就可以。
from bireme.
jinxiaoxin
commented on June 12, 2024
@RebeccaZxy 改了之后，还是跟以前以前，没有报错信息，假死状态。。
from bireme.
wangzw
commented on June 12, 2024
@jinxiaoxin Please also change the root level of log to trace level and try again.
from bireme.
wangzw
commented on June 12, 2024
@jinxiaoxin It is also helpful to print stack of bireme using command line jstack
from bireme.
jinxiaoxin
commented on June 12, 2024
@wangzw bireme.out还是没有error,倒是bireme.err里面多了一堆信息，但是也没有error
from bireme.
wangzw
commented on June 12, 2024
Any message would help. Please paste to this issue or mail me by email. jstack information would be also useful.
from bireme.
jinxiaoxin
commented on June 12, 2024
@wangzw
jstack info：
jstack -l pid
jstack_message.txt
能看出什么问题吗？
我突然又发现如果kafka中未消费数据过多~
启动bireme
基本会在1分钟左右就假死~
from bireme.
wangzw
commented on June 12, 2024
"Provider mysql" #38 prio=5 os_prio=0 tid=0x0000000001e16000 nid=0x8e9 waiting on condition [0x00007fefd4a3e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003cfd06170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:385)
at cn.hashdata.bireme.provider.KafkaProvider.call(KafkaProvider.java:161)
at cn.hashdata.bireme.provider.KafkaProvider.call(KafkaProvider.java:35)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000003cfc967e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Dispatcher" #37 prio=5 os_prio=0 tid=0x0000000001d0d800 nid=0x8e8 waiting for monitor entry [0x00007fefd4d3e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at cn.hashdata.bireme.RowCache.createBatch(RowCache.java:91)
- waiting to lock <0x00000003cfe7c4e0> (a cn.hashdata.bireme.RowCache)
at cn.hashdata.bireme.RowCache.addRows(RowCache.java:62)
at cn.hashdata.bireme.Dispatcher.insertRowSet(Dispatcher.java:139)
at cn.hashdata.bireme.Dispatcher.checkTansformResults(Dispatcher.java:124)
at cn.hashdata.bireme.Dispatcher.call(Dispatcher.java:73)
at cn.hashdata.bireme.Dispatcher.call(Dispatcher.java:34)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000003cfce25f0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"TaskGenerator" #36 prio=5 os_prio=0 tid=0x0000000001d0b800 nid=0x8e7 waiting on condition [0x00007fefd4e40000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003cfe7c768> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:385)
at cn.hashdata.bireme.RowCache.createBatch(RowCache.java:116)
- locked <0x00000003cfe7c4e0> (a cn.hashdata.bireme.RowCache)
at cn.hashdata.bireme.RowCache.fetchBatch(RowCache.java:130)
at cn.hashdata.bireme.TaskGenerator.generateMergeTask(TaskGenerator.java:96)
at cn.hashdata.bireme.TaskGenerator.call(TaskGenerator.java:63)
at cn.hashdata.bireme.TaskGenerator.call(TaskGenerator.java:32)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000003cfd54b30> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Seems like a dead lock
from bireme.
jinxiaoxin
commented on June 12, 2024
emmmmm,这如何解决了。。。
from bireme.
wangzw
commented on June 12, 2024
Remove this line may work. And we will finally fix this issue in 2.0 release.
from bireme.
jinxiaoxin
commented on June 12, 2024
这段代码移除之后 是一条一条的 copy了吗？没有merge了？
from bireme.
wangzw
commented on June 12, 2024
Nope, still in batch.
from bireme.
jinxiaoxin
commented on June 12, 2024
1、注释掉了createBatch(); 还是同样的情况，kafka数据堆积过多，就会导致1分钟之内就会假死。
2、为什么我的HTTP服务器一直报500错误
from bireme.
wangzw
commented on June 12, 2024
We will fix this issue in next release with 500 error together.
from bireme.
shubifeng
commented on June 12, 2024
from bireme.
jinxiaoxin
commented on June 12, 2024
@shubifeng 什么意思？
from bireme.
shubifeng
commented on June 12, 2024
我也遇见同样的问题，同步单个表时正常。但200多个表时，基本不消费了，且没有任何日志。 不知道是不是机器配置太低的问题。
from bireme.
jinxiaoxin
commented on June 12, 2024
你是maxwell--->kafka--->bireme?
from bireme.
shubifeng
commented on June 12, 2024
是的，你的问题解决没。我这边启动就卡死，没有任何错误信息，机器只有几十MB可用估计跟这个有关。你扣扣多少，加你一起交流下。
from bireme.
shubifeng
commented on June 12, 2024
@jinxiaoxin 兄弟 都搞定了吗
from bireme.
jinxiaoxin
commented on June 12, 2024
我后边自己写的同步程序
from bireme.
shubifeng
commented on June 12, 2024
@jinxiaoxin 是在这个基础上改的吗?
from bireme.
jinxiaoxin
commented on June 12, 2024
不是
from bireme.
shubifeng
commented on June 12, 2024
可以共享给我吗?
from bireme.
Related Issues (20)
- 2.0.0版本启动一段时间就挂了 HOT 2
- 目标数据库支持mysql吗？应该怎么配置。
- 库表字段添加,库表新建,库表字段修改这类DML语言能自动同步吗? HOT 1
- kafka+Maxwell+bireme同步问题
- 开发环境配置 HOT 1
- 数据同步丢失问题
- jsvc error: Invalid option -cwd HOT 1
- Greenplum table and MySQL table size are inconsistent! HOT 1
- bireme可以多实例吗
- is kafka 0.9.0.1 support? HOT 2
- com.google.gson.stream.MalformedJsonException
- Greenplum table and MySQL table size are inconsistent!
- 当缓存数据达到一定量，将这些数据合并为一个任务 (Task)
- bire2.0支持mongo配置吗 HOT 5
- kafka数据中有转义符如何处理
- cn.hashdata.bireme.BiremeException: Transform failed.
- 怎样设置同步ddl，比如 alter table tbName add/drop columnNm... HOT 1
- postgres执行DELETE操作时brieme报错：cn.hashdata.bireme.BiremeException: Transform failed.
- Delete event cause failure
- Kafka重复消费问题
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bireme.