Comments (23)
osheroff
commented on June 27, 2024
2
@rtorrero yup. Been looking into maybe porting maxwell over to https://github.com/shyiko/mysql-binlog-connector-java for its GTID support, but it's a large-ish effort.
from maxwell.
akshayi1
commented on June 27, 2024
Got your answer of it not being implemented in the near-future. Thanks! Closing it now.
from maxwell.
osheroff
commented on June 27, 2024
I'll leave this open, in case anyone else comes to want GTID.
from maxwell.
akshayi1
commented on June 27, 2024
Out of curiosity, I looked at open-replicator, and it looks like GTID is implemented there. Can you please clarify then why you said that it'll be quite a bit of work down in open-replicator? Am I looking at the wrong thing in open-replicator, or is it just that the boilerplate is present, but actual implementation is a lot more work? I'm just a little confused. Thanks!
from maxwell.
osheroff
commented on June 27, 2024
not as far as I can tell. There's some support for parsing GTID
events from the logs, but there's no support for setting up GTID
replication positions (which are fundamentally different beasts than
file/position based replication positions)
from maxwell.
rtorrero
commented on June 27, 2024
Like someone already said on #251 this, combined with consul lock would improve a lot the HA of maxwell. Another instance could be on hold, resuming exactly where the first one left as soon as it fails.
from maxwell.
smferguson
commented on June 27, 2024
bump. any word here? (feeling like it's stale) ... wondering if anyone runs maxwell with 5.7 in prod (without gtids)
from maxwell.
yogiGueta
commented on June 27, 2024
Is there any way to get the xid number of the transactions ordered? like if i have two mariadb clients writing to mariadb, sometimes i got the xid in maxwell not in the right order....like:
2017-02-01 17:24:02.557 [main] INFO : {"database":"DB0","table":"T0","type":"update","ts":1485962642,"xid":2452507,"data":{"id":219499,"F0":437933182,"F1":15432081,"F2":453857335,"F3":1963974132},"old":{"F1":1601269758,"F2":1394049406}}
2017-02-01 17:24:02.560 [main] INFO: {"database":"DB0","table":"T0","type":"update","ts":1485962642,"xid":2452507,"data":{"id":244946,"F0":1226648266,"F1":530254919,"F2":1841928968,"F3":528578396},"old":{"F0":1400441802,"F1":719180255,"F2":769416959,"F3":223175808}}
2017-02-01 17:24:02.561 [main] INFO :{"database":"DB0","table":"T0","type":"update","ts":1485962642,"xid":2452507,"data":{"id":183584,"F0":779004500,"F1":1753604934,"F2":1618632991,"F3":1124771345},"old":{"F0":1398288673,"F1":1799403100,"F2":1641979330,"F3":592561791}}
2017-02-01 17:24:02.581 [main] INFO: {"database":"DB0","table":"T0","type":"update","ts":1485962642,"xid":2452507,"commit":true,"data":{"id":197743,"F0":663913596,"F1":1488959783,"F2":1712400104,"F3":10388106},"old":{"F0":2040813092,"F1":337639973,"F2":546838057,"F3":2054566224}}
2017-02-01 17:24:02.872 [main] INFO: {"database":"DB0","table":"T0","type":"delete","ts":1485962642,"xid":2452499,"data":{"id":253688,"F0":1660559508,"F1":773498766,"F2":1777714247,"F3":240264446}}
so there is a way to prevent it without gtid? using xid??
from maxwell.
osheroff
commented on June 27, 2024
I think the xid is dependent on when a transaction starts, not commits; so the ordering is not guaranteed like that. The ordering of updates *is* properly serialized though. You could use the Kafka offset of the message if you need something monotonic.
… On Feb 1, 2017, at 09:59, yogiGueta ***@***.***> wrote:
Is there any way to get the xid number of the transactions ordered? like if i have two mariadb clients writing to mariadb, sometimes i got the xid in maxwell not in the right order....like:
2017-02-01 17:24:02.557 [main] INFO : {"database":"DB0","table":"T0","type":"update","ts":1485962642,"xid":2452507,"data":{"id":219499,"F0":437933182,"F1":15432081,"F2":453857335,"F3":1963974132},"old":{"F1":1601269758,"F2":1394049406}}
2017-02-01 17:24:02.560 [main] INFO: {"database":"DB0","table":"T0","type":"update","ts":1485962642,"xid":2452507,"data":{"id":244946,"F0":1226648266,"F1":530254919,"F2":1841928968,"F3":528578396},"old":{"F0":1400441802,"F1":719180255,"F2":769416959,"F3":223175808}}
2017-02-01 17:24:02.561 [main] INFO :{"database":"DB0","table":"T0","type":"update","ts":1485962642,"xid":2452507,"data":{"id":183584,"F0":779004500,"F1":1753604934,"F2":1618632991,"F3":1124771345},"old":{"F0":1398288673,"F1":1799403100,"F2":1641979330,"F3":592561791}}
2017-02-01 17:24:02.581 [main] INFO: {"database":"DB0","table":"T0","type":"update","ts":1485962642,"xid":2452507,"commit":true,"data":{"id":197743,"F0":663913596,"F1":1488959783,"F2":1712400104,"F3":10388106},"old":{"F0":2040813092,"F1":337639973,"F2":546838057,"F3":2054566224}}
2017-02-01 17:24:02.872 [main] INFO: {"database":"DB0","table":"T0","type":"delete","ts":1485962642,"xid":2452499,"data":{"id":253688,"F0":1660559508,"F1":773498766,"F2":1777714247,"F3":240264446}}
so there is a way to prevent it without gtid? using xid??
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.
from maxwell.
yogiGueta
commented on June 27, 2024
can you elaborate please on how can i do it using offset? do you know mybe
if gtid also depend on transaction start?
thanks!
On Feb 1, 2017 9:59 PM, "Ben Osheroff" <[email protected]> wrote:
I think the xid is dependent on when a transaction starts, not commits; so
the ordering is not guaranteed like that. The ordering of updates *is*
properly serialized though. You could use the Kafka offset of the message
if you need something monotonic.
On Feb 1, 2017, at 09:59, yogiGueta ***@***.***> wrote:
Is there any way to get the xid number of the transactions ordered? like
if i have two mariadb clients writing to mariadb, sometimes i got the xid
in maxwell not in the right order....like:
2017-02-01 17:24:02.557 [main] INFO : {"database":"DB0","table":"T0"
,"type":"update","ts":1485962642,"xid":2452507,"data":{"id":219499,"F0":
437933182,"F1":15432081,"F2":453857335,"F3":1963974132},"
old":{"F1":1601269758,"F2":1394049406}}
2017-02-01 17:24:02.560 [main] INFO: {"database":"DB0","table":"T0"
,"type":"update","ts":1485962642,"xid":2452507,"data":{"id":244946,"F0":
1226648266,"F1":530254919,"F2":1841928968,"F3":528578396},"
old":{"F0":1400441802,"F1":719180255,"F2":769416959,"F3":223175808}}
2017-02-01 17:24:02.561 [main] INFO :{"database":"DB0","table":"
T0","type":"update","ts":1485962642,"xid":2452507,"data":{"id":183584,"F0":
779004500,"F1":1753604934,"F2":1618632991,"F3":1124771345},"
old":{"F0":1398288673,"F1":1799403100,"F2":1641979330,"F3":592561791}}
2017-02-01 17:24:02.581 [main] INFO: {"database":"DB0","table":"T0"
,"type":"update","ts":1485962642,"xid":2452507,"commit":true,"data":{"id":
197743,"F0":663913596,"F1":1488959783,"F2":1712400104,"
F3":10388106},"old":{"F0":2040813092,"F1":337639973,"F2"
:546838057,"F3":2054566224}}
2017-02-01 17:24:02.872 [main] INFO: {"database":"DB0","table":"T0"
,"type":"delete","ts":1485962642,"xid":2452499,"data":{"id":253688,"F0":
1660559508,"F1":773498766,"F2":1777714247,"F3":240264446}}
so there is a way to prevent it without gtid? using xid??
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#69 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AQzInyF2yzchxIfRlE_MVrYJ9cwNnrLOks5rYOQSgaJpZM4FGu5c>
.
from maxwell.
osheroff
commented on June 27, 2024
when you receive the message in your kafka consumer it will have an offset property that you can access.
GTIDs should be monotonic for a particular server, I think. Maybe @HenryCaiHaiying can confirm, he's working on maxwell support for them currently.
from maxwell.
HenryCaiHaiying
commented on June 27, 2024
In the normal situation, GTID should be increasing on a particular server since it's composed of server-uuid:local-tx-id. However there are two modes I haven't really tested it:
- When you replicate binlogs from server A to server B, you can use multi-threaded mode, so it's possible that some threads on B will fetch later binlogs from A first, but I think the binlog apply should still keep the original order;
- If you setup A as master and B as slave, later A crashed, the cluster failed over to B. And later A recovered, A resumes as master. I am not sure whether the new GTIDs issued from A will have a gap or not.
from maxwell.
yogiGueta
commented on June 27, 2024
Thank you @osheroff and @HenryCaiHaiying.
@HenryCaiHaiying how can i use your maxwell version?
from maxwell.
w7089
commented on June 27, 2024
It would be really helpful to have gtid support. Is there a version of maxwell in github that supports it?
from maxwell.
w7089
commented on June 27, 2024
@HenryCaiHaiying Does GTID parsing feature you added support MariaDB?
We saw here the following remark:
Note that MariaDB and MySQL have different GTID implementations, and that these are not compatible with each other.
from maxwell.
yogiGueta
commented on June 27, 2024
I try to run your project with GTID support, but somthing is not right: the "gtid_set" colunm inside table positions is always null and never change...
from maxwell.
HenryCaiHaiying
commented on June 27, 2024
That's a bit weird, did you turn on shyiko_mode and other mysql settings as I mentioned in readme file?
from maxwell.
HenryCaiHaiying
commented on June 27, 2024
For MariaDB, I don't know whether it will work or not. We are using shyiko's binlog connector, if his connector can support mariadb, we might be fine.
from maxwell.
yogiGueta
commented on June 27, 2024
Yes for mysql settings, what do you mean by shyiko-mode?
On Feb 2, 2017 8:55 PM, "Henry Haiying Cai" <[email protected]> wrote:
That's a bit weird, did you turn on shyiko_mode and other mysql settings as
I mentioned in readme file?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#69 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AQzInzpSMVTJiOb19Uo5JQ7EI80Qd0PAks5rYiangaJpZM4FGu5c>
.
from maxwell.
HenryCaiHaiying
commented on June 27, 2024
set system env:
SHYKO_MODE
Take a look at MaxwellConfig.java, you will see how to set it either in
system env or from command line arg.
…On Thu, Feb 2, 2017 at 11:08 AM, yogiGueta ***@***.***> wrote:
Yes for mysql settings, what do you mean by shyiko-mode?
On Feb 2, 2017 8:55 PM, "Henry Haiying Cai" ***@***.***>
wrote:
That's a bit weird, did you turn on shyiko_mode and other mysql settings as
I mentioned in readme file?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#69 (comment)>, or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/
AQzInzpSMVTJiOb19Uo5JQ7EI80Qd0PAks5rYiangaJpZM4FGu5c>
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#69 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AHCX_2faeJ55FmMLt552LLSLLnGR-mfyks5rYim1gaJpZM4FGu5c>
.
from maxwell.
yogiGueta
commented on June 27, 2024
Ok, i did do that! thanks you!
On Feb 2, 2017 9:44 PM, "Henry Haiying Cai" <[email protected]>
wrote:
… set system env:
SHYKO_MODE
Take a look at MaxwellConfig.java, you will see how to set it either in
system env or from command line arg.
On Thu, Feb 2, 2017 at 11:08 AM, yogiGueta ***@***.***>
wrote:
> Yes for mysql settings, what do you mean by shyiko-mode?
> On Feb 2, 2017 8:55 PM, "Henry Haiying Cai" ***@***.***>
> wrote:
>
> That's a bit weird, did you turn on shyiko_mode and other mysql settings
as
> I mentioned in readme file?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#69 (comment)>,
or
> mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/
> AQzInzpSMVTJiOb19Uo5JQ7EI80Qd0PAks5rYiangaJpZM4FGu5c>
>
> .
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#69 (comment)>,
or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHCX_
2faeJ55FmMLt552LLSLLnGR-mfyks5rYim1gaJpZM4FGu5c>
> .
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#69 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AQzIn5MxTmZFTZc6DzP53RTqpdDAjOF8ks5rYjI1gaJpZM4FGu5c>
.
from maxwell.
osheroff
commented on June 27, 2024
use the binlog_connector config setting instead -- SHYKO_MODE is just for tests and will go away
from maxwell.
osheroff
commented on June 27, 2024
Maxwell got gtid mode long ago
from maxwell.
Related Issues (20)
- Maxwell network occupation
- Why SQL_BLACKLIST is needed HOT 1
- Running maxwell on kubernetes HOT 3
- linux/arm64 docker image getting amd64 library problem HOT 4
- Startup error HOT 1
- ALTER TABLE tasks DROP COLUMN IF EXISTS snoozed_until
- Missing documentation for per-table topics HOT 1
- InflightMessageList functionality is broken in maxwell 1.40.3 HOT 2
- Running Maxwell on Docker
- Failure to publish more than 35 changes if kafka is down HOT 3
- Unable to start Maxwell container along with mysql HOT 1
- Schema capturing in maxwell HOT 1
- filtering pattern not work, the docker container panic HOT 8
- when db failover, and set Read_only=true, what exception maxwell will throw? HOT 1
- parse error on algorithm=NOCOPY
- Reduce memory overhead on startup (OOM) HOT 9
- Allow persistent variables in Javascript filters HOT 4
- User sessions
- Failed to deserialize data of EventHeaderV4 HOT 4
- NOT FOUND mysqlBaseListener class HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maxwell.