Git Product home page Git Product logo

mysql-ripple's Introduction

Ripple

Ripple is a server that can serve as a middleman in MySQL replication.

The Ripple server connects to a MySQL master the same way a replica would, but rather than creating a copy of the data in MySQL, Ripple just downloads the binlogs and saves them locally. MySQL servers can then be directed to replicate from Ripple instead of the master. Ripple will serve downloaded binlogs to them the same way the master would. This can significantly reduce load on the MySQL master and improve durability of the binlogs.

Ripple supports replication to/from MariaDB and MySQL using GTIDs (of MariaDB and MySQL flavor respectively). Replication using filename and position is not supported. Ripple has been tested with MariaDB 10.0 and MySQL 5.6 and 5.7, but it likely will work with later versions as well.

Build instructions

Following build procedure works on Ubuntu 18.10 cloud image.

Install bazel

apt-get install pkg-config zip g++ zlib1g-dev unzip python libssl-dev default-jdk-headless libmariadbclient-dev
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -
apt-get update && sudo apt-get install bazel

Clone source

git clone https://github.com/google/mysql-ripple.git

Build

cd mysql-ripple
bazel build :all
bazel test :all

If all went OK, your binary is at

./bazel-bin/rippled

mysql-ripple's People

Contributors

codedumper1 avatar dveeden avatar michaelchirico avatar nerdatmath avatar pivanof avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mysql-ripple's Issues

mysql_server_connection.cc:112:48: error: too many arguments to function 'my_bool my_net_init(NET*, Vio*, unsigned int)'

when buiding the package, I meet the errors.
Env:
14.04.1-Ubuntu
OPENSSL 1.1.1
libmariadbclient-dev

Error Message:
src/main/tools/process-tools.cc:118: sigaction(32, &sa, nullptr) failed
src/main/tools/process-tools.cc:118: sigaction(33, &sa, nullptr) failed
mysql_server_connection.cc: In static member function 'static mysql_ripple::mysql::ServerConnection* mysql_ripple::mysql::ServerConnection::Accept(Vio*)':
mysql_server_connection.cc:112:48: warning: converting to non-pointer type 'unsigned int' from NULL [-Wconversion-null]
if (my_net_init(&mysql->net, vio, NULL, flags)) {
^
mysql_server_connection.cc:112:48: error: too many arguments to function 'my_bool my_net_init(NET*, Vio*, unsigned int)'
In file included from external/external_libs/mysql/mysql.h:73:0,
from mysql_server_connection.cc:24:
external/external_libs/mysql/mysql_com.h:535:9: note: declared here
my_bool my_net_init(NET net, Vio vio, unsigned int my_flags);
^
src/main/tools/linux-sandbox-pid1.cc:437: waitpid returned 2
src/main/tools/linux-sandbox-pid1.cc:457: child exited with code 1
src/main/tools/linux-sandbox.cc:204: child exited normally with exitcode 1

How can I fix it?
Thank you.

CentOS 7: ERROR: missing input file '@external_libs//:mysql/byte_order_generic_x86_64.h'

On CentOS 7.6 64bit with MariaDB 10.3.12, resolved older git issue #12 only to run into another issue

bazel version  
Build label: 0.23.1- (@non-git)
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Mar 4 13:43:11 2019 (1551706991)
Build timestamp: 1551706991
Build timestamp as int: 1551706991

Using GCC 8.2.1

gcc --version 
gcc (GCC) 8.2.1 20180905 (Red Hat 8.2.1-3)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Git updated from 1.8.3 via IUS Community repo https://dl.iuscommunity.org/pub/ius/stable/CentOS/7/x86_64/repoview/git2u.html

git --version
git version 2.16.5

MariaDB 10.3.12 via official mariadb.org YUM repo for CentOS 7

yum list MariaDB* -q | egrep 'MariaDB|Available|Installed'
Installed Packages
MariaDB-client.x86_64                     10.3.13-1.el7.centos    @mariadb      
MariaDB-common.x86_64                     10.3.13-1.el7.centos    @mariadb      
MariaDB-compat.x86_64                     10.3.13-1.el7.centos    @mariadb      
MariaDB-devel.x86_64                      10.3.13-1.el7.centos    @mariadb      
MariaDB-server.x86_64                     10.3.13-1.el7.centos    @mariadb      
MariaDB-shared.x86_64                     10.3.13-1.el7.centos    @mariadb      
Available Packages
MariaDB-backup.x86_64                     10.3.13-1.el7.centos    mariadb       
MariaDB-cassandra-engine.x86_64           10.3.13-1.el7.centos    mariadb       
MariaDB-connect-engine.x86_64             10.3.13-1.el7.centos    mariadb       
MariaDB-cracklib-password-check.x86_64    10.3.13-1.el7.centos    mariadb       
MariaDB-gssapi-server.x86_64              10.3.13-1.el7.centos    mariadb       
MariaDB-oqgraph-engine.x86_64             10.3.13-1.el7.centos    mariadb       
MariaDB-rocksdb-engine.x86_64             10.3.13-1.el7.centos    mariadb       
MariaDB-test.x86_64                       10.3.13-1.el7.centos    mariadb       
MariaDB-tokudb-engine.x86_64              10.3.13-1.el7.centos    mariadb 
mysqladmin ver
mysqladmin  Ver 9.1 Distrib 10.3.13-MariaDB, for Linux on x86_64
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Server version          10.3.12-MariaDB
Protocol version        10
Connection              Localhost via UNIX socket
UNIX socket             /var/lib/mysql/mysql.sock
Uptime:                 25 days 15 hours 28 min 9 sec

Threads: 8  Questions: 5238456  Slow queries: 0  Opens: 35  Flush tables: 1  Open tables: 27  Queries per second avg: 2.364
yum -y install bazel
cd /svr-setup
git clone https://github.com/google/mysql-ripple.git
cd mysql-ripple
source /opt/rh/devtoolset-8/enable
export CC=gcc
bazel build :all
bazel test :all
bazel build :all
Starting local Bazel server and connecting to it...
DEBUG: Rule 'com_google_googletest' modified arguments {"shallow_since": "1537409963 -0400"}
DEBUG: Rule 'com_google_protobuf' modified arguments {"shallow_since": "1537216885 -0700"}
DEBUG: Rule 'bazel_skylib' modified arguments {"shallow_since": "1535465090 -0400"}
DEBUG: Rule 'com_google_absl' modified arguments {"shallow_since": "1538751188 -0400"}
DEBUG: Rule 'com_google_glog' modified arguments {"shallow_since": "1531725486 +0200"}
DEBUG: Rule 'com_github_gflags_gflags' modified arguments {"shallow_since": "1530829475 +0100"}
INFO: Analysed 50 targets (39 packages loaded, 1394 targets configured).
INFO: Found 50 targets...
INFO: From Compiling log_event.cc:
log_event.cc: In member function 'bool mysql_ripple::RawLogEventData::ParseFromBuffer(const uint8_t*, int)':
log_event.cc:62:30: warning: comparison of integer expressions of different signedness: 'uint32_t' {aka 'unsigned int'} and 'int' [-Wsign-compare]
   return header.event_length <= buffer_length;
          ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
log_event.cc: In member function 'virtual bool mysql_ripple::FormatDescriptorEvent::ParseFromBuffer(const uint8_t*, int)':
log_event.cc:206:66: warning: comparison of integer expressions of different signedness: 'uint' {aka 'unsigned int'} and 'int' [-Wsign-compare]
   for (uint offset = kServerVersionStringLen + 2 + 4 + 1; offset < len - 1;
                                                           ~~~~~~~^~~~~~~~~
log_event.cc: In member function 'virtual bool mysql_ripple::QueryEvent::SerializeToBuffer(uint8_t*, int) const':
log_event.cc:426:14: warning: comparison of integer expressions of different signedness: 'int' and 'std::basic_string<char>::size_type' {aka 'long unsigned int'} [-Wsign-compare]
   if (remain < query.size()) {
       ~~~~~~~^~~~~~~~~~~~~~
INFO: From Compiling gtid.cc:
gtid.cc: In member function 'void mysql_ripple::GTIDList::CheckMerge(mysql_ripple::GTIDList::GTIDStream*, int)':
gtid.cc:635:13: warning: comparison of integer expressions of different signedness: 'int' and 'std::vector<mysql_ripple::GTIDSet::Interval>::size_type' {aka 'long unsigned int'} [-Wsign-compare]
   if (i + 1 >= stream->intervals.size())
       ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
gtid.cc: In member function 'bool mysql_ripple::GTIDList::ValidSuccessor(const mysql_ripple::GTID&) const':
gtid.cc:490:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
ERROR: missing input file '@external_libs//:mysql/byte_order_generic_x86_64.h'
ERROR: /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/external_libs/BUILD.bazel:15:1: @external_libs//:mysqlclient: missing input file '@external_libs//:mysql/byte_order_generic_x86_64.h'
ERROR: /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/external_libs/BUILD.bazel:15:1 1 input file(s) do not exist
INFO: Elapsed time: 49.073s, Critical Path: 4.61s
INFO: 64 processes: 64 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

location for byte_order_generic_x86_64.h

locate byte_order_generic_x86_64.h
/usr/include/mysql/server/byte_order_generic_x86_64.h

missing input file

when i execute bazel build :all,i meet errors as follows

Starting local Bazel server and connecting to it...
INFO: Writing tracer profile to '/root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/command.profile.gz'
INFO: Analyzed 50 targets (43 packages loaded, 1268 targets configured).
INFO: Found 50 targets...
INFO: Deleting stale sandbox base /root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/sandbox
ERROR: missing input file '@external_libs//:mysql/byte_order_generic_x86_64.h'
ERROR: missing input file '@external_libs//:mysql/my_attribute.h'
ERROR: missing input file '@external_libs//:mysql/my_net.h'
ERROR: missing input file '@external_libs//:mysql/my_pthread.h'
ERROR: missing input file '@external_libs//:mysql/my_valgrind.h'
ERROR: missing input file '@external_libs//:mysql/private/violite.h'
ERROR: /root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/external/external_libs/BUILD.bazel:17:1: @external_libs//:mysqlclient: missing input file '@external_libs//:mysql/byte_order_generic_x86_64.h'
ERROR: /root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/external/external_libs/BUILD.bazel:17:1: @external_libs//:mysqlclient: missing input file '@external_libs//:mysql/my_attribute.h'
ERROR: /root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/external/external_libs/BUILD.bazel:17:1: @external_libs//:mysqlclient: missing input file '@external_libs//:mysql/my_net.h'
ERROR: /root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/external/external_libs/BUILD.bazel:17:1: @external_libs//:mysqlclient: missing input file '@external_libs//:mysql/my_pthread.h'
ERROR: /root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/external/external_libs/BUILD.bazel:17:1: @external_libs//:mysqlclient: missing input file '@external_libs//:mysql/my_valgrind.h'
ERROR: /root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/external/external_libs/BUILD.bazel:17:1: @external_libs//:mysqlclient: missing input file '@external_libs//:mysql/private/violite.h'
ERROR: /root/.cache/bazel/_bazel_root/e1a522577afd7e5ef7716dccda52fc23/external/external_libs/BUILD.bazel:17:1 6 input file(s) do not exist
INFO: Elapsed time: 4.358s, Critical Path: 0.04s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

i can't find these file in my mysql-ripple folder,how can i get these file and add?

Has anyone built this without MariaDB?

I'm attempting to build this against Percona (instead of MariaDB), and a bunch of the dependencies (like my_net.h) are MariaDB-specific and not present in Percona or upstream MySQL. Is there any chance of building this without using MariaDB? It seems like a super-cool tool (especially since Booking.com never actually followed through with open-sourcing their binlog proxy), but switching from libmysqlclient to libmariadbclient is definitely a no-go for me...

centos 7 fatal error: my_config_x86_64.h: No such file or directory

uname -a

Linux qabb-qa-mysql2 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux\

os version centos 7.6

==============================
when i execute bazel build :all ,I meet error logs as follows

ERROR: /root/mysql-ripple/BUILD.bazel:352:1: C++ compilation of rule '//:my_crypt' failed (Exit 1) process-wrapper failed: error executing command
  (cd /root/.cache/bazel/_bazel_root/f386c459487244300d44883709cb4cd2/sandbox/processwrapper-sandbox/179/execroot/__main__ && \
  exec env - \
    PATH=/opt/isys/bin/db/zandb_agent/scripts:/opt/tsar/bin:/opt/python/bin:/opt/java/bin:/opt/git/bin:/opt/mydumper/bin:/opt/percona-toolkit/bin:/opt/mysql/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/isys/bin:/opt/isys/sbin:/root/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /root/.cache/bazel/_bazel_root/install/2049aedb20b27bd42311c3e0dec4b818/_embedded_binaries/process-wrapper '--timeout=0' '--kill_delay=15' /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/_objs/my_crypt/my_crypt_key_management.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/my_crypt/my_crypt_key_management.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/external_libs -iquote bazel-out/k8-fastbuild/bin/external/external_libs -isystem external/external_libs/mysql -isystem bazel-out/k8-fastbuild/bin/external/external_libs/mysql -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c my_crypt_key_management.cc -o bazel-out/k8-fastbuild/bin/_objs/my_crypt/my_crypt_key_management.pic.o)
In file included from external/external_libs/mysql/my_global.h:80:0,
                 from my_crypt_key_management.h:21,
                 from my_crypt_key_management.cc:16:
external/external_libs/mysql/my_config.h:14:30: fatal error: my_config_x86_64.h: No such file or directory
 #include "my_config_x86_64.h"
                              ^
compilation terminated.
INFO: Elapsed time: 1.495s, Critical Path: 1.39s
INFO: 3 processes: 3 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

but the file my_config_x86_64.h exists in /usr/include/mysql/

ls /usr/include/mysql/my_config*
/usr/include/mysql/my_config.h /usr/include/mysql/my_config_x86_64.h

how can i solve the problem ?

Unable to set up mysql semi sync with ripple

Hi, I am trying to set up semi sync between master and ripple, but semi sync always switched off automatically when I am running sysbench, the errors on both sides are like below:

Master error.log:

2020-02-25T09:40:06.868855Z 0 [ERROR] /usr/sbin/mysqld: Got packets out of order
2020-02-25T09:40:06.868902Z 0 [ERROR] /usr/sbin/mysqld: Got packets out of order
2020-02-25T09:40:06.868932Z 0 [ERROR] /usr/sbin/mysqld: Couldn't uncompress communication packet
2020-02-25T09:40:06.868941Z 0 [ERROR] /usr/sbin/mysqld: Got packets out of order
2020-02-25T09:40:06.868948Z 0 [ERROR] /usr/sbin/mysqld: Got packets out of order
2020-02-25T09:40:06.870024Z 0 [ERROR] /usr/sbin/mysqld: Got timeout reading communication packets
2020-02-25T09:40:06.875397Z 0 [ERROR] /usr/sbin/mysqld: Got an error reading communication packets
2020-02-25T09:40:06.882687Z 29 [Note] While initializing dump thread for slave with server_id <112211>, found a zombie dump thread with the same server_id. Master is killing the zombie dump thread(3).
2020-02-25T09:40:06.882770Z 3 [Note] Stop semi-sync binlog_dump to slave (server_id: 112211)
2020-02-25T09:40:06.882815Z 29 [Note] Start binlog_dump to master_thread_id(29) slave_server(112211), pos(, 4)
2020-02-25T09:40:06.882837Z 29 [Note] Start semi-sync binlog_dump to slave (server_id: 112211), pos(, 4)
2020-02-25T09:40:06.989646Z 30 [Note] While initializing dump thread for slave with server_id <112211>, found a zombie dump thread with the same server_id. Master is killing the zombie dump thread(29).
2020-02-25T09:40:06.989714Z 29 [Note] Stop semi-sync binlog_dump to slave (server_id: 112211)
2020-02-25T09:40:06.989770Z 30 [Note] Start binlog_dump to master_thread_id(30) slave_server(112211), pos(, 4)
2020-02-25T09:40:06.989793Z 30 [Note] Start semi-sync binlog_dump to slave (server_id: 112211), pos(, 4)
2020-02-25T09:41:46.868547Z 23 [Warning] Timeout waiting for reply of binlog (file: mysql-bin.000001, pos: 596), semi-sync up to file , position 4.
2020-02-25T09:41:46.868601Z 23 [Note] Semi-sync replication switched OFF.

Rpl_semi_sync_master_status also turned OFF during the run:

mysql> show status like '%rpl%';
+--------------------------------------------+-------+
| Variable_name | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients | 1 |
| Rpl_semi_sync_master_net_avg_wait_time | 0 |
| Rpl_semi_sync_master_net_wait_time | 0 |
| Rpl_semi_sync_master_net_waits | 3 |
| Rpl_semi_sync_master_no_times | 1 |
| Rpl_semi_sync_master_no_tx | 2 |
| Rpl_semi_sync_master_status | OFF |
| Rpl_semi_sync_master_timefunc_failures | 0 |
| Rpl_semi_sync_master_tx_avg_wait_time | 0 |
| Rpl_semi_sync_master_tx_wait_time | 0 |
| Rpl_semi_sync_master_tx_waits | 0 |
| Rpl_semi_sync_master_wait_pos_backtraverse | 0 |
| Rpl_semi_sync_master_wait_sessions | 0 |
| Rpl_semi_sync_master_yes_tx | 0 |
+--------------------------------------------+-------+

Ripple log:

I0225 09:40:06.894847 24855 mysql_protocol.cc:550] Send event, type: Xid, timestamp: 1582623606, length: 31, nextpos: 14648
I0225 09:40:06.894914 24855 mysql_protocol.cc:550] Send event, type: Gtid, timestamp: 1582623606, length: 65, nextpos: 14729
I0225 09:40:06.894984 24855 mysql_protocol.cc:550] Send event, type: Query, timestamp: 1582623606, length: 74, nextpos: 14819
I0225 09:40:06.895058 24855 mysql_protocol.cc:550] Send event, type: Unknown event: 19, timestamp: 1582623606, length: 90, nextpos: 14925
I0225 09:40:06.895129 24855 mysql_protocol.cc:550] Send event, type: Unknown event: 30, timestamp: 1582623606, length: 290, nextpos: 15231
I0225 09:40:06.895203 24855 mysql_protocol.cc:550] Send event, type: Xid, timestamp: 1582623606, length: 31, nextpos: 15278

E0225 09:40:06.986371 24811 mysql_master_session.cc:298] Failed to read packet: Got error reading packet from server: Lost connection to MySQL server during query

I0225 09:40:06.986407 24811 binlog.cc:675] Connection closed last position binlog file: binlog.000000:15278, gtid: 0-0-25
I0225 09:40:06.986430 24811 mysql_master_session.cc:273] Disconnecting from master
I0225 09:40:06.987946 24811 mysql_client_connection.cc:148] connected to host: xxx, port: 3306
I0225 09:40:06.990568 24811 mysql_master_session.cc:137] Connected to host: xxx, port: 3306, server_id: 1, server_name:
I0225 09:40:06.991778 24811 mysql_master_session.cc:202] master has semi sync enabled
I0225 09:40:06.991804 24811 mysql_master_session.cc:207] start replicating from '02b13200-56ca-11ea-945b-000d3aa16eab:0-0-25'
I0225 09:40:06.992206 24811 mysql_master_session.cc:230] Master session entering main loop
I0225 09:40:06.992585 24811 binlog.cc:626] Update binlog position to end_pos: binlog.000000:15341, gtid: 0-0-25
I0225 09:40:06.992621 24811 binlog.cc:616] Skip writing event [ Previous_gtids len = 27 ]
I0225 09:40:06.992635 24811 binlog.cc:626] Update binlog position to end_pos: binlog.000000:15341, gtid: 0-0-25

Background:
Mysql version: 5.7
instances on Azure VM Ubuntu, master and ripple are on different 2 VMs,

Commands I am running:
./bazel-bin/rippled -ripple_datadir=/mnt/mysql-log -ripple_master_address=xxx -ripple_master_port=3306 -ripple_master_user=repl_user -ripple_master_password=root123 -ripple_semi_sync_slave_enabled=true

I also verified that:

  1. Async replica works fine between Master & Ripple and slave (ripple & slave on same machine)
  2. Semi sync also works fine with Master & slave (without using ripple)
  3. Increase max_allowed_packet or rpl-semi-sync-master-timeout don't help

Could someone help provide guidance on what I I did wrong or missing ?

Thanks

Integrating with OSS-Fuzz

Greetings mysql-ripple developers and contributors,

Weโ€™re reaching out because your project is an important part of the open source ecosystem, and weโ€™d like to invite you to integrate with our fuzzing service, OSS-Fuzz. OSS-Fuzz is a free fuzzing infrastructure you can use to identify security vulnerabilities and stability bugs in your project. OSS-Fuzz will:

  • Continuously run at scale all the fuzzers you write.
  • Alert you when it finds issues.
  • Automatically close issues after theyโ€™ve been fixed by a commit.

Many widely used open source projects like OpenSSL, FFmpeg, LibreOffice, and ImageMagick are fuzzing via OSS-Fuzz, which helps them find and remediate critical issues.

Even though typical integrations can be done in < 100 LoC, we have a reward program in place which aims to recognize folks who are not just contributing to open source, but are also working hard to make it more secure.

We want to stress that anyone who meets the eligibility criteria and integrates a project with OSS-Fuzz is eligible for a reward.

To help you getting started, we can attach our internal fuzzer for your project that you are welcome to use directly, or to use it as a starting point.

If you're not interested in integrating with OSS-Fuzz, it would be helpful for us to understand whyโ€”lack of interest, lack of time, or something elseโ€”so we can better support projects like yours in the future.

If weโ€™ve missed your question in our FAQ, feel free to reply or reach out to us at [email protected].

Thanks!

Tommy
OSS-Fuzz Team

Error during the build (bazel build :all)

Hi, During build (step: /root/bin/bazel build :all), I am encountering following error. Can you please take a look into it:

 /root/bin/bazel build :all
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Call stack for the definition of repository 'com_google_protobuf' which is a git_repository (rule definition at /root/.cache/bazel/_bazel_root/ed5f0021fd3de9b63d37b11c2a4ed8c7/external/bazel_tools/tools/build_defs/repo/git.bzl:181:18):
 - /home/xxxxx/mysql-ripple/WORKSPACE:22:1
ERROR: An error occurred during the fetch of repository 'com_google_protobuf':
   Traceback (most recent call last):
	File "/root/.cache/bazel/_bazel_root/ed5f0021fd3de9b63d37b11c2a4ed8c7/external/bazel_tools/tools/build_defs/repo/git.bzl", line 163
		_clone_or_update(ctx)
	File "/root/.cache/bazel/_bazel_root/ed5f0021fd3de9b63d37b11c2a4ed8c7/external/bazel_tools/tools/build_defs/repo/git.bzl", line 36, in _clone_or_update
		git_repo(ctx, directory)
	File "/root/.cache/bazel/_bazel_root/ed5f0021fd3de9b63d37b11c2a4ed8c7/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 91, in git_repo
		_update(ctx, git_repo)
	File "/root/.cache/bazel/_bazel_root/ed5f0021fd3de9b63d37b11c2a4ed8c7/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 101, in _update
		init(ctx, git_repo)
	File "/root/.cache/bazel/_bazel_root/ed5f0021fd3de9b63d37b11c2a4ed8c7/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 115, in init
		_error(ctx.name, cl, st.stderr)
	File "/root/.cache/bazel/_bazel_root/ed5f0021fd3de9b63d37b11c2a4ed8c7/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 169, in _error
		fail(("error running '%s' while worki...)))
	File "/root/.cache/bazel/_bazel_root/ed5f0021fd3de9b63d37b11c2a4ed8c7/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 169, in fail
		" ".join(command)
sequence element must be a string (got 'path'). See https://github.com/bazelbuild/bazel/issues/7802 for information about --incompatible_string_join_requires_strings.
Internal error thrown during build. Printing stack trace: java.util.MissingFormatArgumentException: Format specifier '%s'
	at java.base/java.util.Formatter.format(Unknown Source)
	at java.base/java.util.Formatter.format(Unknown Source)
	at java.base/java.lang.String.format(Unknown Source)
	at com.google.devtools.build.lib.skyframe.SkyframeDependencyResolver.getTargets(SkyframeDependencyResolver.java:145)
	at com.google.devtools.build.lib.analysis.DependencyResolver.dependentNodeMap(DependencyResolver.java:274)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.computeDependencies(ConfiguredTargetFunction.java:562)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:317)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.bubbleErrorUp(AbstractExceptionalParallelEvaluator.java:495)
	at com.google.devtools.build.skyframe.ParallelEvaluator.bubbleErrorUpExceptionally(ParallelEvaluator.java:77)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.waitForCompletionAndConstructResult(AbstractExceptionalParallelEvaluator.java:282)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.doMutatingEvaluation(AbstractExceptionalParallelEvaluator.java:265)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.evalExceptionally(AbstractExceptionalParallelEvaluator.java:201)
	at com.google.devtools.build.skyframe.ParallelEvaluator.eval(ParallelEvaluator.java:70)
	at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.evaluate(InMemoryMemoizingEvaluator.java:205)
	at com.google.devtools.build.skyframe.SequentialBuildDriver.evaluate(SequentialBuildDriver.java:38)
	at com.google.devtools.build.lib.skyframe.SkyframeExecutor.configureTargets(SkyframeExecutor.java:2381)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.configureTargets(SkyframeBuildView.java:373)
	at com.google.devtools.build.lib.analysis.BuildView.update(BuildView.java:399)
	at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.runAnalysisPhase(AnalysisPhaseRunner.java:198)
	at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:110)
	at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:141)
	at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:268)
	at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:98)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:524)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:192)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:573)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$2(GrpcServerImpl.java:624)
	at java.base/java.lang.Thread.run(Unknown Source)

INFO: Elapsed time: 6.850s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (7 packages loaded, 21 targets configured)
Internal error thrown during build. Printing stack trace: java.util.MissingFormatArgumentException: Format specifier '%s'
	at java.base/java.util.Formatter.format(Unknown Source)
	at java.base/java.util.Formatter.format(Unknown Source)
	at java.base/java.lang.String.format(Unknown Source)
	at com.google.devtools.build.lib.skyframe.SkyframeDependencyResolver.getTargets(SkyframeDependencyResolver.java:145)
	at com.google.devtools.build.lib.analysis.DependencyResolver.dependentNodeMap(DependencyResolver.java:274)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.computeDependencies(ConfiguredTargetFunction.java:562)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:317)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.bubbleErrorUp(AbstractExceptionalParallelEvaluator.java:495)
	at com.google.devtools.build.skyframe.ParallelEvaluator.bubbleErrorUpExceptionally(ParallelEvaluator.java:77)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.waitForCompletionAndConstructResult(AbstractExceptionalParallelEvaluator.java:282)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.doMutatingEvaluation(AbstractExceptionalParallelEvaluator.java:265)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.evalExceptionally(AbstractExceptionalParallelEvaluator.java:201)
	at com.google.devtools.build.skyframe.ParallelEvaluator.eval(ParallelEvaluator.java:70)
	at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.evaluate(InMemoryMemoizingEvaluator.java:205)
	at com.google.devtools.build.skyframe.SequentialBuildDriver.evaluate(SequentialBuildDriver.java:38)
	at com.google.devtools.build.lib.skyframe.SkyframeExecutor.configureTargets(SkyframeExecutor.java:2381)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.configureTargets(SkyframeBuildView.java:373)
	at com.google.devtools.build.lib.analysis.BuildView.update(BuildView.java:399)
	at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.runAnalysisPhase(AnalysisPhaseRunner.java:198)
	at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:110)
	at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:141)
	at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:268)
	at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:98)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:524)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:192)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:573)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$2(GrpcServerImpl.java:624)
	at java.base/java.lang.Thread.run(Unknown Source)
java.util.MissingFormatArgumentException: Format specifier '%s'
	at java.base/java.util.Formatter.format(Unknown Source)
	at java.base/java.util.Formatter.format(Unknown Source)
	at java.base/java.lang.String.format(Unknown Source)
	at com.google.devtools.build.lib.skyframe.SkyframeDependencyResolver.getTargets(SkyframeDependencyResolver.java:145)
	at com.google.devtools.build.lib.analysis.DependencyResolver.dependentNodeMap(DependencyResolver.java:274)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.computeDependencies(ConfiguredTargetFunction.java:562)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:317)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.bubbleErrorUp(AbstractExceptionalParallelEvaluator.java:495)
	at com.google.devtools.build.skyframe.ParallelEvaluator.bubbleErrorUpExceptionally(ParallelEvaluator.java:77)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.waitForCompletionAndConstructResult(AbstractExceptionalParallelEvaluator.java:282)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.doMutatingEvaluation(AbstractExceptionalParallelEvaluator.java:265)
	at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.evalExceptionally(AbstractExceptionalParallelEvaluator.java:201)
	at com.google.devtools.build.skyframe.ParallelEvaluator.eval(ParallelEvaluator.java:70)
	at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.evaluate(InMemoryMemoizingEvaluator.java:205)
	at com.google.devtools.build.skyframe.SequentialBuildDriver.evaluate(SequentialBuildDriver.java:38)
	at com.google.devtools.build.lib.skyframe.SkyframeExecutor.configureTargets(SkyframeExecutor.java:2381)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.configureTargets(SkyframeBuildView.java:373)
	at com.google.devtools.build.lib.analysis.BuildView.update(BuildView.java:399)
	at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.runAnalysisPhase(AnalysisPhaseRunner.java:198)
	at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:110)
	at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:141)
	at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:268)
	at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:98)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:524)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:192)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:573)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$2(GrpcServerImpl.java:624)
FAILED: Build did NOT complete successfully (7 packages loaded, 21 targets configured)

Build failure - git version required for CentOS 7 ?

I am trying to install on CentOS 7.6 64bit and installed bazel via https://docs.bazel.build/versions/master/install-redhat.html

bazel version  
Build label: 0.23.1- (@non-git)
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Mar 4 13:43:11 2019 (1551706991)
Build timestamp: 1551706991
Build timestamp as int: 1551706991

Using GCC 8.2.1

gcc --version 
gcc (GCC) 8.2.1 20180905 (Red Hat 8.2.1-3)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Git

git --version
git version 1.8.3.1
yum -y install bazel
cd /svr-setup
git clone https://github.com/google/mysql-ripple.git
cd mysql-ripple
source /opt/rh/devtoolset-8/enable
export CC=gcc
bazel build :all
bazel test :all

errors complaining of git Unknown option: -C

bazel build :all             
ERROR: /svr-setup/mysql-ripple/BUILD.bazel:492:1: no such package '@com_google_googletest//': Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/bazel_tools/tools/build_defs/repo/git.bzl", line 164
                _clone_or_update(ctx)
        File "/root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/bazel_tools/tools/build_defs/repo/git.bzl", line 74, in _clone_or_update
                fail(("error cloning %s:\n%s" % (ctx....)))
error cloning com_google_googletest:
+ cd /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external
+ rm -rf /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest
+ git clone https://github.com/google/googletest.git /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest
+ git -C /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest reset --hard 9ea01728503a445179353113d2854492f41bee84
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
+ git -C /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest fetch origin 9ea01728503a445179353113d2854492f41bee84:9ea01728503a445179353113d2854492f41bee84
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
+ git -C /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest fetch origin 9ea01728503a445179353113d2854492f41bee84:9ea01728503a445179353113d2854492f41bee84
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
 and referenced by '//:log_event_unittest'
ERROR: Analysis of target '//:log_event_unittest' failed; build aborted: no such package '@com_google_googletest//': Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/bazel_tools/tools/build_defs/repo/git.bzl", line 164
                _clone_or_update(ctx)
        File "/root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/bazel_tools/tools/build_defs/repo/git.bzl", line 74, in _clone_or_update
                fail(("error cloning %s:\n%s" % (ctx....)))
error cloning com_google_googletest:
+ cd /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external
+ rm -rf /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest
+ git clone https://github.com/google/googletest.git /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest
+ git -C /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest reset --hard 9ea01728503a445179353113d2854492f41bee84
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
+ git -C /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest fetch origin 9ea01728503a445179353113d2854492f41bee84:9ea01728503a445179353113d2854492f41bee84
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
+ git -C /root/.cache/bazel/_bazel_root/09eccf19d891989ba4aefba106852854/external/com_google_googletest fetch origin 9ea01728503a445179353113d2854492f41bee84:9ea01728503a445179353113d2854492f41bee84
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
INFO: Elapsed time: 5.563s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (8 packages loaded, 10 targets configured)
    Fetching @com_google_protobuf; Cloning a6e1cc7e328c45a0cb9856c530c8f6cd23314163 of https://github.com/protocolbuffers/protobuf.git 5s
    Fetching @com_google_googletest; Cloning 9ea01728503a445179353113d2854492f41bee84 of https://github.com/google/googletest.git

centos 7 build๏ผšfatal error: my_global.h: No such file or directory

$bazel build :all
INFO: Analyzed 50 targets (1 packages loaded, 54 targets configured).
INFO: Found 50 targets...
ERROR: /opt/mysql-ripple/BUILD.bazel:361:1: C++ compilation of rule '//:my_crypt' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 24 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from my_crypt_key_management.cc:16:0:
my_crypt_key_management.h:21:23: fatal error: my_global.h: No such file or directory
#include "my_global.h"
^
compilation terminated.
INFO: Elapsed time: 0.365s, Critical Path: 0.15s
INFO: 1 process: 1 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

$rpm -ql MariaDB-devel | grep my_global.h
/usr/include/mysql/my_global.h
/usr/include/mysql/server/my_global.h

$cat external_libs.BUILD.bazel
includes = [
"mysql",
],
hdrs = [
"mysql/server/byte_order_generic_x86_64.h",
"mysql/server/little_endian.h",
"mysql/server/m_ctype.h",
"mysql/server/m_string.h",
"mysql/server/my_alloc.h",
"mysql/server/my_attribute.h",
"mysql/server/my_byteorder.h",
"mysql/server/my_config.h",
"mysql/server/my_compiler.h",
"mysql/server/my_dbug.h",
"mysql/server/my_dir.h",
"mysql/server/my_global.h",
"mysql/server/my_list.h",
"mysql/server/my_net.h",
"mysql/server/my_pthread.h",
"mysql/server/my_sys.h",
"mysql/server/my_valgrind.h",
"mysql/server/mysql.h",
"mysql/server/mysql_com.h",
"mysql/server/mysql_time.h",
"mysql/server/mysql_version.h",
"mysql/server/mysqld_error.h",
"mysql/server/private/violite.h",
"mysql/server/typelib.h",
],

How do I monitor ripple status and parse it data files?

I am going to use ripple on a small scale in a production environment.Does Ripple has a console or similar tool?

Currently,I have two questions

  1. How do I query some indicators(eg.IO/SQL thread status,delay with master) for monitor when ripple is running
  2. How do I parse the data file of ripple to find out something i want to know(It is more used to trace the operation of the system)

Could someone help provide guidance on what I did missing ?

Password auth support

Thanks ๐Ÿ‘ I can confirm the instructions work as expected. And started exploring the tool, it seems like rippled doesnt use a password field for connection. Any idea if there is a plan to add support for that in it?

Originally posted by @akshaysuryawanshi in #1 (comment)

Part 1: Add password support for connecting to a master: #5
Part 2: Get rippled to verify passwords for things connecting to it (mysql_native_password)
Part 3: Add TLS support
Part 4: Support caching_sha2_password as used in MySQL 8.0, which requires TLS (or RSA pubkey)

On slave how to config change master

On slave,I try to issue change master to๏ผŒbut I do not know how to confirm the value of master_user and master_password๏ผŒthe following๏ผš
change master to
master_host='192.168.184.10',
master_user='???',
master_password='???',
master_port=15000,
master_auto_position=1;

Thanks in advance

Generating new binlog.index from logs

Hi!

Currently binlog.index format is like this:
filename= start_pos= start_master_pos= end_pos= last_master_pos=

Is it possible to add a new flag to generate a new binlog.index file from a list (or dir) of binlogs ? i.e: parse all binlogs and generate resulting binlog.index ? So that a slave can connect and apply all the binlogs ripple has?

slave parallel thread failure while syncing from binlog server

Hello - I was trying to use slave parallel threads to replicate from the ripple server.
However, I hit the below issue after some time:

Last_SQL_Error: Cannot execute the current event group in the parallel mode. Encountered event Gtid, relay-log name /mysql_data/relaylogs/relay-log.000005, position 27023962 which prevents execution of this event group in parallel mode. Reason: The master event is logically timestamped incorrectly.

I tracked this down to a binlog rotation happening on the master and the sequence number getting reset to 1 and hence the parallel thread failure. Could you confirm if handling the rotate events is an issue and any workarounds ?

Thanks!

can't create log file using log_dir=xxx

when ripple is running,The log is only printed on the terminal, not in the file
my config as follows:
-ripple_datadir=/data1/ripple/ripple3391/data/
-ripple_requested_start_gtid_position=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-558559972
-ripple_master_address=...
-ripple_master_port=3301
-ripple_master_user=repuser
-ripple_master_password=*****
-ripple_purge_expire_logs_days=3
-ripple_server_address=...
-ripple_server_ports=3391
-ripple_server_name=repuser
-ripple_server_password_hash=3892277DE2B05CE64880EBA13A99A441B5E60A22
-alsologtostderr=true
-log_dir=/data1/ripple/ripple3391/log
ps:the log_dir has been created

any help?? thx in advance!

Cannot read event from Percona Server 5.7.29-32-log

Hello,

I tried setting up ripple and making it read binlogs from PS 5.7.29. I had the issues:

  1. it doesn't seem to be working when GTID mode is off on the master
  2. When I turned on GTID mode, Ripple was able to connect, but the binlog file it produced seems to be corrupt:
vagrant@db11:~/mysql-ripple/data1$ mysqlbinlog --base64-output=DECODE-ROWS -vvv binlog.000000 
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#700101  0:00:00 server id 112211  end_log_pos 244 	Start: binlog v 4, server v 5.6.0-ripple created 700101  0:00:00
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
# at 244
#700101  0:00:00 server id 101  end_log_pos 359 	Start: binlog v 4, server v 5.7.29-32-log created 700101  0:00:00 at startup
ROLLBACK/*!*/;
ERROR: Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 43, event_type: -106
ERROR: Could not read entry at offset 359: Error in log format or read error.
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

Binlog from master:

vagrant@db11:~/mysql-ripple/data1$ sudo su -
root@db11:~# cd /var/lib/mysql
root@db11:/var/lib/mysql# mysqlbinlog --base64-output=DECODE-ROWS -vvv binlog.000001 
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#210109  8:19:35 server id 101  end_log_pos 123 CRC32 0xa1b3245a 	Start: binlog v 4, server v 5.7.29-32-log created 210109  8:19:35 at startup
# Warning: this binlog is either in use or was not closed properly.
ROLLBACK/*!*/;
# at 123
#210109  8:19:35 server id 101  end_log_pos 154 CRC32 0xbc7cd6d1 	Previous-GTIDs
# [empty]
# at 154
#210109  8:20:06 server id 101  end_log_pos 219 CRC32 0xb4db53c3 	GTID	last_committed=0	sequence_number=1	rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= '4ebd1ef0-9424-11ea-aa6c-02430e0005fc:1'/*!*/;
# at 219
#210109  8:20:06 server id 101  end_log_pos 303 CRC32 0x471500ba 	Query	thread_id=106	exec_time=0	error_code=0
SET TIMESTAMP=1610180406/*!*/;
SET @@session.pseudo_thread_id=106/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1077936128/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=8/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
BEGIN
/*!*/;
# at 303
#210109  8:20:06 server id 101  end_log_pos 360 CRC32 0x248b9398 	Table_map: `replication_test`.`t2` mapped to number 109
# at 360
#210109  8:20:06 server id 101  end_log_pos 400 CRC32 0x6a28cb2e 	Write_rows: table id 109 flags: STMT_END_F
### INSERT INTO `replication_test`.`t2`
### SET
###   @1=2 /* INT meta=0 nullable=0 is_null=0 */
# at 400
#210109  8:20:06 server id 101  end_log_pos 431 CRC32 0xa39f38e3 	Xid = 682
COMMIT/*!*/;
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

on master:

mysql> show master status ;
+---------------+----------+--------------+------------------+----------------------------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                      |
+---------------+----------+--------------+------------------+----------------------------------------+
| binlog.000001 |      431 |              |                  | 4ebd1ef0-9424-11ea-aa6c-02430e0005fc:1 |
+---------------+----------+--------------+------------------+----------------------------------------+
1 row in set (0.00 sec)

Ripple log:

vagrant@db11:~/mysql-ripple$ ./bazel-bin/rippled -ripple_datadir=./data1 -ripple_master_address=127.0.0.1 -ripple_master_port=3306 -ripple_master_user=repl -ripple_master_password=123 -ripple_server_ports=3307
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0109 08:19:51.753654  4388 rippled.cc:48] InitPlugins
I0109 08:19:51.754017  4388 rippled.cc:60] Setup
I0109 08:19:51.754103  4388 binlog.cc:307] Starting binlog recovery
I0109 08:19:51.754614  4388 binlog.cc:350] Scanning binlog file: binlog.000000
I0109 08:19:51.768043  4388 binlog.cc:307] Starting binlog recovery
I0109 08:19:51.768257  4388 binlog.cc:325] Found empty index, creating new file
I0109 08:19:51.791939  4388 binlog.cc:350] Scanning binlog file: binlog.000000
I0109 08:19:51.792306  4388 binlog.cc:417] Binlog recovery complete
binlog file: binlog.000000, offset: 244, gtid:
I0109 08:19:51.792382  4388 rippled.cc:121] Recovered binlog
I0109 08:19:51.794548  4388 mysql_server_port_tcpip.cc:150] Listen on host: localhost, port: 3307
I0109 08:19:51.794705  4388 rippled.cc:62] Start
I0109 08:19:51.795480  4390 mysql_master_session.cc:182] Master session starting
I0109 08:19:51.800148  4390 mysql_client_connection.cc:148] connected to host: 127.0.0.1, port: 3306
I0109 08:19:51.807732  4390 mysql_master_session.cc:137] Connected to host: 127.0.0.1, port: 3306, server_id: 101, server_name:
W0109 08:19:51.812094  4390 mysql_master_session.cc:198] master does not support semi sync
I0109 08:19:51.812681  4390 mysql_master_session.cc:207] start replicating from ''
I0109 08:19:51.813442  4390 mysql_master_session.cc:230] Master session entering main loop
I0109 08:19:51.814479  4390 binlog.cc:626] Update binlog position to end_pos: binlog.000000:462, gtid: 0-0-0
I0109 08:19:51.814869  4390 binlog.cc:616] Skip writing event [ Previous_gtids len = 27 ]
I0109 08:19:51.815209  4390 binlog.cc:626] Update binlog position to end_pos: binlog.000000:462, gtid: 0-0-0
I0109 08:20:06.321724  4390 binlog.cc:626] Update binlog position to end_pos: binlog.000000:819, gtid: 0-0-1

My goal was to use MySQL binlogs by placing them in the ripple data dir, so that it would pick up and serve them to clients. I didn't have much luck with that though. Also, there is side a effect when Ripple wipes out binlogs it thinks are impossible to read. Any thoughts on this one?

Ripple with Percona Mysql

Hello
I want to use Ripple with Percona Mysql.
The problem is that the binlog writes in the MariaDB-style GTID.
During the tests, I can not reconnect the slave.

Example:

mysql_protocol.cc:202] Client Authentication Plugin: mysql_native_password
mysql_protocol.cc:205] Authenticating user: rslave
mysql_slave_session.cc:690] Slave server_id: 16436425 server_name:  uuid: 'e17228f7-16f9-11eb-b274-fa163e29f249' request to start from: '7d5db1d9-1214-11eb-9950-d094663823b0:10942052-10942054'
binlog_reader.cc:53] Fatal error. Connecting slave requested to start from GTID 7d5db1d9-1214-11eb-9950-d094663823b0:0-0-[10942052-10942054], which is not in the master's binlog (min found 7d5db1d9-1214-11eb-9950-d094663823b0:0-0-10942051)
mysql_slave_session.cc:703] Failed to open binlog for binlog dump: Fatal error. Connecting slave requested to start from GTID 7d5db1d9-1214-11eb-9950-d094663823b0:0-0-[10942052-10942054], which is not in the master's binlog (min found 7d5db1d9-1214-11eb-9950-d094663823b0:0-0-10942051)
mysql_slave_session.cc:125] Slave session stopping
mysql_slave_session.cc:75] Slave session starting

Is it possible to write binlog in Mysql-style GTID?
Or can you tell me where and what can be edited to change GTID style?

trouble to start from GTID positions

rippled .... -ripple_requested_start_gtid_position="ca25025f-294e-11e9-821f-e4434b3e4305:3084088612"
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0410 14:01:30.890588 37048 rippled.cc:48] InitPlugins
I0410 14:01:30.890686 37048 rippled.cc:60] Setup
F0410 14:01:30.890753 37048 rippled.cc:98] Check failed: start_pos.Parse(FLAGS_ripple_requested_start_gtid_position)
*** Check failure stack trace: ***
Aborted

Please, give to me advice for this purpose
Thanks

Request for documentation for supported features

Congratulations on releaseing ripple! Appreciate the effort it took to get this released.

Is it possible please to clarify whether:

  • semi-sync replication is supported? (both master- and replica- side)
  • parallel replication should be affected? I assume replicas using writesets should be good to go; how about group-commit based replication?
  • which binlog_format are supported?
  • which binlog_row_image are supported?

Thank you!

[Build]centos build error

external/external_libs/mysql/my_sys.h:1066:53: note: suggested alternative: 'charset_info_st'
 extern size_t escape_string_for_mysql(CHARSET_INFO *charset_info,
                                                     ^~~~~~~~~~~~
                                                     charset_info_st
external/external_libs/mysql/my_sys.h:1067:39: error: expected primary-expression before 'char'
                                       char *to, size_t to_length,
                                       ^~~~
external/external_libs/mysql/my_sys.h:1067:56: error: expected primary-expression before 'to_length'
                                       char *to, size_t to_length,
                                                        ^~~~~~~~~
external/external_libs/mysql/my_sys.h:1068:39: error: expected primary-expression before 'const'
                                       const char *from, size_t length,
                                       ^~~~~
external/external_libs/mysql/my_sys.h:1068:64: error: expected primary-expression before 'length'
                                       const char *from, size_t length,
                                                                ^~~~~~
external/external_libs/mysql/my_sys.h:1069:47: error: expected primary-expression before '*' token
                                       my_bool *overflow);
                                               ^
external/external_libs/mysql/my_sys.h:1069:48: error: 'overflow' was not declared in this scope
                                       my_bool *overflow);
                                                ^~~~~~~~
external/external_libs/mysql/my_sys.h:1069:48: note: suggested alternative: 'erfl'
                                       my_bool *overflow);
                                                ^~~~~~~~
                                                erfl
external/external_libs/mysql/my_sys.h:1069:56: error: expression list treated as compound expression in initializer [-fpermissive]
                                       my_bool *overflow);
                                                        ^
external/external_libs/mysql/my_sys.h:1077:39: error: 'CHARSET_INFO' was not declared in this scope
 extern size_t escape_quotes_for_mysql(CHARSET_INFO *charset_info,
                                       ^~~~~~~~~~~~
external/external_libs/mysql/my_sys.h:1077:39: note: suggested alternative: 'MY_CHARSET_INFO'
 extern size_t escape_quotes_for_mysql(CHARSET_INFO *charset_info,
                                       ^~~~~~~~~~~~
                                       MY_CHARSET_INFO
external/external_libs/mysql/my_sys.h:1077:53: error: 'charset_info' was not declared in this scope
 extern size_t escape_quotes_for_mysql(CHARSET_INFO *charset_info,
                                                     ^~~~~~~~~~~~
external/external_libs/mysql/my_sys.h:1077:53: note: suggested alternative: 'charset_info_st'
 extern size_t escape_quotes_for_mysql(CHARSET_INFO *charset_info,
                                                     ^~~~~~~~~~~~
                                                     charset_info_st
external/external_libs/mysql/my_sys.h:1078:39: error: expected primary-expression before 'char'
                                       char *to, size_t to_length,
                                       ^~~~
external/external_libs/mysql/my_sys.h:1078:56: error: expected primary-expression before 'to_length'
                                       char *to, size_t to_length,
                                                        ^~~~~~~~~
external/external_libs/mysql/my_sys.h:1079:39: error: expected primary-expression before 'const'
                                       const char *from, size_t length,
                                       ^~~~~
external/external_libs/mysql/my_sys.h:1079:64: error: expected primary-expression before 'length'
                                       const char *from, size_t length,
                                                                ^~~~~~
external/external_libs/mysql/my_sys.h:1080:47: error: expected primary-expression before '*' token
                                       my_bool *overflow);
                                               ^
external/external_libs/mysql/my_sys.h:1080:48: error: 'overflow' was not declared in this scope
                                       my_bool *overflow);
                                                ^~~~~~~~
external/external_libs/mysql/my_sys.h:1080:48: note: suggested alternative: 'erfl'
                                       my_bool *overflow);
                                                ^~~~~~~~
                                                erfl
external/external_libs/mysql/my_sys.h:1080:56: error: expression list treated as compound expression in initializer [-fpermissive]
                                       my_bool *overflow);
                                                        ^
external/external_libs/mysql/my_sys.h:1103:1: error: 'C_MODE_END' does not name a type; did you mean 'COM_END'?
 C_MODE_END
 ^~~~~~~~~~
 COM_END
In file included from external/external_libs/mysql/private/violite.h:26,
                 from mysql_server_connection.cc:27:
/usr/include/mysql/psi/mysql_socket.h:115:22: error: return type 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'} is incomplete
 mysql_socket_invalid()
                      ^
/usr/include/mysql/psi/mysql_socket.h: In function 'void mysql_socket_invalid()':
/usr/include/mysql/psi/mysql_socket.h:117:16: error: variable 'MYSQL_SOCKET mysql_socket' has initializer but incomplete type
   MYSQL_SOCKET mysql_socket= {INVALID_SOCKET, 0, 0, 0, NULL};
                ^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:117:31: error: 'INVALID_SOCKET' was not declared in this scope
   MYSQL_SOCKET mysql_socket= {INVALID_SOCKET, 0, 0, 0, NULL};
                               ^~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:117:31: note: suggested alternative: 'MYSQL_SOCKET'
   MYSQL_SOCKET mysql_socket= {INVALID_SOCKET, 0, 0, 0, NULL};
                               ^~~~~~~~~~~~~~
                               MYSQL_SOCKET
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:135:16: error: 'socket' has incomplete type
   MYSQL_SOCKET socket __attribute__ ((unused)),
   ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:156:14: error: 'socket' has incomplete type
 MYSQL_SOCKET socket __attribute__ ((unused))
 ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:172:33: error: 'mysql_socket' has incomplete type
 mysql_socket_getfd(MYSQL_SOCKET mysql_socket)
                    ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: In function 'void mysql_socket_setfd(MYSQL_SOCKET*, my_socket)':
/usr/include/mysql/psi/mysql_socket.h:186:7: error: 'likely' was not declared in this scope
   if (likely(mysql_socket != NULL))
       ^~~~~~
/usr/include/mysql/psi/mysql_socket.h:187:17: error: invalid use of incomplete type 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
     mysql_socket->fd= fd;
                 ^~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:605:9: error: return type 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'} is incomplete
   int fd)
         ^
/usr/include/mysql/psi/mysql_socket.h: In function 'void inline_mysql_socket_fd(int)':
/usr/include/mysql/psi/mysql_socket.h:607:16: error: variable 'MYSQL_SOCKET mysql_socket' has initializer but incomplete type
   MYSQL_SOCKET mysql_socket= MYSQL_INVALID_SOCKET;
                ^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:610:34: error: 'INVALID_SOCKET' was not declared in this scope
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
                                  ^~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:610:34: note: suggested alternative: 'MYSQL_SOCKET'
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
                                  ^~~~~~~~~~~~~~
                                  MYSQL_SOCKET
/usr/include/mysql/psi/mysql_socket.h:610:3: error: 'DBUG_ASSERT' was not declared in this scope
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
   ^~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:610:3: note: suggested alternative: 'ABSL_ASSERT'
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
   ^~~~~~~~~~~
   ABSL_ASSERT
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:638:37: error: return type 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'} is incomplete
   int domain, int type, int protocol)
                                     ^
/usr/include/mysql/psi/mysql_socket.h: In function 'void inline_mysql_socket_socket(int, int, int)':
/usr/include/mysql/psi/mysql_socket.h:640:16: error: variable 'MYSQL_SOCKET mysql_socket' has initializer but incomplete type
   MYSQL_SOCKET mysql_socket= MYSQL_INVALID_SOCKET;
                ^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:667:16: error: 'mysql_socket' has incomplete type
   MYSQL_SOCKET mysql_socket, const struct sockaddr *addr, size_t len)
   ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:707:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, struct sockaddr *addr, socklen_t *len)
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:745:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, const struct sockaddr *addr, socklen_t len)
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:783:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, struct sockaddr *addr, socklen_t *len)
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:821:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, const SOCKBUF_T *buf, size_t n, int flags)
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: In function 'ssize_t inline_mysql_socket_send(MYSQL_SOCKET, const void*, size_t, int)':
/usr/include/mysql/psi/mysql_socket.h:824:34: error: 'INVALID_SOCKET' was not declared in this scope
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
                                  ^~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:824:34: note: suggested alternative: 'MYSQL_SOCKET'
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
                                  ^~~~~~~~~~~~~~
                                  MYSQL_SOCKET
/usr/include/mysql/psi/mysql_socket.h:824:3: error: 'DBUG_ASSERT' was not declared in this scope
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
   ^~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:824:3: note: suggested alternative: 'ABSL_ASSERT'
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
   ^~~~~~~~~~~
   ABSL_ASSERT
/usr/include/mysql/psi/mysql_socket.h:849:46: error: expected primary-expression before 'int'
   result= send(mysql_socket.fd, buf, IF_WIN((int),) n, flags);
                                              ^~~
/usr/include/mysql/psi/mysql_socket.h:849:46: error: expected ')' before 'int'
   result= send(mysql_socket.fd, buf, IF_WIN((int),) n, flags);
                                             ~^~~
                                              )
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:862:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket,  SOCKBUF_T *buf, size_t n, int flags)
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: In function 'ssize_t inline_mysql_socket_recv(MYSQL_SOCKET, void*, size_t, int)':
/usr/include/mysql/psi/mysql_socket.h:865:34: error: 'INVALID_SOCKET' was not declared in this scope
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
                                  ^~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:865:34: note: suggested alternative: 'MYSQL_SOCKET'
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
                                  ^~~~~~~~~~~~~~
                                  MYSQL_SOCKET
/usr/include/mysql/psi/mysql_socket.h:865:3: error: 'DBUG_ASSERT' was not declared in this scope
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
   ^~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:865:3: note: suggested alternative: 'ABSL_ASSERT'
   DBUG_ASSERT(mysql_socket.fd != INVALID_SOCKET);
   ^~~~~~~~~~~
   ABSL_ASSERT
/usr/include/mysql/psi/mysql_socket.h:890:46: error: expected primary-expression before 'int'
   result= recv(mysql_socket.fd, buf, IF_WIN((int),) n, flags);
                                              ^~~
/usr/include/mysql/psi/mysql_socket.h:890:46: error: expected ')' before 'int'
   result= recv(mysql_socket.fd, buf, IF_WIN((int),) n, flags);
                                             ~^~~
                                              )
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:903:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, const SOCKBUF_T *buf, size_t n, int flags, const struct sockaddr *addr, socklen_t addr_len)
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: In function 'ssize_t inline_mysql_socket_sendto(MYSQL_SOCKET, const void*, size_t, int, const sockaddr*, socklen_t)':
/usr/include/mysql/psi/mysql_socket.h:931:48: error: expected primary-expression before 'int'
   result= sendto(mysql_socket.fd, buf, IF_WIN((int),) n, flags, addr, addr_len);
                                                ^~~
/usr/include/mysql/psi/mysql_socket.h:931:48: error: expected ')' before 'int'
   result= sendto(mysql_socket.fd, buf, IF_WIN((int),) n, flags, addr, addr_len);
                                               ~^~~
                                                )
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:944:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, SOCKBUF_T *buf, size_t n, int flags,
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: In function 'ssize_t inline_mysql_socket_recvfrom(MYSQL_SOCKET, void*, size_t, int, sockaddr*, socklen_t*)':
/usr/include/mysql/psi/mysql_socket.h:973:50: error: expected primary-expression before 'int'
   result= recvfrom(mysql_socket.fd, buf, IF_WIN((int),) n, flags, addr, addr_len);
                                                  ^~~
/usr/include/mysql/psi/mysql_socket.h:973:50: error: expected ')' before 'int'
   result= recvfrom(mysql_socket.fd, buf, IF_WIN((int),) n, flags, addr, addr_len);
                                                 ~^~~
                                                  )
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:986:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, int level, int optname, SOCKBUF_T *optval, socklen_t *optlen)
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:1024:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, int level, int optname, const SOCKBUF_T *optval,
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: In function 'int set_socket_nonblock(my_socket)':
/usr/include/mysql/psi/mysql_socket.h:1068:25: error: 'F_GETFL' was not declared in this scope
     fd_flags= fcntl(fd, F_GETFL, 0);
                         ^~~~~~~
/usr/include/mysql/psi/mysql_socket.h:1068:15: error: 'fcntl' was not declared in this scope
     fd_flags= fcntl(fd, F_GETFL, 0);
               ^~~~~
/usr/include/mysql/psi/mysql_socket.h:1068:15: note: suggested alternative: 'fcvt'
     fd_flags= fcntl(fd, F_GETFL, 0);
               ^~~~~
               fcvt
/usr/include/mysql/psi/mysql_socket.h:1080:19: error: 'F_SETFL' was not declared in this scope
     if (fcntl(fd, F_SETFL, fd_flags) == -1)
                   ^~~~~~~
/usr/include/mysql/psi/mysql_socket.h:1080:19: note: suggested alternative: 'FD_SET'
     if (fcntl(fd, F_SETFL, fd_flags) == -1)
                   ^~~~~~~
                   FD_SET
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:1095:16: error: 'mysql_socket' has incomplete type
   MYSQL_SOCKET mysql_socket
   ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:1135:15: error: 'mysql_socket' has incomplete type
  MYSQL_SOCKET mysql_socket, int backlog)
  ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:1173:16: error: 'socket_listen' has incomplete type
   MYSQL_SOCKET socket_listen, struct sockaddr *addr, socklen_t *addr_len)
   ~~~~~~~~~~~~~^~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:1173:73: error: return type 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'} is incomplete
   MYSQL_SOCKET socket_listen, struct sockaddr *addr, socklen_t *addr_len)
                                                                         ^
/usr/include/mysql/psi/mysql_socket.h: In function 'void inline_mysql_socket_accept(MYSQL_SOCKET, sockaddr*, socklen_t*)':
/usr/include/mysql/psi/mysql_socket.h:1179:16: error: aggregate 'MYSQL_SOCKET socket_accept' has incomplete type and cannot be defined
   MYSQL_SOCKET socket_accept;
                ^~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:1254:16: error: 'mysql_socket' has incomplete type
   MYSQL_SOCKET mysql_socket)
   ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h: In function 'int inline_mysql_socket_close(MYSQL_SOCKET)':
/usr/include/mysql/psi/mysql_socket.h:1282:11: error: 'closesocket' was not declared in this scope
   result= closesocket(mysql_socket.fd);
           ^~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:1282:11: note: suggested alternative: 'PSI_socket'
   result= closesocket(mysql_socket.fd);
           ^~~~~~~~~~~
           PSI_socket
/usr/include/mysql/psi/mysql_socket.h: At global scope:
/usr/include/mysql/psi/mysql_socket.h:1295:16: error: 'mysql_socket' has incomplete type
   MYSQL_SOCKET mysql_socket, int how)
   ~~~~~~~~~~~~~^~~~~~~~~~~~
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
In file included from mysql_server_connection.cc:27:
external/external_libs/mysql/private/violite.h:87:27: error: 'uchar' has not been declared
 size_t vio_read(Vio *vio, uchar * buf, size_t size);
                           ^~~~~
external/external_libs/mysql/private/violite.h:88:33: error: 'uchar' has not been declared
 size_t  vio_read_buff(Vio *vio, uchar * buf, size_t size);
                                 ^~~~~
external/external_libs/mysql/private/violite.h:89:34: error: 'uchar' does not name a type; did you mean 'u_char'?
 size_t vio_write(Vio *vio, const uchar * buf, size_t size);
                                  ^~~~~
                                  u_char
external/external_libs/mysql/private/violite.h:111:44: error: 'uint16' has not been declared
 my_bool vio_peer_addr(Vio *vio, char *buf, uint16 *port, size_t buflen);
                                            ^~~~~~
external/external_libs/mysql/private/violite.h:241:17: error: field 'mysql_socket' has incomplete type 'MYSQL_SOCKET' {aka 'st_mysql_socket'}
   MYSQL_SOCKET  mysql_socket;     /* Instrumented socket */
                 ^~~~~~~~~~~~
In file included from external/external_libs/mysql/private/violite.h:26,
                 from mysql_server_connection.cc:27:
/usr/include/mysql/psi/mysql_socket.h:99:16: note: forward declaration of 'MYSQL_SOCKET' {aka 'struct st_mysql_socket'}
 typedef struct st_mysql_socket MYSQL_SOCKET;
                ^~~~~~~~~~~~~~~
In file included from mysql_server_connection.cc:27:
external/external_libs/mysql/private/violite.h:257:25: error: 'uchar' has not been declared
   size_t  (*read)(Vio*, uchar *, size_t);
                         ^~~~~
external/external_libs/mysql/private/violite.h:258:32: error: 'uchar' does not name a type; did you mean 'u_char'?
   size_t  (*write)(Vio*, const uchar *, size_t);
                                ^~~~~
                                u_char
external/external_libs/mysql/private/violite.h:264:38: error: 'uint16' has not been declared
   my_bool (*peer_addr)(Vio*, char *, uint16*, size_t);
                                      ^~~~~~
mysql_server_connection.cc:35:36: error: 'uint16' has not been declared
                                    uint16 port)
                                    ^~~~~~
mysql_server_connection.cc:34:1: error: no declaration matches 'mysql_ripple::mysql::ServerConnection::ServerConnection(MYSQL*, std::__cxx11::string, int)'
 ServerConnection::ServerConnection(MYSQL *mysql, std::string address,
 ^~~~~~~~~~~~~~~~
In file included from mysql_server_connection.cc:15:
mysql_server_connection.h:39:7: note: candidates are: 'mysql_ripple::mysql::ServerConnection::ServerConnection(const mysql_ripple::mysql::ServerConnection&)'
 class ServerConnection : public Connection {
       ^~~~~~~~~~~~~~~~
mysql_server_connection.h:87:3: note:                 'mysql_ripple::mysql::ServerConnection::ServerConnection(MYSQL*, std::__cxx11::string, uint16_t)'
   ServerConnection(MYSQL *mysql, std::string address, uint16_t port);
   ^~~~~~~~~~~~~~~~
mysql_server_connection.h:39:7: note: 'class mysql_ripple::mysql::ServerConnection' defined here
 class ServerConnection : public Connection {
       ^~~~~~~~~~~~~~~~
In file included from mysql_server_connection.cc:27:
mysql_server_connection.cc: In static member function 'static mysql_ripple::mysql::ServerConnection* mysql_ripple::mysql::ServerConnection::Accept(Vio*)':
mysql_server_connection.cc:120:3: error: cannot convert 'uint16_t*' {aka 'short unsigned int*'} to 'int*' in argument passing
   vio_peer_addr(vio, address, &port, sizeof(address));
   ^~~~~~~~~~~~~
MariaDB 10.6

Is the version too new?

Steps to build this tool

This is pretty exciting tool, I wanted to start testing it, any estimates on when the steps to build it will be available as part of the README.

couldn't connect MySQL8.0 with caching_sha2_password

According to README.md, Ripple needs "libmariadbclient-dev" to build.
However it doesn't support "caching_sha2_password", so we couldn't connect default MySQL 8.0 with below error :

W0319 18:52:37.991261 16878 mysql_master_session.cc:164] Failed to connected to host: 192.168.10.10, port: 3306, err: Failed to connect: Authentication plugin 'caching_sha2_password' cannot be loaded: /usr/lib/x86_64-linux-gnu/mariadb18/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory

As a result, we have to enable "default_authentication_plugin= mysql_native_password" setting.
I'm very glad if we could build Ripple by latest "libmysqlclient-dev".

Not able to open the binary log files using mysqlbinlog utility.

Hi - I was using the Ripple from the link : https://lefred.be/content/ripple-binlog-server-for-mysql/
I was able to download all the binlogs by connecting to master. But when I tried to open the binlogs downloaded using the mysqlbinlog utility - I get the error :
ERROR: Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 43, event_type: -106
ERROR: Could not read entry at offset 359: Error in log format or read error.

My master is 5.7.25
binlog_format is MIXED
Async replication.

Any pointers to debug this further ?

Thanks!

Failed to read packet: Got error reading packet from server: The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION

I1008 09:31:33.836370 73346 mysql_master_session.cc:182] Master session starting
I1008 09:31:34.758591 73346 mysql_client_connection.cc:148] connected to host: x.x.x.x, port: 3306
I1008 09:31:35.422768 73346 mysql_master_session.cc:137] Connected to host: x.x.x.x, port: 3306, server_id: 604687941, server_name:
I1008 09:31:35.691968 73346 mysql_master_session.cc:202] master has semi sync enabled
I1008 09:31:35.692040 73346 mysql_master_session.cc:207] start replicating from ''
I1008 09:31:36.161306 73346 mysql_master_session.cc:230] Master session entering main loop
E1008 09:31:36.382496 73346 mysql_master_session.cc:297] Failed to read packet: Got error reading packet from server: The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.
I1008 09:31:36.382568 73346 binlog.cc:675] Connection closed last position binlog file: binlog.000000:244, gtid: 0-0-0
I1008 09:31:37.055959 73346 mysql_client_connection.cc:148] connected to host: x.x.x.x, port: 3306
I1008 09:31:37.710474 73346 mysql_master_session.cc:137] Connected to host: x.x.x.x, port: 3306, server_id: 604687941, server_name:
I1008 09:31:37.929569 73346 mysql_master_session.cc:202] master has semi sync enabled
I1008 09:31:37.929639 73346 mysql_master_session.cc:207] start replicating from ''
I1008 09:31:38.364151 73346 mysql_master_session.cc:230] Master session entering main loop
E1008 09:31:38.582463 73346 mysql_master_session.cc:297] Failed to read packet: Got error reading packet from server: The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.
I1008 09:31:38.582531 73346 binlog.cc:675] Connection closed last position binlog file: binlog.000000:244, gtid: 0-0-0
I1008 09:31:39.254842 73346 mysql_client_connection.cc:148] connected to host: x.x.x.x, port: 3306
I1008 09:31:39.916255 73346 mysql_master_session.cc:137] Connected to host: x.x.x.x, port: 3306, server_id: 604687941, server_name:

ripple Management grpc seems not yet finished?

From what I can tell, it seems that the management grpc service is not yet finished?

https://github.com/google/mysql-ripple/blob/master/management.proto
https://github.com/google/mysql-ripple/blob/master/manager.h#L43-L59 (those function are unused functions)

$ ss -tlpn
State         Recv-Q        Send-Q               Local Address:Port                Peer Address:Port       Process       
LISTEN        0             32                         0.0.0.0:51005                    0.0.0.0:*           users:(("rippled",pid=4,fd=6))

The management grpc not bound to any address? does the management grpc is internal use only?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.