Comments (10)
siddontang
commented on August 26, 2024
Thanks, it seems that we decode a []byte data but cannot compare directly using interface{}, I will check it later.
Could you tell me what's your table schema? or the data you update?
from go-mysql-elasticsearch.
EagleChen
commented on August 26, 2024
I've no idea which row of data causes this error.
Here are the schemas of the two tables I used
+-------------------+----------------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+----------------------+------+-----+---------------------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| some1_id | int(10) unsigned | NO | MUL | NULL | |
| some2_id | int(10) unsigned | NO | MUL | NULL | |
| some3_id | int(10) unsigned | NO | MUL | NULL | |
| created_at | datetime | NO | | NULL | |
| updated_at | datetime | NO | | NULL | |
| some_time_at | datetime | NO | | 1970-01-01 00:00:00 | |
| deleted_at | datetime | NO | | 1970-01-01 00:00:00 | |
| type1 | tinyint(1) unsigned | NO | | 0 | |
| type2 | tinyint(1) unsigned | NO | | 0 | |
| count | smallint(5) unsigned | NO | | 0 | |
| field1 | varchar(64) | NO | | NULL | |
| field2 | text | NO | | NULL | |
+-------------------+----------------------+------+-----+---------------------+----------------+
+-------------+------------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------------------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| some1_id | int(10) unsigned | NO | MUL | NULL | |
| some2_id | int(10) unsigned | NO | MUL | NULL | |
| some3_id | int(10) unsigned | NO | MUL | NULL | |
| created_at | datetime | NO | | NULL | |
| updated_at | datetime | NO | | NULL | |
| deleted_at | datetime | NO | | 1970-01-01 00:00:00 | |
| somedate | date | NO | | 9999-12-31 | |
| sometype | tinyint(1) | NO | | NULL | |
| str1 | varchar(1024) | NO | | NULL | |
| text1 | text | YES | | NULL | |
| p1 | int(1) | YES | | 1 | |
| number | int(10) | YES | | NULL | |
| count | mediumint(9) | YES | | NULL | |
+-------------+------------------+------+-----+---------------------+----------------+
from go-mysql-elasticsearch.
siddontang
commented on August 26, 2024
Thanks, I will see it later, I guess the text type may cause this, maybe.
from go-mysql-elasticsearch.
siddontang
commented on August 26, 2024
@EagleChen you can update the code and check again.
from go-mysql-elasticsearch.
EagleChen
commented on August 26, 2024
@siddontang Thanks. It works now.
By the way, I notice that when the text field is synced into ES. Sometimes the content is base64-encoded.
How can I prevent this transformation?
from go-mysql-elasticsearch.
siddontang
commented on August 26, 2024
@EagleChen I don't do any extra handler for the mysql syncing data, so base64-encoded string may be already in the mysql before the syncing or changed in ES.
I just sync []byte for blob and text type, so I don't know whether ES will save this using base64, if yes, I think I will use string for these types.
from go-mysql-elasticsearch.
EagleChen
commented on August 26, 2024
@siddontang I think that's the issue.
IMO, the data in mysql is synced from mysql into ES in two steps:
- mysqldump
- binlog sync
For step 1, the text type field looks exactly the same in mysql and ES.
For step 2 ( insert , update ), the text type field is in encoded using base64.
from go-mysql-elasticsearch.
EagleChen
commented on August 26, 2024
Pull request for decode issue:
#18
from go-mysql-elasticsearch.
siddontang
commented on August 26, 2024
this seems that ES encode []byte using base64?
from go-mysql-elasticsearch.
EagleChen
commented on August 26, 2024
@siddontang I think so.
from go-mysql-elasticsearch.
Related Issues (20)
- 同步过程中服务老是出现停的情况，错误 [error] river.go:315 start canal err ERROR 1236 (HY000): A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'mysql-bin.000470' at 775402324, the last event read from './mysql-bin.000470' at 882910125, the last byte read from './mysql-bin.000470' at 882910125. HOT 1
- es 7 使用 Parent-Child Relationship
- how can I sync mysql which in docker container to elasticsearch?
- binlog error on restart (invalid character 'R' after top-level value)
- 执行exe同步报错 invalid sequence 8!=1 HOT 2
- 多个子表聚合到一个index
- 能支持dockerfile
- 能否同步one-many嵌套表作为对象关联 到es中？需要怎么配置
- 无法实现同步
- 你好，同步报错如下：sync.go:156 do ES bulk err invalid character '<' looking for beginning of value, close sync HOT 1
- mysql8 mysqldump出错 HOT 1
- 阿里云/腾讯云 mysql to es WITH MAX_USER_CONNECTIONS 10000 HOT 1
- mysql，datetime字段同步到ES的date字段，会变成加8时区的格式 HOT 1
- How do I connect to an ES cluster HOT 1
- Startup error
- start canal err line 1 column 29 near "ON *.* FROM 'root'@'%'"
- go mod tidy HOT 1
- MySql int filed to ES text filed.
- panic: runtime error: index out of range [16] with length 16
- 同步es8需要删除client.go 文件的type字段检测 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-mysql-elasticsearch.