Comments (9)
-
很抱歉,loader 统计耗时的地方有点小问题,并不是
UPDATE tidb_loader.checkpoin
耗时上百秒,而是整个事务的耗时。这个事务主要有三条 sql。use [db]; insert into xxxx (batch insert); UPDATE tidb_loader.checkpoint ... ;不过有一点可以确定的是,当时 tidb 真的很慢。 -
请问您的这 6 台服务器都是什么配置呢? CPU 多少核? 内存多大?
-
当时 tidb、pd、tikv 有没有相关的错误/警告日志呢? 如果有 请附带上,谢谢。
from docs-cn.
6 台 4 核 8G,硬盘 SSD
看到 TiDB 日志有大量的 tiki reports ServerIsBusy, reason: scheduler is busy
,似乎大部分的错误集中在固定的几个 region_id 上。
tikv 上的日志是这样的:
2017/07/11 11:09:15.875 apply.rs:619: [INFO] [region 2644] 2646 execute admin command cmd_type: CompactLog compact_log {compact_index: 3996 compact_term: 6} at [term: 6, index: 3998]
2017/07/11 11:09:17.585 scheduler.rs:197: [WARN] [region 2152] scheduler handle command: batch_get, ts: 393148512555499521 [takes Duration { secs: 1, nanos: 203776507 }]
2017/07/11 11:09:17.795 scheduler.rs:197: [WARN] [region 2152] scheduler handle command: resolve_lock, ts: 393137059706961921 [takes Duration { secs: 39428, nanos: 42244721 }]
2017/07/11 11:09:20.948 scheduler.rs:197: [WARN] [region 2752] scheduler handle command: batch_get, ts: 393148513328824321 [takes Duration { secs: 1, nanos: 852540755 }]
2017/07/11 11:09:21.646 scheduler.rs:197: [WARN] [region 2712] scheduler handle command: resolve_lock, ts: 393136936905605121 [takes Duration { secs: 39427, nanos: 429143668 }]
2017/07/11 11:09:25.243 scheduler.rs:197: [WARN] [region 2504] scheduler handle command: resolve_lock, ts: 393136040871198721 [takes Duration { secs: 21665, nanos: 901228119 }]
2017/07/11 11:09:28.935 scheduler.rs:197: [WARN] [region 2] scheduler handle command: resolve_lock, ts: 393137281244856321 [takes Duration { secs: 39230, nanos: 861153552 }]
2017/07/11 11:09:29.464 scheduler.rs:197: [WARN] [region 2672] scheduler handle command: resolve_lock, ts: 393136831182929921 [takes Duration { secs: 39229, nanos: 144934985 }]
2017/07/11 11:09:32.433 scheduler.rs:197: [WARN] [region 2504] scheduler handle command: resolve_lock, ts: 393136040871198721 [takes Duration { secs: 21651, nanos: 798761757 }]
2017/07/11 11:09:36.035 scheduler.rs:197: [WARN] [region 2152] scheduler handle command: resolve_lock, ts: 393137059706961921 [takes Duration { secs: 39434, nanos: 438453427 }]
2017/07/11 11:09:39.950 scheduler.rs:197: [WARN] [region 2] scheduler handle command: resolve_lock, ts: 393137281244856321 [takes Duration { secs: 39220, nanos: 928453533 }]
2017/07/11 11:09:40.972 scheduler.rs:197: [WARN] [region 2752] scheduler handle command: batch_get, ts: 393148518597918721 [takes Duration { secs: 1, nanos: 710168763 }]
2017/07/11 11:09:43.066 apply.rs:619: [INFO] [region 2380] 2382 execute admin command cmd_type: CompactLog compact_log {compact_index: 7054 compact_term: 7} at [term: 7, index: 7057]
2017/07/11 11:09:43.690 scheduler.rs:197: [WARN] [region 2504] scheduler handle command: resolve_lock, ts: 393136040871198721 [takes Duration { secs: 21641, nanos: 619822940 }]
2017/07/11 11:09:44.018 scheduler.rs:197: [WARN] [region 2152] scheduler handle command: batch_get, ts: 393148519476101121 [takes Duration { secs: 1, nanos: 252688147 }]
2017/07/11 11:09:45.328 apply.rs:619: [INFO] [region 1956] 1958 execute admin command cmd_type: CompactLog compact_log {compact_index: 18683 compact_term: 7} at [term: 7, index: 18685]
2017/07/11 11:09:45.873 apply.rs:619: [INFO] [region 2272] 2274 execute admin command cmd_type: CompactLog compact_log {compact_index: 10150 compact_term: 6} at [term: 6, index: 10152]
2017/07/11 11:09:45.876 apply.rs:619: [INFO] [region 2296] 2298 execute admin command cmd_type: CompactLog compact_log {compact_index: 11078 compact_term: 8} at [term: 8, index: 11080]
2017/07/11 11:09:47.052 scheduler.rs:197: [WARN] [region 2752] scheduler handle command: batch_get, ts: 393148520183889921 [takes Duration { secs: 1, nanos: 692004783 }]
2017/07/11 11:09:47.574 scheduler.rs:197: [WARN] [region 2152] scheduler handle command: resolve_lock, ts: 393137059706961921 [takes Duration { secs: 39445, nanos: 753905515 }]
2017/07/11 11:09:51.175 scheduler.rs:197: [WARN] [region 2712] scheduler handle command: resolve_lock, ts: 393136936905605121 [takes Duration { secs: 39448, nanos: 260540361 }]
另外 loader 出现大量错误
2017/07/11 11:10:33 db.go:147: [error] exec sqls[[***]] commit failed Error 1105: [try again later]: backoffer.maxSleep 15000ms is exceeded, errors:
server is busy, ctx: region_id:2 region_epoch:<conf_ver:3 version:67 > peer:<id:6 store_id:4 >
server is busy, ctx: region_id:2 region_epoch:<conf_ver:3 version:67 > peer:<id:6 store_id:4 >
server is busy, ctx: region_id:2 region_epoch:<conf_ver:3 version:67 > peer:<id:6 store_id:4 >
2017/07/11 11:10:33 db.go:67: [warning] exec sql retry 4 - [***]
还出现过因为多次 retry 失败后 loader 进程退出。
from docs-cn.
@zhangjinpeng1987 PTAL
from docs-cn.
另外发现三台 TiKV 服务器压力不平均,两台几乎没有负载,另一台压力很大,大量的警告日志也只出现在那台压力很大的服务器上
from docs-cn.
@qinix 能否提供相关的 grafana 截图?
from docs-cn.
各种指标有点多,你看看具体需要哪些部分的截图
from docs-cn.
你的tikv是怎么配置的? 用的默认配置?
我们测试的环境用默认的tikv的默认配置,开始导入数据, 结果一个多月只导入了1TiB左右的数据。速度完全没法接受。日志也是被ServerIsBusy塞满
后来修改了tikv的配置,把raftstore的sync-log改为false,重新导入数据,速度终于稍微能接受一些了,目前大概两个星期导入了6TiB的数据,不过我们用的都是机械硬盘。
from docs-cn.
@darren 感谢关注和使用 TiDB,我们很关注你的问题,可以给 [email protected] 发邮件同我们建立联系
from docs-cn.
我们也遇到了同样的问题,一共6台服务器,tikv都是3.2的pcie卡,内存256,lcpu 40,当在tidb上跑load的时候,其他的应用的写,基本没法做了都积压了,理论我压测下来的,性能都很高,不知道为什么会这样,最后还报错了2018/07/03 07:27:56 loader.go:124: [fatal] Error 9003: TiKV server is busy[try again later]
from docs-cn.
Related Issues (20)
- 按照文档操作,使用tidb-cdc的增量同步的时候,会出现Error: [CDC:ErrMetaListDatabases] HOT 1
- 这个edit-config前可以添加一个备份当前配置的步骤么? HOT 1
- HAProxy
- HAProxy建立连接失败
- HAPROXY建连失败 HOT 2
- Inconsistency with the English doc: txn-local-latches HOT 2
- 在解决写偏斜问题的Golang版本的案例代码出错
- tidb-jdbc驱动版本错误 HOT 2
- 文档链接 404
- DDL的执行方式(并行或者串行)描述不准确 HOT 1
- token-limit 配置项最大值不合理 HOT 3
- Fast ddl和pitr不兼容描述不太准确 HOT 2
- 那么Index Join与Index Hash Join的区别是? HOT 1
- 关于日志示例问题的疑惑 HOT 4
- 该页面内容不适用于 8.1 版本,需要更新 HOT 1
- ticdc-open-protocol 的 h 标识如何理解 HOT 1
- TiProxy 部署信息,tiproxy的服务器个数 HOT 2
- 为 PD 监控说明添加针对不同调度器事件的介绍
- auto_random(5,54) 超出分配范围 HOT 2
- PD 微服务使用文档,能否增加关于切换primary的描述 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docs-cn.