TiDB

TiDB

TiDB Weekly [2017.02.13]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1392 次浏览 • 2017-02-18 13:54 • 来自相关话题

## Weekly update in TiDB Last two weeks, we landed [25 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is ...查看全部
## Weekly update in TiDB

Last two weeks, we landed [25 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2017-02-06..2017-02-12%20) in the TiDB repositories.

## Added

* Support basic privilege framework: [#2423](https://github.com/pingcap/tidb/pull/2423), [#2557](https://github.com/pingcap/tidb/pull/2557), [#2603](https://github.com/pingcap/tidb/pull/2603), [#2607](https://github.com/pingcap/tidb/pull/2607),

* [Support `ALTER [COLUMN] col_name SET DEFAULT` statement.](https://github.com/pingcap/tidb/pull/2608)

* [Validate column default value when creating table.](https://github.com/pingcap/tidb/pull/2614)

* [Support `ALTER TABLE ... DROP DEFAULT` statement.](https://github.com/pingcap/tidb/pull/2616)

* [Support changing default value and comment in alter table statement.](https://github.com/pingcap/tidb/pull/2621)

## Fixed

* [Fix build on i386.](https://github.com/pingcap/tidb/pull/2591)

* [Fix output format of prometheus interval log.](https://github.com/pingcap/tidb/pull/2594)

* Clean up log: [#2599](https://github.com/pingcap/tidb/pull/2599), [#2601](https://github.com/pingcap/tidb/pull/2601)

* [Fix a bug in HashJoin executor](https://github.com/pingcap/tidb/pull/2605): some errors are ignored.

* [Fix a bug in arithmetic expression type inference.](https://github.com/pingcap/tidb/pull/2610)

* [Fix a bug in builtin function timediff.](https://github.com/pingcap/tidb/pull/2611)


## Improved

* [Speed up unit tests.](https://github.com/pingcap/tidb/pull/2590)

* Make test cases more stable: [#2595](https://github.com/pingcap/tidb/pull/2595), [#2596](https://github.com/pingcap/tidb/pull/2596)

* [Remove useless code in parser package.](https://github.com/pingcap/tidb/pull/2604)

* [Change string default collation to utf8_bin.](https://github.com/pingcap/tidb/pull/2617) We do not support case insensitive comparation yet.

# Weekly update in TiKV

Last week, We landed [14 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2017-02-05..2017-02-11&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Use [`seek_for_prev`](https://github.com/pingcap/tikv/pull/1581) directly.

* Refactor [`cut_row` and `cut_idx_key`](https://github.com/pingcap/tikv/pull/1590) for row value analysis in coprocessor.

* Extract an [interface](https://github.com/pingcap/tikv/pull/1579) to support different snapshot format.

## Fixed

* Fix panic when [stop scheduler](https://github.com/pingcap/tikv/pull/1580) worker.

* Fix [invalid link](https://github.com/pingcap/pd/pull/500) in PD readme.

## Improved

* Evaluate [logic operations](https://github.com/pingcap/tikv/pull/1565) lazily.

* Use [prefix seek](https://github.com/pingcap/tikv/pull/1509) to speed up read for Write CF.

* Separate [`advance` to `advance_ready` and `advance_apply`](https://github.com/pingcap/tikv/pull/1573) for async-apply later.

* Use [`shutdown`](https://github.com/pingcap/tikv/pull/1586) to do cleanup when stop worker.

[原文链接](http://weekly.pingcap.com/2017/02/13/tidb-weekly/)

TiDB Weekly [2017.02.05]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1324 次浏览 • 2017-02-18 13:52 • 来自相关话题

## Weekly update in TiDB Last two weeks, we landed [43 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is ...查看全部
## Weekly update in TiDB

Last two weeks, we landed [43 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2017-01-23..2017-02-05%20) in the TiDB repositories.

## Added

* [Support information_schema.table_constraints.](https://github.com/pingcap/tidb/pull/2586)

* [Support the `UTC_TIMESTAMP` builtin function](https://github.com/pingcap/tidb/pull/2592)

* [Increase transaction entry count limit.](https://github.com/pingcap/tidb/pull/2537)

* Add logs for expensive query and big transaction: [2536](https://github.com/pingcap/tidb/pull/2536), [2545](https://github.com/pingcap/tidb/pull/2545), [2546](https://github.com/pingcap/tidb/pull/2546)

## Fixed

* [Fix GC lifetime metrics.](https://github.com/pingcap/tidb/pull/2587)

* [Fix primary key name parsing.](https://github.com/pingcap/tidb/pull/2582)

* [Fix a bug of left outer semi join.](https://github.com/pingcap/tidb/pull/2573)

* [Fix a bug of exists sub query.](https://github.com/pingcap/tidb/pull/2549)

* [Fix a bug abaout prefix index.](https://github.com/pingcap/tidb/pull/2445)

## Improved

* [Only plans that have apply will add cache.](https://github.com/pingcap/tidb/pull/2564)

* [Use a short-cut way to way And/Or/Xor expresson.](https://github.com/pingcap/tidb/pull/2561)

* [Refine the range calculating.](https://github.com/pingcap/tidb/pull/2534)

# Weekly update in TiKV

Last two weeks, we landed [20 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2017-01-22..2017-02-04&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

+ Add [`start_ts`](https://github.com/pingcap/tikv/pull/1511) for scan in `tikv-ctl`.

+ Support [HTTP scheme](https://github.com/pingcap/tikv/pull/1543) parsing.

## Fixed

+ Fix [too long](https://github.com/pingcap/tikv/pull/1566) scan slow log.

## Improved

+ Use [`reverse_seek_le`](https://github.com/pingcap/tikv/pull/1498) for the reverse seek write.

+ [Ignore overwriting data](https://github.com/pingcap/tikv/pull/1500) in `prewrite`.

+ [Update raft in raftstore](https://github.com/pingcap/tikv/pull/1540) after handling ConfChange.

+ Make [coprocessor task](https://github.com/pingcap/tikv/pull/1551) run more concurrently.

+ [Ignore deleting the value](https://github.com/pingcap/tikv/pull/1553) when roll back `lock`.

+ [Update leader lease](https://github.com/pingcap/tikv/pull/1560) before applying raft log.


[原文链接](http://weekly.pingcap.com/2017/02/05/tidb-weekly/)

【PingCAP】SRE & Tools Engineer

招聘应聘qiuyesuifeng 发表了文章 • 0 个评论 • 1458 次浏览 • 2017-02-18 13:45 • 来自相关话题

岗位职责: - 维护 TiDB 在生产系统中平稳运行,包括产品部署、监控和线上调试,及时应对并处理关键服务模块相关的问题 - 构建各种自动化流程和工具,以自动化的手段应对任何发生过和预料会发生的情况和问题 - 针对系统能力和 ...查看全部
岗位职责:
- 维护 TiDB 在生产系统中平稳运行,包括产品部署、监控和线上调试,及时应对并处理关键服务模块相关的问题
- 构建各种自动化流程和工具,以自动化的手段应对任何发生过和预料会发生的情况和问题
- 针对系统能力和需求进行评估,软件性能分析和系统调优
- 构建自动化测试系统,模拟各种灾难场景验证 TiDB 的可靠性,以测试驱动开发
- 负责内部的 DevOps 平台建设,提升团队生产效率
- TiDB 商业工具开发,包含自动化部署,数据迁移和同步工具,备份恢复工具,诊断和调试工具,系统监控工具等开发工作


职位要求:
- 2 年以上在运维开发相关领域经验
- 扎实的编程能力,熟悉 C/C++/Go/Rust/Python 一种编程语言,以及 shell/Perl 一种脚本语言
- 具备大型分布式系统的监控、分析和故障排除等相关经验
- 熟悉常用算法和数据结构,对操作系统和网络有深入的理解和应用经验
- 良好的沟通能力和技巧


加分项:
- 从事过 SRE 相关职位,具备 on-call 经历
- 爱折腾,强烈的 Hack 精神
- 熟悉 k8s 容器编排系统


待遇:

20K-40K + 期权, 13 薪 + 奖金, 优秀者可面议


联系方式:

hire@pingcap.com


地址:

北京市海淀区西小口路 66 号东升科技园 D-3 号楼 413

Or **Remote**

【PingCAP】Database Storage Engineer

招聘应聘qiuyesuifeng 发表了文章 • 0 个评论 • 1330 次浏览 • 2017-02-18 13:43 • 来自相关话题

岗位职责: - 负责分布式数据库 TiKV 相关的设计,开发 - 负责构建分布式压力测试框架,稳定性测试框架 职位要求: - 3 年以上相关领域开发经验,扎实的编程能力,精通 C/C+ ...查看全部
岗位职责:
- 负责分布式数据库 TiKV 相关的设计,开发
- 负责构建分布式压力测试框架,稳定性测试框架


职位要求:
- 3 年以上相关领域开发经验,扎实的编程能力,精通 C/C++/Go/Rust 中的一种
- 对分布式系统的架构和原理有比较深入的了解
- 优秀的发现和解决问题能力,良好的沟通能力,具备团队合作精神


加分项:
- 拥抱开源,对前沿技术有浓厚的热情和探索欲望,有开源项目经历
- 熟悉 Paxos/Raft 等分布式一致性算法
- 熟悉分布式事务模型
- 熟悉操作系统底层知识,有 TCP/IP, IO 等系统调优经验


待遇:

20K-40K + 期权, 13 薪 + 奖金, 优秀者可面议


联系方式:

hire@pingcap.com


地址:

北京市海淀区西小口路 66 号东升科技园 D-3 号楼 413
Or **Remote**

【PingCAP】Database Kernel Engineer

招聘应聘qiuyesuifeng 发表了文章 • 0 个评论 • 1237 次浏览 • 2017-02-18 13:40 • 来自相关话题

岗位职责: - 负责分布式数据库查询优化器相关的设计,开发,文档撰写和新人指导 - 负责分布式数据库 SQL 层的设计,开发和性能优化 - 参与分布式数据库底层系统存储系统的设计 职位 ...查看全部
岗位职责:
- 负责分布式数据库查询优化器相关的设计,开发,文档撰写和新人指导
- 负责分布式数据库 SQL 层的设计,开发和性能优化
- 参与分布式数据库底层系统存储系统的设计


职位要求:
- 3 年以上相关领域开发经验,扎实的编程能力,熟悉 C/C++/Go/Java/Python 中的一种
- 对分布式系统的架构和原理有比较深入的了解
- 熟悉 MapReduce/Spark/Hive 等分布式计算框架中的一种或多种
- 熟悉 MySQL/PostgreSQL/Greenplum 等数据库系统实现原理
- 优秀的发现和解决问题能力,良好的沟通能力,具备团队合作精神


加分项:
- 拥抱开源,对前沿技术有浓厚的热情和探索欲望,有开源项目经历
- 熟悉 Spark 内核,并阅读过其中的源码
- 熟悉 MySQL/PostgreSQL/Greenplum 的查询引擎,并阅读过其中的源码


待遇:

20K-40K + 期权, 13 薪 + 奖金, 优秀者可面议


联系方式:

hire@pingcap.com


地址:

北京市海淀区西小口路 66 号东升科技园 D-3 号楼 413
Or **Remote**

TiDB Weekly [2017.01.24]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1641 次浏览 • 2017-02-04 11:43 • 来自相关话题

## Weekly update in TiDB Last two weeks, we landed [87 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is ...查看全部
## Weekly update in TiDB

Last two weeks, we landed [87 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2017-01-09..2017-01-22%20) in the TiDB repositories.

## Added

* [Support statistic on index data](https://github.com/pingcap/tidb/pull/2349)

* [Support file sort operator](https://github.com/pingcap/tidb/pull/2377)

* [Add key prefix length limitation on index](https://github.com/pingcap/tidb/pull/2380)

* [Support the `TIMESTAMPDIFF` built-in function.](https://github.com/pingcap/tidb/pull/2386)

* [Support the `CONV` built-in function.](https://github.com/pingcap/tidb/pull/2390)

* [Support the `SUBSTR` built-in function.](https://github.com/pingcap/tidb/pull/2422)

* [Support the `SIGN` built-in function.](https://github.com/pingcap/tidb/pull/2427)

* [Support the `FROM_DAYS` built-in function.](https://github.com/pingcap/tidb/pull/2434)

* [Support the `FIELD` built-in function.](https://github.com/pingcap/tidb/pull/2449)

* [Support the `FLOOR` built-in function.](https://github.com/pingcap/tidb/pull/2484)

* [Support the `SQRT` built-in function.](https://github.com/pingcap/tidb/pull/2493)

* Add some metrics: [#2417](https://github.com/pingcap/tidb/pull/2417), [#2419](https://github.com/pingcap/tidb/pull/2419), [#2435](https://github.com/pingcap/tidb/pull/2435), [#2451](https://github.com/pingcap/tidb/pull/2451), [#2460](https://github.com/pingcap/tidb/pull/2460), [#2477])(https://github.com/pingcap/tidb/pull/2477)

* [Set a limitation about the key-value count and size in a single transaction.](https://github.com/pingcap/tidb/pull/2426)

* [Support `Rename Table` and `Alter Table Rename Table` statement.](https://github.com/pingcap/tidb/pull/2444)



## Fixed

* [Add a length limitation for logging sql statement when meet sytax error.](https://github.com/pingcap/tidb/pull/2415)

* [Fix a bug when binary literal has charset prefix.](https://github.com/pingcap/tidb/pull/2438)

* Fix a few bugs about optimizer: [#2439](https://github.com/pingcap/tidb/pull/2439), [#2440](https://github.com/pingcap/tidb/pull/2440), [#2452](https://github.com/pingcap/tidb/pull/2452)

* [Fix a bug about parsing aggragate function.](https://github.com/pingcap/tidb/pull/2453)

* [Fix a bug about two phrase commit.](https://github.com/pingcap/tidb/pull/2454)

* [Fix a bug about `round` function.](https://github.com/pingcap/tidb/pull/2461)

* [Fix a bug occurs when subquery contains a aggragetion.](https://github.com/pingcap/tidb/pull/2486)

* [Fix a memory leak in Join Executor.](https://github.com/pingcap/tidb/pull/2505)


## Improved

* [Use cache to store privilege info.](https://github.com/pingcap/tidb/pull/2388)

* [Refactor correlated subquery optimization.](https://github.com/pingcap/tidb/pull/2411)

* [Speed up delete statement executor.](https://github.com/pingcap/tidb/pull/2421)

* [Handle raft entry too large error.](https://github.com/pingcap/tidb/pull/2425)

* [Change the argument name for displaying version from `v` to `V`:](https://github.com/pingcap/tidb/pull/2442) keep consistent with pd/tikv.

* [Convert distinct to aggragate.](https://github.com/pingcap/tidb/pull/2515)


## New contributor

* [zhexuany](https://github.com/zhexuany)


# Weekly update in TiKV

Last two weeks, We landed [50 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2017-01-08..2017-01-21&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Add `-V` flag for [PD](https://github.com/pingcap/pd/pull/472) and [TiKV](https://github.com/pingcap/tikv/pull/1499).

* Set a limitation for the [max size](https://github.com/pingcap/tikv/pull/1480) of the raft entry.

* Make [disabling manual compaction](https://github.com/pingcap/tikv/pull/1491) configurable.

* Add fail reason tag in [raft request](https://github.com/pingcap/tikv/pull/1492) and [coprocessor](https://github.com/pingcap/tikv/pull/1518) metrics.

* Add [snapshot KV count and size](https://github.com/pingcap/tikv/pull/1493) metrics.

* Add [count of written keys](https://github.com/pingcap/tikv/pull/1496) metric.

* Add raft [log lag](https://github.com/pingcap/tikv/pull/1502) metric.

* Add [thread CPU](https://github.com/pingcap/tikv/pull/1494) collector.

* Support [`compaction_readahead_size`](https://github.com/pingcap/tikv/pull/1510) for spinning disk.

* Support [getting region with key](https://github.com/pingcap/pd/pull/474) in `pd-ctl` .

* Statistic region write key-value [count and bytes](https://github.com/pingcap/tikv/pull/1517).

## Fixed

* Fix a [pitfall](https://github.com/pingcap/pd/pull/470) when use different context package.

* Count [KV count in other CF](https://github.com/pingcap/tikv/pull/1495) when check splitting.

* [Avoid submitting secondary key](https://github.com/pingcap/tikv/pull/1515) if primary key is locked.

* Use disk size as storage capacity when [capacity is set to 0](https://github.com/pingcap/tikv/pull/1526).

* [Check locked](https://github.com/pingcap/tikv/pull/1530) for `ResolveLock` in latch.

## Improved

* [Keep large integer precision and check integer overflow](https://github.com/pingcap/tikv/pull/1437).

* Check [promotable](https://github.com/pingcap/tikv/pull/1473) when handle raft MsgTimeoutNow.

* Unify [logger](https://github.com/pingcap/tikv/pull/1476).

* [Resume paused follower](https://github.com/pingcap/tikv/pull/1483) when receive `MsgHeartbeatResp` message.

* [Reset the pending state](https://github.com/pingcap/tikv/pull/1489) when add node.

* Allow [running one GC command](https://github.com/pingcap/tikv/pull/1497) at the same time.

* Guarantee [replicas are safe](https://github.com/pingcap/pd/pull/473) if possible.

* [Keep leader alive](https://github.com/pingcap/pd/pull/475) before initialization.

* Log slow request for [scheduler](https://github.com/pingcap/tikv/pull/1520).

* Record [scan efficiency](https://github.com/pingcap/tikv/pull/1521) for coprocessor.

* Make [raft log GC](https://github.com/pingcap/tikv/pull/1535) not depend on PeerStorage.

* [Accelerate ticks](https://github.com/pingcap/tikv/pull/1544) for new split region leader.

[原文链接](http://weekly.pingcap.com/2017/01/24/tidb-weekly/)

TiDB Weekly [2017.01.08]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1263 次浏览 • 2017-02-04 11:42 • 来自相关话题

## Weekly update in TiDB Last week, we landed [38 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr ...查看全部
## Weekly update in TiDB

Last week, we landed [38 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2017-01-02..2017-01-08%20) in the TiDB repositories.

## Added

* [Add nested loop join.](https://github.com/pingcap/tidb/pull/2365)

* [Support the `UNIX_TIMESTAMP` built-in function.](https://github.com/pingcap/tidb/pull/2369)

* [Support the `INTERVAL` built-in function.](https://github.com/pingcap/tidb/pull/2370)

* [Support the `FIND_IN_SET` built-in function.](https://github.com/pingcap/tidb/pull/2373)

* [Support the `DATEDIFF` built-in function.](https://github.com/pingcap/tidb/pull/2374)

* [Enable pushing down `IF` expr to TiKV coprocessor](https://github.com/pingcap/tidb/pull/2387)

## Fixed

* [In prepared statement, limit and offset could be parameter marker.](https://github.com/pingcap/tidb/pull/2364)

* [When creating table, index option could be a list.](https://github.com/pingcap/tidb/pull/2366)

* [Fix a bug in creating table, some field length is missing.](https://github.com/pingcap/tidb/pull/2382)

* [Fix a bug about parsing datetime overflow.](https://github.com/pingcap/tidb/pull/2401)

* [Trim leading zeros before parsing int literal.](https://github.com/pingcap/tidb/pull/2404)

* [Fix float truncate bug.](https://github.com/pingcap/tidb/pull/2405)

## Improved

* [Improve TiDB schema lease checker.](https://github.com/pingcap/tidb/pull/2327)

* [Speed up alter table add index statement.](https://github.com/pingcap/tidb/pull/2341)

* [Refactor system variable related code.](https://github.com/pingcap/tidb/pull/2359)

* Refactor built-in function, add function class and function signature: [#2361](https://github.com/pingcap/tidb/pull/2361), [#2384](https://github.com/pingcap/tidb/pull/2384), [#2385](https://github.com/pingcap/tidb/pull/2385), [#2389](https://github.com/pingcap/tidb/pull/2389), [#2391](https://github.com/pingcap/tidb/pull/2391), [#2399](https://github.com/pingcap/tidb/pull/2399), [#2410](https://github.com/pingcap/tidb/pull/2410)

* [Add unique key information into plan's schema](https://github.com/pingcap/tidb/pull/2376), this will be used for plan optimizing.

* [Fetch tso and compiling statement concurrently](https://github.com/pingcap/tidb/pull/2393): reduce the latency of small transaction.

* [Load data in a batch way](https://github.com/pingcap/tidb/pull/2394): make it easier for loading large data.

* [Extract a built-in function factory for date arithmetic operations.](https://github.com/pingcap/tidb/pull/2403)


## New contributor

* [idlesummerbreeze](https://github.com/idlesummerbreeze)


# Weekly update in TiKV

Last week, We landed [15 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2017-01-02..2017-01-08&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Support [pre vote](https://github.com/pingcap/tikv/pull/1444) feature for raft.

* Schedule replicas according to the [location of the stores](https://github.com/pingcap/pd/pull/462).

* Coprocessor support [`if`](https://github.com/pingcap/tikv/pull/1459), [`IsNull`](https://github.com/pingcap/tikv/pull/1460), [`IfNull`](https://github.com/pingcap/tikv/pull/1461) and [`NullIf`](https://github.com/pingcap/tikv/pull/1463).

## Fixed

* Check [cluseter ID](https://github.com/pingcap/pd/pull/456) to avoid PD joining different cluster.

## Improved

* Return [`StoreNotMatch`](https://github.com/pingcap/tikv/pull/1457) error when store ID doesn't match.

* Use [upper bound](https://github.com/pingcap/tikv/pull/1470) for scanner.

* [Unify logger](https://github.com/pingcap/tikv/pull/1476) to make the log format the same with TiDB/PD.

[原文链接](http://weekly.pingcap.com/2017/01/08/tidb-weekly/)

TiDB Weekly [2017.01.01]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1498 次浏览 • 2017-01-07 13:13 • 来自相关话题

## Weekly update in TiDB Last week, we landed [28 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr ...查看全部
## Weekly update in TiDB

Last week, we landed [28 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-12-26..2017-01-01%20) in the TiDB repositories.

## Added

* [Support the `CHAR_LENGTH` built-in function.](https://github.com/pingcap/tidb/pull/2323)

* [Support the `CRC32` built-in function.](https://github.com/pingcap/tidb/pull/2347)

* [Support the `LEAST` built-in function.](https://github.com/pingcap/tidb/pull/2360)

## Fixed

* [Fix a bug in Add Column with invalid default value.](https://github.com/pingcap/tidb/pull/2316)

* [Fix a bug about parsing string to float.](https://github.com/pingcap/tidb/pull/2337)

* [Fix a bug when using int and uint as join key.](https://github.com/pingcap/tidb/pull/2355)

* [Fix a bug in MySQL Protocol layer about prepared statement.](https://github.com/pingcap/tidb/pull/2351)

## Improved

* [Improve string to date parser.](https://github.com/pingcap/tidb/pull/2310)

* [Improve TiDB schema lease checker.](https://github.com/pingcap/tidb/pull/2137)

* Refactor optimizer: [#2321](https://github.com/pingcap/tidb/pull/2321), [#2322](https://github.com/pingcap/tidb/pull/2322)

* [Set a limitation on the quantity of the data in a single transaction.](https://github.com/pingcap/tidb/pull/2325)

* [Improve mocked tikv](https://github.com/pingcap/tidb/pull/2331): Split data in a table into multiple mocked regions. This will used in unit tests.

* [Speed up creating table](https://github.com/pingcap/tidb/pull/2332): In some cases, it is not necessary to wait a long time to create table.

* [Use MySQL standard error code when meeting incorrect function argument count error.](https://github.com/pingcap/tidb/pull/2335)

* [Find a better way to handle StoreNotMatch error.](https://github.com/pingcap/tidb/pull/2339)

* Refactor built-in function: [#2343](https://github.com/pingcap/tidb/pull/2343), [#2344](https://github.com/pingcap/tidb/pull/2344), [#2362](https://github.com/pingcap/tidb/pull/2362), [#2367](https://github.com/pingcap/tidb/pull/2367)

* Clean up tidb-server codes: [#2356](https://github.com/pingcap/tidb/pull/2356), [#2358](https://github.com/pingcap/tidb/pull/2358)

## New contributor

* [Zyguan](https://github.com/zyguan)

* [AndreMouche](https://github.com/AndreMouche)

# Weekly update in TiKV

Last week, We landed [19 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-12-26..2017-01-01&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Move `raw_get` to [thread pool](https://github.com/pingcap/tikv/pull/1434).

* Add `ResourceKind` for operator and don't resend duplicated `AddPeer` response when the peer is still pending, see [#449](https://github.com/pingcap/pd/pull/449).

* Add [member command](https://github.com/pingcap/pd/pull/455) for `pd-ctl`.

## Fixed

* Fix getting [valid float](https://github.com/pingcap/tikv/pull/1454).

## Improved

* Update default configuration to [speed up scheduler](https://github.com/pingcap/pd/pull/452).

* [Don't panic](https://github.com/pingcap/tikv/pull/1443) when receiving stale snapshot.

* Remove unnecessary [region cache](https://github.com/pingcap/pd/pull/457).

* Remove no used [constraint feature](https://github.com/pingcap/pd/pull/458).

* [Exit](https://github.com/pingcap/tikv/pull/1451) with error message if clusert ID mismatches directly.

* Add [replication section](https://github.com/pingcap/pd/pull/460) in configuration.

[原文链接](http://weekly.pingcap.com/2017/01/01/tidb-weekly/)

TiDB Weekly [2016.12.26]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1386 次浏览 • 2017-01-07 13:12 • 来自相关话题

# New Release [TiDB RC1](https://github.com/pingcap/tidb/releases/tag/rc1) is released! ## Weekly updat ...查看全部
# New Release

[TiDB RC1](https://github.com/pingcap/tidb/releases/tag/rc1) is released!

## Weekly update in TiDB

Last week, we landed [34 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-12-19..2016-12-25%20) in the TiDB repositories.

## Added

* [Support the `RPAD` built-in function.](https://github.com/pingcap/tidb/pull/2270).

* [Support the `show keys from table from database` statement.](https://github.com/pingcap/tidb/pull/2308)

## Fixed

* [Retry infinite times if the commit primary key times out.](https://github.com/pingcap/tidb/pull/2276)

* [Do not push aggregation down to the memory tables.](https://github.com/pingcap/tidb/pull/2296)

* [Fix a bug about the `alter table` statement](https://github.com/pingcap/tidb/pull/2297).

## Improved

* Refactor the time type related code: [#2259](https://github.com/pingcap/tidb/pull/2259), [#2280](https://github.com/pingcap/tidb/pull/2280), [#2284](https://github.com/pingcap/tidb/pull/2284), #2289, [#2292](https://github.com/pingcap/tidb/pull/2292)

* Refactor optimizer: [extract initialization related code into physical Initialization.](https://github.com/pingcap/tidb/pull/2263)

* [Speed up the DDL statement.](https://github.com/pingcap/tidb/pull/2268)

* [Avoid generating `parser.go` every time.](https://github.com/pingcap/tidb/pull/2281)

* [Skip the constraint check ](https://github.com/pingcap/tidb/pull/2288)for prewrite to improve the loading data speed.

* [Speed up the `add index` statement.](https://github.com/pingcap/tidb/pull/2309)

## New contributor

* [silentred](https://github.com/silentred)

# Weekly update in TiKV

Last week, We landed [14 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-12-18..2016-12-25&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Add configuration to control the [replica scheduling speed](https://github.com/pingcap/pd/pull/426).

* [Skip constraint check](https://github.com/pingcap/tikv/pull/1411) for prewrite to improve the loading data speed.

* Add [region and store commands](https://github.com/pingcap/pd/pull/439) to pd-ctl.

* Add [configuration](https://github.com/pingcap/tikv/pull/1422) to cache index and filter blocks in the block cache.

## Fixed

* [Report snapshot](https://github.com/pingcap/tikv/pull/1394) sending status reliably to fix [#1377](https://github.com/pingcap/tikv/issues/1377).

* Store [short value](https://github.com/pingcap/tikv/pull/1407) in write cf directly to save space and improve performance.

## Improved

* Remove unnecessary [admin operators](https://github.com/pingcap/pd/pull/427).

* Handle Raft ready [append log](https://github.com/pingcap/tikv/pull/1420)[ in one WriteBatch](https://github.com/pingcap/tikv/pull/1420) to reduce the CPU usage and improve performance.

* [Remove the down peer first](https://github.com/pingcap/pd/pull/446) when scheduling replicas.

[原文链接](http://weekly.pingcap.com/2016/12/26/tidb-weekly/)

TiDB Weekly [2016.12.19]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1541 次浏览 • 2016-12-25 13:46 • 来自相关话题

# Weekly update in TiDB Last week, we landed [32 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3A ...查看全部
# Weekly update in TiDB
Last week, we landed [32 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-12-12..2016-12-18) in the TiDB repositories.

## Added

+ [Add the `FlagIgnoreTruncate/FlagTruncateAsWarning` flag to control the behavior of truncated errors.](https://github.com/pingcap/tidb/pull/2212)

+ [Add the `prompt` text flag.](https://github.com/pingcap/tidb/pull/2227)

+ [Add the rawkv metrics](https://github.com/pingcap/tidb/pull/2228) to profile the rawkv API performance.

+ [Add a comparable varint encoding/decoding method](https://github.com/pingcap/tidb/pull/2236) to make encoded data smaller.

+ [Support the `timediff` built-in function.](https://github.com/pingcap/tidb/pull/2249)

+ [Support the following built-in functions](https://github.com/pingcap/tidb/pull/2258): ln(), log(), log2(), log10().

## Fixed

+ [A bug that ignores primary key’s unsigned attribute.](https://github.com/pingcap/tidb/pull/2222)

+ [A bug that ignores error in the `distsql` layer.](https://github.com/pingcap/tidb/pull/2226)

+ [Allow default value to be Null when the column has the `auto_increament` attribute](https://github.com/pingcap/tidb/pull/2230) to be compatible with MySQL 5.6.

+ [A bug in the `insert` statement that ignores error.](https://github.com/pingcap/tidb/pull/2241)

+ Fix bugs in the cost-based optimization framework: [#2243](https://github.com/pingcap/tidb/pull/2243)

## Improved

+ Refactor the time type related code: [#2185](https://github.com/pingcap/tidb/pull/2185), [#2190](https://github.com/pingcap/tidb/pull/2190), [#2206](https://github.com/pingcap/tidb/pull/2206), [#2233](https://github.com/pingcap/tidb/pull/2233), [#2261](https://github.com/pingcap/tidb/pull/2261)

+ [Remove the util/bytes package](https://github.com/pingcap/tidb/pull/2221) to clean up the code.

+ Refactor the code to remove the `evaluator.Eval()` method: [#2222](https://github.com/pingcap/tidb/pull/2222),

+ [Improve test coverage for the `util/segmentmap` package.](https://github.com/pingcap/tidb/pull/2235)

+ [Recover from panics caused by malformated mysql packet](https://github.com/pingcap/tidb/pull/2267) to make tidb-server more robust.

## New contributor

+ [Bai Yang](https://github.com/hamo)

# Weekly update in TiKV

Last week, we landed [11 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-12-11..2016-12-17&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

+ Add a configuration to [disable data sync](https://github.com/pingcap/tikv/pull/1369) to speed up loading data.

+ Filter [the pending peers](https://github.com/pingcap/pd/pull/421) for Placement Driver (PD) scheduler.

+ Add [`pd-ctl`](https://github.com/pingcap/pd/pull/431) to operate PD more easily.

## Fixed

+ Update [the advertise peer urls](https://github.com/pingcap/pd/pull/438) from etcd to fix [#435](https://github.com/pingcap/pd/issues/435).

## Improved

+ [Read and verify snapshot](https://github.com/pingcap/tikv/pull/1381) file in one step.

+ Use a [smaller interval](https://github.com/pingcap/tikv/pull/1401) to make Raft tick more accurate.

+ Clean up [the tombstone store](https://github.com/pingcap/pd/issues/428) to fix [#401](https://github.com/pingcap/pd/issues/401).

+ Use [`delete_file_in_range`](https://github.com/pingcap/tikv/pull/1409) when clean up the tombstone regions.

[原文链接](http://weekly.pingcap.com/2016/12/19/tidb-weekly/)

TiDB Weekly [2016.12.12]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1442 次浏览 • 2016-12-17 14:14 • 来自相关话题

# Weekly update in TiDB Last week, we landed [41 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr% ...查看全部
# Weekly update in TiDB

Last week, we landed [41 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-12-05..2016-12-11%20) in the TiDB repositories.

## Added

+ [Support the built-in function: `str_to_date`. ](https://github.com/pingcap/tidb/pull/2078)

+ [Support built-in function schema().](https://github.com/pingcap/tidb/pull/2173)

+ [Support pushing the `case-when` expression to TiKV.](https://github.com/pingcap/tidb/pull/2171)

+ [Support changing the type and name of a column.](https://github.com/pingcap/tidb/pull/2174)

+ [Add the `](https://github.com/pingcap/tidb/pull/2194)[session_variables` and `plugins` of memory table to infoschema.](https://github.com/pingcap/tidb/pull/2194)

+ [Make the `union all` operator run parallelly.](https://github.com/pingcap/tidb/pull/2195)

+ [Support explaining the `union` statement.](https://github.com/pingcap/tidb/pull/2216)

## Fixed

+ [A bug that causes infinite loop.](https://github.com/pingcap/tidb/pull/2163)

+ [A bug in the `on duplicate…` statement when updating the primary key (PK)](https://github.com/pingcap/tidb/pull/2179)

+ [Make the charset name case-insensitive.](https://github.com/pingcap/tidb/pull/2184)

+ [Forbid dropping columns with the `auto_inc` and `PK` attribute.](https://github.com/pingcap/tidb/pull/2203)

+ [Fix bugs in the parser.](https://github.com/pingcap/tidb/pull/2210)

## Improved

+ [Pass the filter to TiKV when scanning indexes.](https://github.com/pingcap/tidb/pull/2166)

+ [Allocate column/index IDs in the table space](https://github.com/pingcap/tidb/pull/2205) to make the IDs shorter.

# Weekly update in TiKV

Last week, we landed [34 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-12-04..2016-12-10&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

+ Support [online backup](https://github.com/pingcap/tikv/pull/1355) of the RocksDB data for `tikv-ctl` debugging.

+ Add [replication constraints](https://github.com/pingcap/pd/pull/402) to schedule replicas.

+ Support [`GrantLeaderScheduler`](https://github.com/pingcap/pd/pull/406) to transfer all leaders to one store.

+ Support [`ShuffleLeaderScheduler`](https://github.com/pingcap/pd/pull/409) to shuffle leaders in different stores.

+ Support Circle CI for [TiKV](https://github.com/pingcap/tikv/pull/1384) and [Placement Driver (PD)](https://github.com/pingcap/pd/pull/412).

+ Add the [state filter argument](https://github.com/pingcap/pd/pull/425) to get the stores API.

## Fixed

+ [Use `channel`](https://github.com/pingcap/tikv/pull/1375) to fix the possible stale snapshot state, issue [#1373](https://github.com/pingcap/tikv/issues/1373).

+ Report [`ServerIsBusy`](https://github.com/pingcap/tikv/pull/1390) and let [PD ignore the busy store](https://github.com/pingcap/pd/issues/420) when scheduling to fix [#414](https://github.com/pingcap/pd/issues/414).

## Improved

+ [Speed up the shutdown duration](https://github.com/pingcap/tikv/pull/1385) to reduce the close waiting time.

+ [Add retry](https://github.com/pingcap/pd/pull/417) when initializing the cluster ID.

+ Support [safe ConfChange](https://github.com/pingcap/tikv/pull/1398) to fix [#1366](https://github.com/pingcap/tikv/issues/1366).

+ [Report pending peers](https://github.com/pingcap/tikv/pull/1395) to PD to improve its scheduler.

+ [Bind ports lazily](https://github.com/pingcap/tikv/pull/1400) to avoid the message channel full error when starting up.

[原文链接](http://weekly.pingcap.com/2016/12/12/tidb-weekly/)

TiDB Weekly [2016.12.05]

文章分享qiuyesuifeng 发表了文章 • 0 个评论 • 1348 次浏览 • 2016-12-07 11:42 • 来自相关话题

# Weekly update in TiDB Last week, we landed [48 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr% ...查看全部
# Weekly update in TiDB

Last week, we landed [48 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-11-28..2016-12-04) in the TiDB repositories, [6 PRs](https://github.com/pingcap/docs/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-11-28..2016-12-04%20) in the TiDB docs repositories.

## Added

+ [Support the built-in function: str\_to\_date. ](https://github.com/pingcap/tidb/pull/2078)

+ Refactor the time structure: [Introduce a `TimeInternal` interface to replace the go time representation.](https://github.com/pingcap/tidb/pull/2098)

+ [Add the raw Key-Value API ](https://github.com/pingcap/tidb/pull/2101) and make TiKV a raw Key-Value engine.

+ [Add a bench tool for the raw Key-Value ](https://github.com/pingcap/tidb/pull/2109)[API](https://github.com/pingcap/tidb/pull/2101).

+ [Support the PARTITION keyword](https://github.com/pingcap/tidb/pull/2115): parsed but ignored.

+ [Support the `Alter User` statement to change user’s password.](https://github.com/pingcap/tidb/pull/2144)

+ [Support the `set password = pwd;` statement.](https://github.com/pingcap/tidb/pull/2155)

+ [Use circleci.](https://github.com/pingcap/tidb/pull/2154)

## Fixed

+ [Check duplicate column names when adding index.](https://github.com/pingcap/tidb/pull/2103)

+ [Check duplicate index columns when creating table.](https://github.com/pingcap/tidb/pull/2123)

+ [Fix a bug when `join` exists in subquery.](https://github.com/pingcap/tidb/pull/2106)

+ [Make the `schema out of date` error retry automatically within TiDB.](https://github.com/pingcap/tidb/pull/2110)

+ [Fix a typo in the `INFORMATION_SCHEMA.COLUMNS` table.](https://github.com/pingcap/tidb/pull/2126)

+ [Fix the `date_format` type infer bug.](https://github.com/pingcap/tidb/pull/2152)

+ [Do not prune the `set variable` expression in projection.](https://github.com/pingcap/tidb/pull/2160)

## Improved

+ [Update the template for new issue.](https://github.com/pingcap/tidb/pull/2131)

+ [Use the statement context to handle truncated error.](https://github.com/pingcap/tidb/pull/2147)

+ [Push the aggregate operator down under union all.](https://github.com/pingcap/tidb/pull/2150)

## Document change

The following guides are updated:

* [TiDB Binary Deployment](https://github.com/pingcap/docs/blob/master/op-guide/binary-deployment.md)

* [TiDB Docker Deployment](https://github.com/pingcap/docs/blob/master/op-guide/docker-deployment.md)

* [Compatibility with MySQL](https://github.com/pingcap/docs/blob/master/op-guide/mysql-compatibility.md)

# Weekly update in TiKV

Last week, we landed [22 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-11-27..2016-12-03&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

+ Support [consistency check](https://github.com/pingcap/tikv/pull/1344) to find if data is corrupted or not dynamically.

+ Add configuration to control the [max running task count](https://github.com/pingcap/tikv/pull/1361).

+ Add the raw [Key-Value API](https://github.com/pingcap/tikv/pull/1364).

## Fixed

+ [Check the Placement Driver list](https://github.com/pingcap/tikv/pull/1201) to fix [#1186](https://github.com/pingcap/tikv/issues/1186).

+ Check whether the term is stale for the Raft command to fix [#1317](https://github.com/pingcap/tikv/issues/1317).

+ Schedule [the log Garbage Collection by size](https://github.com/pingcap/tikv/pull/1362) to fix [#1337](https://github.com/pingcap/tikv/issues/1337).

## Improved

+ Replace the score type with [resource kind](https://github.com/pingcap/pd/pull/393) to calculate the scores more easily.

+ Replace origin concept `balance` with [`schedule`](https://github.com/pingcap/tikv/pull/1344) and simplify configurations.

+ [Use coordinator](https://github.com/pingcap/pd/pull/398) to control the speed of different schedulers.

[原文链接](http://weekly.pingcap.com/2016/12/05/tidb-weekly/)

TiDB Weekly [2016.11.28]

文章分享qiuyesuifeng 发表了文章 • 0 个评论 • 1401 次浏览 • 2016-11-30 12:12 • 来自相关话题

# Weekly update in TiDB Last week, we landed [44 PRs](https://github.com/pingcap/tidb/pulls?utf8=✓&q=is%3Apr%20is%3Am ...查看全部
# Weekly update in TiDB

Last week, we landed [44 PRs](https://github.com/pingcap/tidb/pulls?utf8=✓&q=is%3Apr%20is%3Amerged%20merged%3A2016-11-21..2016-11-27%20) in the TiDB repositories, [3 PRs](https://github.com/pingcap/docs/pulls?utf8=✓&q=is%3Apr%20is%3Amerged%20merged%3A2016-11-21..2016-11-27%20) in the TiDB docs repositories.

## Added

+ [Support creating anonymous index.](https://github.com/pingcap/tidb/pull/2075)

+ [Add the mailing list for TiDB users.](https://github.com/pingcap/tidb/pull/2090)

+ [Support the `show events` syntax.](https://github.com/pingcap/tidb/pull/2099)

## Fixed

+ [Enlarge the ](https://github.com/pingcap/tidb/pull/2004)[Time To Live (TTL)](https://github.com/pingcap/tidb/pull/2004)[ for large transactions.](https://github.com/pingcap/tidb/pull/2004)

+ [Parse float literal using the decimal parser.](https://github.com/pingcap/tidb/pull/2044)

+ [Prevent panic for malformed packets.](https://github.com/pingcap/tidb/pull/2048)

+ [Fix the behavior in the aggregate operator:](https://github.com/pingcap/tidb/pull/2057) for the `select a, c from t groupby t.b` statement, `a` and `c` should use the first row in the group.

+[ Add sequence number in binlog](https://github.com/pingcap/tidb/pull/2060) to preserve the original mutation order.

+ [Reset the current database after dropping the current database.](https://github.com/pingcap/tidb/pull/2089)

## Improved

+ [Prevent loading schema by multiple threads.](https://github.com/pingcap/tidb/pull/1984)

+ [Make unit test run faster.](https://github.com/pingcap/tidb/pull/2010)

+ [Make explain result clearer.](https://github.com/pingcap/tidb/pull/2063)

+ [Remove `driver.go` from the TiDB project](https://github.com/pingcap/tidb/pull/2066) to enable users to use the MySQL official driver.

## Document change

Add the following new guides:

* [TiDB Cluster Troubleshooting Guide](https://github.com/pingcap/docs/blob/master/trouble-shooting.md)
* [TiKV Tuning Guide](https://github.com/pingcap/docs/blob/master/op-guide/tune-TiKV.md)

# Weekly update in TiKV

Last week, we landed [20 PRs](https://github.com/search?p=1&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-11-20..2016-11-26&ref=searchresults&type=Issues&utf8=%E2%9C%93) in the TiKV repositories.

## Added

+ Add and report the [store labels](https://github.com/pingcap/tikv/pull/1349) to Placement Driver (PD).

+ Add [pending task metrics](https://github.com/pingcap/tikv/pull/1354) for Worker.

+ Dump all the statistics about [Column Family compaction and Database](https://github.com/pingcap/tikv/pull/1357).

## Fixed

+ Use [`fs2`](https://github.com/pingcap/tikv/pull/1323) to get disk states to fix [#1318](https://github.com/pingcap/tikv/issues/1318)

+ Use the [monotonic clock time](https://github.com/pingcap/tikv/pull/1327) to improve the safety of leader lease read, to fix [#964](https://github.com/pingcap/tikv/issues/964).

+ Check the format of the [listening and advertise address](https://github.com/pingcap/tikv/pull/1335) to fix [#1332](https://github.com/pingcap/tikv/issues/1332).

+ [Get the first value](https://github.com/pingcap/tikv/pull/1339) no matter if it is Null in aggregation.

+ [Fix a Garbage Collection bug](https://github.com/pingcap/tikv/pull/1350) which deletes the latest deleted key before SafePoint.

+ [Stop attaching term to the MsgReadIndex message](https://github.com/pingcap/tikv/pull/1353) to fix [#1240](https://github.com/pingcap/tikv/issues/1240).

## Improved

+ Clean up the [command flags and configurations](https://github.com/pingcap/tikv/pull/1307) parsing.

+ [Abstract a Selector](https://github.com/pingcap/pd/pull/389) to schedule region peer.

+ Split the [`Raft Ready` handle](https://github.com/pingcap/pd/pull/1322) to two handles: `Append` and `Apply`.

+ Use [larger Heartbeat and Election timeout](https://github.com/pingcap/pd/pull/391) to reduce the network pressure.

+ [Ignore outdated tasks](https://github.com/pingcap/tikv/pull/1324) in Coprocessor to fix [#1305](https://github.com/pingcap/tikv/issues/1305).

[原文链接](http://weekly.pingcap.com/2016/11/28/tidb-weekly/)

Percolator 和 TiDB 事务算法

文章分享qiuyesuifeng 发表了文章 • 1 个评论 • 2442 次浏览 • 2016-11-24 11:24 • 来自相关话题

本文先概括的讲一下 Google Percolator 的大致流程。Percolator 是 Google 的上一代分布式事务解决方案,构建在 BigTable 之上,在 Google 内部 用于网页索引更新的业务,原始的论文[在此](http://resea ...查看全部
本文先概括的讲一下 Google Percolator 的大致流程。Percolator 是 Google 的上一代分布式事务解决方案,构建在 BigTable 之上,在 Google 内部 用于网页索引更新的业务,原始的论文[在此](http://research.google.com/pubs/pub36726.html)。原理比较简单,总体来说就是一个经过优化的二阶段提交的实现,进行了一个二级锁的优化。TiDB 的事务模型沿用了 Percolator 的事务模型。
总体的流程如下:

###读写事务

1) 事务提交前,在客户端 buffer 所有的 update/delete 操作。
2) Prewrite 阶段:

首先在所有行的写操作中选出一个作为 primary,其他的为 secondaries。

PrewritePrimary: 对 primaryRow 写入 L 列(上锁),L 列中记录本次事务的开始时间戳。写入 L 列前会检查:

1. 是否已经有别的客户端已经上锁 (Locking)。
2. 是否在本次事务开始时间之后,检查 W 列,是否有更新 [startTs, +Inf) 的写操作已经提交 (Conflict)。

在这两种种情况下会返回事务冲突。否则,就成功上锁。将行的内容写入 row 中,时间戳设置为 startTs。

将 primaryRow 的锁上好了以后,进行 secondaries 的 prewrite 流程:

1. 类似 primaryRow 的上锁流程,只不过锁的内容为事务开始时间及 primaryRow 的 Lock 的信息。
2. 检查的事项同 primaryRow 的一致。

当锁成功写入后,写入 row,时间戳设置为 startTs。

3) 以上 Prewrite 流程任何一步发生错误,都会进行回滚:删除 Lock,删除版本为 startTs 的数据。

4) 当 Prewrite 完成以后,进入 Commit 阶段,当前时间戳为 commitTs,且 commitTs> startTs :

1. commit primary:写入 W 列新数据,时间戳为 commitTs,内容为 startTs,表明数据的最新版本是 startTs 对应的数据。
2. 删除L列。

如果 primary row 提交失败的话,全事务回滚,回滚逻辑同 prewrite。如果 commit primary 成功,则可以异步的 commit secondaries, 流程和 commit primary 一致, 失败了也无所谓。

###事务中的读操作

1. 检查该行是否有 L 列,时间戳为 [0, startTs],如果有,表示目前有其他事务正占用此行,如果这个锁已经超时则尝试清除,否则等待超时或者其他事务主动解锁。注意此时不能直接返回老版本的数据,否则会发生幻读的问题。
2. 读取至 startTs 时该行最新的数据,方法是:读取 W 列,时间戳为 [0, startTs], 获取这一列的值,转化成时间戳 t, 然后读取此列于 t 版本的数据内容。

由于锁是分两级的,primary 和 seconary,只要 primary 的行锁去掉,就表示该事务已经成功 提交,这样的好处是 secondary 的 commit 是可以异步进行的,只是在异步提交进行的过程中 ,如果此时有读请求,可能会需要做一下锁的清理工作。

[原文链接](http://www.pingcap.com/blog-percolator-and-txn.html)

TiKV 的 MVCC(Multi-Version Concurrency Control)机制

文章分享qiuyesuifeng 发表了文章 • 0 个评论 • 1867 次浏览 • 2016-11-24 11:20 • 来自相关话题

### 并发控制简介 事务隔离在数据库系统中有着非常重要的作用,因为对于用户来说数据库必须提供这样一个“假象”:当前只有这么一个用户连接到了数据库中,这样可以减轻应用层的开发难度。但是,对于数据库系统来说,因为同一时间可能会存在很多用户连接,那么许 ...查看全部
### 并发控制简介
事务隔离在数据库系统中有着非常重要的作用,因为对于用户来说数据库必须提供这样一个“假象”:当前只有这么一个用户连接到了数据库中,这样可以减轻应用层的开发难度。但是,对于数据库系统来说,因为同一时间可能会存在很多用户连接,那么许多并发问题,比如数据竞争(data race),就必须解决。在这样的背景下,数据库管理系统(简称 DBMS)就必须保证并发操作产生的结果是安全的,通过可串行化(serializability)来保证。


虽然 Serilizability 是一个非常棒的概念,但是很难能够有效的实现。一个经典的方法就是使用一种[两段锁(2PL)][1]。通过 2PL,DBMS 可以维护读写锁来保证可能产生冲突的事务按照一个良好的次序(well-defined) 执行,这样就可以保证 Serializability。但是,这种通过锁的方式也有一些缺点:

1. 读锁和写锁会相互阻滞(block)。
2. 大部分事务都是只读(read-only)的,所以从事务序列(transaction-ordering)的角度来看是无害的。如果使用基于锁的隔离机制,而且如果有一段很长的读事务的话,在这段时间内这个对象就无法被改写,后面的事务就会被阻塞直到这个事务完成。这种机制对于并发性能来说影响很大。

*多版本并发控制(Multi-Version Concurrency Control, 以下简称 MVCC)*以一种优雅的方式来解决这个问题。在 MVCC 中,每当想要更改或者删除某个数据对象时,DBMS 不会在原地去删除或这修改这个已有的数据对象本身,而是创建一个该数据对象的新的版本,这样的话同时并发的读取操作仍旧可以读取老版本的数据,而写操作就可以同时进行。这个模式的好处在于,可以让读取操作不再阻塞,事实上根本就不需要锁。这是一种非常诱人的特型,以至于在很多主流的数据库中都采用了 MVCC 的实现,比如说 PostgreSQL,Oracle,Microsoft SQL Server 等。


### TiKV 中的 MVCC
让我们深入到 TiKV 中的 MVCC,了解 MVCC 在 TiKV 中是如何[实现](https://github.com/pingcap/tikv/tree/master/src/storage)的。


#### Timestamp Oracle(TSO)
因为`TiKV` 是一个分布式的储存系统,它需要一个全球性的授时服务,下文都称作 TSO(Timestamp Oracle),来分配一个单调递增的时间戳。 这样的功能在 TiKV 中是由 PD 提供的,在 Google 的 [Spanner](http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf) 中是由多个原子钟和 GPS 来提供的。


#### Storage
从源码结构上来看,想要深入理解 TiKV 中的 MVCC 部分,[src/storage](https://github.com/pingcap/tikv/blob/master/src/storage/mod.rs) 是一个非常好的入手点。 `Storage` 是实际上接受外部命令的结构体。
```rust
pub struct Storage {
engine: Box,
sendch: SendCh,
handle: Arc>,
}




impl Storage {
pub fn start(&mut self, config: &Config) -> Result<()> {
let mut handle = self.handle.lock().unwrap();
if handle.handle.is_some() {
return Err(box_err!("scheduler is already running"));
}




let engine = self.engine.clone();
let builder = thread::Builder::new().name(thd_name!("storage-scheduler"));
let mut el = handle.event_loop.take().unwrap();
let sched_concurrency = config.sched_concurrency;
let sched_worker_pool_size = config.sched_worker_pool_size;
let sched_too_busy_threshold = config.sched_too_busy_threshold;
let ch = self.sendch.clone();
let h = try!(builder.spawn(move || {
let mut sched = Scheduler::new(engine,
ch,
sched_concurrency,
sched_worker_pool_size,
sched_too_busy_threshold);
if let Err(e) = el.run(&mut sched) {
panic!("scheduler run err:{:?}", e);
}
info!("scheduler stopped");
}));
handle.handle = Some(h);




Ok(())
}
}
```
`start` 这个函数很好的解释了一个 storage 是怎么跑起来的。


#### Engine
首先是 [Engine](https://github.com/pingcap/tikv/blob/master/src/storage/engine/mod.rs#L44)。 `Engine` 是一个描述了在储存系统中接入的的实际上的数据库的接口,[raftkv](https://github.com/pingcap/tikv/blob/master/src/storage/engine/raftkv.rs#L91) 和 [Enginerocksdb](https://github.com/pingcap/tikv/blob/master/src/storage/engine/rocksdb.rs#L66) 分别实现了这个接口。


#### StorageHandle
`StorageHanle` 是处理从`sench` 接受到指令,通过 [mio](https://github.com/carllerche/mio) 来处理 IO。


接下来在`Storage`中实现了`async_get` 和`async_batch_get`等异步函数,这些函数中将对应的指令送到通道中,然后被调度器(scheduler)接收到并异步执行。


Ok,了解完`Storage` 结构体是如何实现的之后,我们终于可以接触到在`Scheduler` [被调用的 MVCC 层](https://github.com/pingcap/tikv/blob/master/src/storage/txn/scheduler.rs#L763)了。


当 storage 接收到从客户端来的指令后会将其传送到调度器中。然后调度器执行相应的过程或者调用相应的[异步函数](https://github.com/pingcap/tikv/blob/master/src/storage/txn/scheduler.rs#L643)。在调度器中有两种操作类型,读和写。读操作在 [MvccReader](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/reader.rs#L20) 中实现,这一部分很容易理解,暂且不表。写操作的部分是MVCC的核心。


#### MVCC
Ok,两段提交(2-Phase Commit,2PC)是在 MVCC 中实现的,整个 TiKV 事务模型的核心。在一段事务中,由两个阶段组成。


##### Prewrite
选择一个 row 作为 primary row, 余下的作为 secondary row。
对primary row [上锁](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L80). 在上锁之前,会检查[是否有其他同步的锁已经上到了这个 row 上](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L71) 或者是是否经有在 startTS 之后的提交操作。这两种情况都会导致冲突,一旦都冲突发生,就会[回滚(rollback)](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L115)。
对于 secondary row 重复以上操作。


##### Commit
[Rollback](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L115) 在`Prewrite` 过程中出现冲突的话就会被调用。


##### Garbage Collector
很容易发现,如果没有[垃圾收集器(Gabage Collector)](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L143) 来移除无效的版本的话,数据库中就会存有越来越多的 MVCC 版本。但是我们又不能仅仅移除某个 safe point 之前的所有版本。因为对于某个 key 来说,有可能只存在一个版本,那么这个版本就必须被保存下来。在`TiKV`中,如果在 safe point 前存在`Put` 或者`Delete`,那么说明之后所有的 writes 都是可以被移除的,不然的话只有`Delete`,`Rollback`和`Lock` 会被删除。


## TiKV-Ctl for MVCC
在开发和 debug 的过程中,我们发现查询 MVCC 的版本信息是一件非常频繁并且重要的操作。因此我们开发了新的工具来查询 MVCC 信息。`TiKV` 将 Key-Value,Locks 和Writes 分别储存在`CF_DEFAULT`,`CF_LOCK`,`CF_WRITE`中。它们以这样的格式进行编码


| | default | lock | write |
| --- | --- | --- | --- |
| **key** | z{encoded_key}{start_ts(desc)} | z{encoded_key} | z{encoded_key}{commit_ts(desc)} |
| **value** | {value} | {flag}{primary_key}{start_ts(varint)} | {flag}{start_ts(varint)} |

Details can be found [here](https://github.com/pingcap/tikv/issues/1077).


因为所有的 MVCC 信息在 Rocksdb 中都是储存在 CF Key-Value 中,所以想要查询一个 Key 的版本信息,我们只需要将这些信息以不同的方式编码,随后在对应的 CF 中查询即可。CF Key-Values 的[表示形式](https://github.com/pingcap/tikv/blob/master/src/bin/tikv-ctl.rs#L210)。

[1]: https://en.wikipedia.org/wiki/Two-phase_locking

[原文链接](http://www.pingcap.com/blog-mvcc-in-tikv.html)

[校招][实习] PingCAP 招前端开发工程师

回复

招聘应聘qiuyesuifeng 发起了问题 • 1 人关注 • 0 个回复 • 3324 次浏览 • 2016-10-11 14:30 • 来自相关话题

[校招][实习] PingCAP 招后端开发工程师

回复

招聘应聘qiuyesuifeng 发起了问题 • 1 人关注 • 0 个回复 • 3910 次浏览 • 2016-10-11 14:27 • 来自相关话题

TiDB Weekly [2017.02.13]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1392 次浏览 • 2017-02-18 13:54 • 来自相关话题

## Weekly update in TiDB Last two weeks, we landed [25 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is ...查看全部
## Weekly update in TiDB

Last two weeks, we landed [25 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2017-02-06..2017-02-12%20) in the TiDB repositories.

## Added

* Support basic privilege framework: [#2423](https://github.com/pingcap/tidb/pull/2423), [#2557](https://github.com/pingcap/tidb/pull/2557), [#2603](https://github.com/pingcap/tidb/pull/2603), [#2607](https://github.com/pingcap/tidb/pull/2607),

* [Support `ALTER [COLUMN] col_name SET DEFAULT` statement.](https://github.com/pingcap/tidb/pull/2608)

* [Validate column default value when creating table.](https://github.com/pingcap/tidb/pull/2614)

* [Support `ALTER TABLE ... DROP DEFAULT` statement.](https://github.com/pingcap/tidb/pull/2616)

* [Support changing default value and comment in alter table statement.](https://github.com/pingcap/tidb/pull/2621)

## Fixed

* [Fix build on i386.](https://github.com/pingcap/tidb/pull/2591)

* [Fix output format of prometheus interval log.](https://github.com/pingcap/tidb/pull/2594)

* Clean up log: [#2599](https://github.com/pingcap/tidb/pull/2599), [#2601](https://github.com/pingcap/tidb/pull/2601)

* [Fix a bug in HashJoin executor](https://github.com/pingcap/tidb/pull/2605): some errors are ignored.

* [Fix a bug in arithmetic expression type inference.](https://github.com/pingcap/tidb/pull/2610)

* [Fix a bug in builtin function timediff.](https://github.com/pingcap/tidb/pull/2611)


## Improved

* [Speed up unit tests.](https://github.com/pingcap/tidb/pull/2590)

* Make test cases more stable: [#2595](https://github.com/pingcap/tidb/pull/2595), [#2596](https://github.com/pingcap/tidb/pull/2596)

* [Remove useless code in parser package.](https://github.com/pingcap/tidb/pull/2604)

* [Change string default collation to utf8_bin.](https://github.com/pingcap/tidb/pull/2617) We do not support case insensitive comparation yet.

# Weekly update in TiKV

Last week, We landed [14 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2017-02-05..2017-02-11&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Use [`seek_for_prev`](https://github.com/pingcap/tikv/pull/1581) directly.

* Refactor [`cut_row` and `cut_idx_key`](https://github.com/pingcap/tikv/pull/1590) for row value analysis in coprocessor.

* Extract an [interface](https://github.com/pingcap/tikv/pull/1579) to support different snapshot format.

## Fixed

* Fix panic when [stop scheduler](https://github.com/pingcap/tikv/pull/1580) worker.

* Fix [invalid link](https://github.com/pingcap/pd/pull/500) in PD readme.

## Improved

* Evaluate [logic operations](https://github.com/pingcap/tikv/pull/1565) lazily.

* Use [prefix seek](https://github.com/pingcap/tikv/pull/1509) to speed up read for Write CF.

* Separate [`advance` to `advance_ready` and `advance_apply`](https://github.com/pingcap/tikv/pull/1573) for async-apply later.

* Use [`shutdown`](https://github.com/pingcap/tikv/pull/1586) to do cleanup when stop worker.

[原文链接](http://weekly.pingcap.com/2017/02/13/tidb-weekly/)

TiDB Weekly [2017.02.05]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1324 次浏览 • 2017-02-18 13:52 • 来自相关话题

## Weekly update in TiDB Last two weeks, we landed [43 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is ...查看全部
## Weekly update in TiDB

Last two weeks, we landed [43 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2017-01-23..2017-02-05%20) in the TiDB repositories.

## Added

* [Support information_schema.table_constraints.](https://github.com/pingcap/tidb/pull/2586)

* [Support the `UTC_TIMESTAMP` builtin function](https://github.com/pingcap/tidb/pull/2592)

* [Increase transaction entry count limit.](https://github.com/pingcap/tidb/pull/2537)

* Add logs for expensive query and big transaction: [2536](https://github.com/pingcap/tidb/pull/2536), [2545](https://github.com/pingcap/tidb/pull/2545), [2546](https://github.com/pingcap/tidb/pull/2546)

## Fixed

* [Fix GC lifetime metrics.](https://github.com/pingcap/tidb/pull/2587)

* [Fix primary key name parsing.](https://github.com/pingcap/tidb/pull/2582)

* [Fix a bug of left outer semi join.](https://github.com/pingcap/tidb/pull/2573)

* [Fix a bug of exists sub query.](https://github.com/pingcap/tidb/pull/2549)

* [Fix a bug abaout prefix index.](https://github.com/pingcap/tidb/pull/2445)

## Improved

* [Only plans that have apply will add cache.](https://github.com/pingcap/tidb/pull/2564)

* [Use a short-cut way to way And/Or/Xor expresson.](https://github.com/pingcap/tidb/pull/2561)

* [Refine the range calculating.](https://github.com/pingcap/tidb/pull/2534)

# Weekly update in TiKV

Last two weeks, we landed [20 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2017-01-22..2017-02-04&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

+ Add [`start_ts`](https://github.com/pingcap/tikv/pull/1511) for scan in `tikv-ctl`.

+ Support [HTTP scheme](https://github.com/pingcap/tikv/pull/1543) parsing.

## Fixed

+ Fix [too long](https://github.com/pingcap/tikv/pull/1566) scan slow log.

## Improved

+ Use [`reverse_seek_le`](https://github.com/pingcap/tikv/pull/1498) for the reverse seek write.

+ [Ignore overwriting data](https://github.com/pingcap/tikv/pull/1500) in `prewrite`.

+ [Update raft in raftstore](https://github.com/pingcap/tikv/pull/1540) after handling ConfChange.

+ Make [coprocessor task](https://github.com/pingcap/tikv/pull/1551) run more concurrently.

+ [Ignore deleting the value](https://github.com/pingcap/tikv/pull/1553) when roll back `lock`.

+ [Update leader lease](https://github.com/pingcap/tikv/pull/1560) before applying raft log.


[原文链接](http://weekly.pingcap.com/2017/02/05/tidb-weekly/)

【PingCAP】SRE & Tools Engineer

招聘应聘qiuyesuifeng 发表了文章 • 0 个评论 • 1458 次浏览 • 2017-02-18 13:45 • 来自相关话题

岗位职责: - 维护 TiDB 在生产系统中平稳运行,包括产品部署、监控和线上调试,及时应对并处理关键服务模块相关的问题 - 构建各种自动化流程和工具,以自动化的手段应对任何发生过和预料会发生的情况和问题 - 针对系统能力和 ...查看全部
岗位职责:
- 维护 TiDB 在生产系统中平稳运行,包括产品部署、监控和线上调试,及时应对并处理关键服务模块相关的问题
- 构建各种自动化流程和工具,以自动化的手段应对任何发生过和预料会发生的情况和问题
- 针对系统能力和需求进行评估,软件性能分析和系统调优
- 构建自动化测试系统,模拟各种灾难场景验证 TiDB 的可靠性,以测试驱动开发
- 负责内部的 DevOps 平台建设,提升团队生产效率
- TiDB 商业工具开发,包含自动化部署,数据迁移和同步工具,备份恢复工具,诊断和调试工具,系统监控工具等开发工作


职位要求:
- 2 年以上在运维开发相关领域经验
- 扎实的编程能力,熟悉 C/C++/Go/Rust/Python 一种编程语言,以及 shell/Perl 一种脚本语言
- 具备大型分布式系统的监控、分析和故障排除等相关经验
- 熟悉常用算法和数据结构,对操作系统和网络有深入的理解和应用经验
- 良好的沟通能力和技巧


加分项:
- 从事过 SRE 相关职位,具备 on-call 经历
- 爱折腾,强烈的 Hack 精神
- 熟悉 k8s 容器编排系统


待遇:

20K-40K + 期权, 13 薪 + 奖金, 优秀者可面议


联系方式:

hire@pingcap.com


地址:

北京市海淀区西小口路 66 号东升科技园 D-3 号楼 413

Or **Remote**

【PingCAP】Database Storage Engineer

招聘应聘qiuyesuifeng 发表了文章 • 0 个评论 • 1330 次浏览 • 2017-02-18 13:43 • 来自相关话题

岗位职责: - 负责分布式数据库 TiKV 相关的设计,开发 - 负责构建分布式压力测试框架,稳定性测试框架 职位要求: - 3 年以上相关领域开发经验,扎实的编程能力,精通 C/C+ ...查看全部
岗位职责:
- 负责分布式数据库 TiKV 相关的设计,开发
- 负责构建分布式压力测试框架,稳定性测试框架


职位要求:
- 3 年以上相关领域开发经验,扎实的编程能力,精通 C/C++/Go/Rust 中的一种
- 对分布式系统的架构和原理有比较深入的了解
- 优秀的发现和解决问题能力,良好的沟通能力,具备团队合作精神


加分项:
- 拥抱开源,对前沿技术有浓厚的热情和探索欲望,有开源项目经历
- 熟悉 Paxos/Raft 等分布式一致性算法
- 熟悉分布式事务模型
- 熟悉操作系统底层知识,有 TCP/IP, IO 等系统调优经验


待遇:

20K-40K + 期权, 13 薪 + 奖金, 优秀者可面议


联系方式:

hire@pingcap.com


地址:

北京市海淀区西小口路 66 号东升科技园 D-3 号楼 413
Or **Remote**

【PingCAP】Database Kernel Engineer

招聘应聘qiuyesuifeng 发表了文章 • 0 个评论 • 1237 次浏览 • 2017-02-18 13:40 • 来自相关话题

岗位职责: - 负责分布式数据库查询优化器相关的设计,开发,文档撰写和新人指导 - 负责分布式数据库 SQL 层的设计,开发和性能优化 - 参与分布式数据库底层系统存储系统的设计 职位 ...查看全部
岗位职责:
- 负责分布式数据库查询优化器相关的设计,开发,文档撰写和新人指导
- 负责分布式数据库 SQL 层的设计,开发和性能优化
- 参与分布式数据库底层系统存储系统的设计


职位要求:
- 3 年以上相关领域开发经验,扎实的编程能力,熟悉 C/C++/Go/Java/Python 中的一种
- 对分布式系统的架构和原理有比较深入的了解
- 熟悉 MapReduce/Spark/Hive 等分布式计算框架中的一种或多种
- 熟悉 MySQL/PostgreSQL/Greenplum 等数据库系统实现原理
- 优秀的发现和解决问题能力,良好的沟通能力,具备团队合作精神


加分项:
- 拥抱开源,对前沿技术有浓厚的热情和探索欲望,有开源项目经历
- 熟悉 Spark 内核,并阅读过其中的源码
- 熟悉 MySQL/PostgreSQL/Greenplum 的查询引擎,并阅读过其中的源码


待遇:

20K-40K + 期权, 13 薪 + 奖金, 优秀者可面议


联系方式:

hire@pingcap.com


地址:

北京市海淀区西小口路 66 号东升科技园 D-3 号楼 413
Or **Remote**

TiDB Weekly [2017.01.24]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1641 次浏览 • 2017-02-04 11:43 • 来自相关话题

## Weekly update in TiDB Last two weeks, we landed [87 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is ...查看全部
## Weekly update in TiDB

Last two weeks, we landed [87 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2017-01-09..2017-01-22%20) in the TiDB repositories.

## Added

* [Support statistic on index data](https://github.com/pingcap/tidb/pull/2349)

* [Support file sort operator](https://github.com/pingcap/tidb/pull/2377)

* [Add key prefix length limitation on index](https://github.com/pingcap/tidb/pull/2380)

* [Support the `TIMESTAMPDIFF` built-in function.](https://github.com/pingcap/tidb/pull/2386)

* [Support the `CONV` built-in function.](https://github.com/pingcap/tidb/pull/2390)

* [Support the `SUBSTR` built-in function.](https://github.com/pingcap/tidb/pull/2422)

* [Support the `SIGN` built-in function.](https://github.com/pingcap/tidb/pull/2427)

* [Support the `FROM_DAYS` built-in function.](https://github.com/pingcap/tidb/pull/2434)

* [Support the `FIELD` built-in function.](https://github.com/pingcap/tidb/pull/2449)

* [Support the `FLOOR` built-in function.](https://github.com/pingcap/tidb/pull/2484)

* [Support the `SQRT` built-in function.](https://github.com/pingcap/tidb/pull/2493)

* Add some metrics: [#2417](https://github.com/pingcap/tidb/pull/2417), [#2419](https://github.com/pingcap/tidb/pull/2419), [#2435](https://github.com/pingcap/tidb/pull/2435), [#2451](https://github.com/pingcap/tidb/pull/2451), [#2460](https://github.com/pingcap/tidb/pull/2460), [#2477])(https://github.com/pingcap/tidb/pull/2477)

* [Set a limitation about the key-value count and size in a single transaction.](https://github.com/pingcap/tidb/pull/2426)

* [Support `Rename Table` and `Alter Table Rename Table` statement.](https://github.com/pingcap/tidb/pull/2444)



## Fixed

* [Add a length limitation for logging sql statement when meet sytax error.](https://github.com/pingcap/tidb/pull/2415)

* [Fix a bug when binary literal has charset prefix.](https://github.com/pingcap/tidb/pull/2438)

* Fix a few bugs about optimizer: [#2439](https://github.com/pingcap/tidb/pull/2439), [#2440](https://github.com/pingcap/tidb/pull/2440), [#2452](https://github.com/pingcap/tidb/pull/2452)

* [Fix a bug about parsing aggragate function.](https://github.com/pingcap/tidb/pull/2453)

* [Fix a bug about two phrase commit.](https://github.com/pingcap/tidb/pull/2454)

* [Fix a bug about `round` function.](https://github.com/pingcap/tidb/pull/2461)

* [Fix a bug occurs when subquery contains a aggragetion.](https://github.com/pingcap/tidb/pull/2486)

* [Fix a memory leak in Join Executor.](https://github.com/pingcap/tidb/pull/2505)


## Improved

* [Use cache to store privilege info.](https://github.com/pingcap/tidb/pull/2388)

* [Refactor correlated subquery optimization.](https://github.com/pingcap/tidb/pull/2411)

* [Speed up delete statement executor.](https://github.com/pingcap/tidb/pull/2421)

* [Handle raft entry too large error.](https://github.com/pingcap/tidb/pull/2425)

* [Change the argument name for displaying version from `v` to `V`:](https://github.com/pingcap/tidb/pull/2442) keep consistent with pd/tikv.

* [Convert distinct to aggragate.](https://github.com/pingcap/tidb/pull/2515)


## New contributor

* [zhexuany](https://github.com/zhexuany)


# Weekly update in TiKV

Last two weeks, We landed [50 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2017-01-08..2017-01-21&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Add `-V` flag for [PD](https://github.com/pingcap/pd/pull/472) and [TiKV](https://github.com/pingcap/tikv/pull/1499).

* Set a limitation for the [max size](https://github.com/pingcap/tikv/pull/1480) of the raft entry.

* Make [disabling manual compaction](https://github.com/pingcap/tikv/pull/1491) configurable.

* Add fail reason tag in [raft request](https://github.com/pingcap/tikv/pull/1492) and [coprocessor](https://github.com/pingcap/tikv/pull/1518) metrics.

* Add [snapshot KV count and size](https://github.com/pingcap/tikv/pull/1493) metrics.

* Add [count of written keys](https://github.com/pingcap/tikv/pull/1496) metric.

* Add raft [log lag](https://github.com/pingcap/tikv/pull/1502) metric.

* Add [thread CPU](https://github.com/pingcap/tikv/pull/1494) collector.

* Support [`compaction_readahead_size`](https://github.com/pingcap/tikv/pull/1510) for spinning disk.

* Support [getting region with key](https://github.com/pingcap/pd/pull/474) in `pd-ctl` .

* Statistic region write key-value [count and bytes](https://github.com/pingcap/tikv/pull/1517).

## Fixed

* Fix a [pitfall](https://github.com/pingcap/pd/pull/470) when use different context package.

* Count [KV count in other CF](https://github.com/pingcap/tikv/pull/1495) when check splitting.

* [Avoid submitting secondary key](https://github.com/pingcap/tikv/pull/1515) if primary key is locked.

* Use disk size as storage capacity when [capacity is set to 0](https://github.com/pingcap/tikv/pull/1526).

* [Check locked](https://github.com/pingcap/tikv/pull/1530) for `ResolveLock` in latch.

## Improved

* [Keep large integer precision and check integer overflow](https://github.com/pingcap/tikv/pull/1437).

* Check [promotable](https://github.com/pingcap/tikv/pull/1473) when handle raft MsgTimeoutNow.

* Unify [logger](https://github.com/pingcap/tikv/pull/1476).

* [Resume paused follower](https://github.com/pingcap/tikv/pull/1483) when receive `MsgHeartbeatResp` message.

* [Reset the pending state](https://github.com/pingcap/tikv/pull/1489) when add node.

* Allow [running one GC command](https://github.com/pingcap/tikv/pull/1497) at the same time.

* Guarantee [replicas are safe](https://github.com/pingcap/pd/pull/473) if possible.

* [Keep leader alive](https://github.com/pingcap/pd/pull/475) before initialization.

* Log slow request for [scheduler](https://github.com/pingcap/tikv/pull/1520).

* Record [scan efficiency](https://github.com/pingcap/tikv/pull/1521) for coprocessor.

* Make [raft log GC](https://github.com/pingcap/tikv/pull/1535) not depend on PeerStorage.

* [Accelerate ticks](https://github.com/pingcap/tikv/pull/1544) for new split region leader.

[原文链接](http://weekly.pingcap.com/2017/01/24/tidb-weekly/)

TiDB Weekly [2017.01.08]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1263 次浏览 • 2017-02-04 11:42 • 来自相关话题

## Weekly update in TiDB Last week, we landed [38 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr ...查看全部
## Weekly update in TiDB

Last week, we landed [38 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2017-01-02..2017-01-08%20) in the TiDB repositories.

## Added

* [Add nested loop join.](https://github.com/pingcap/tidb/pull/2365)

* [Support the `UNIX_TIMESTAMP` built-in function.](https://github.com/pingcap/tidb/pull/2369)

* [Support the `INTERVAL` built-in function.](https://github.com/pingcap/tidb/pull/2370)

* [Support the `FIND_IN_SET` built-in function.](https://github.com/pingcap/tidb/pull/2373)

* [Support the `DATEDIFF` built-in function.](https://github.com/pingcap/tidb/pull/2374)

* [Enable pushing down `IF` expr to TiKV coprocessor](https://github.com/pingcap/tidb/pull/2387)

## Fixed

* [In prepared statement, limit and offset could be parameter marker.](https://github.com/pingcap/tidb/pull/2364)

* [When creating table, index option could be a list.](https://github.com/pingcap/tidb/pull/2366)

* [Fix a bug in creating table, some field length is missing.](https://github.com/pingcap/tidb/pull/2382)

* [Fix a bug about parsing datetime overflow.](https://github.com/pingcap/tidb/pull/2401)

* [Trim leading zeros before parsing int literal.](https://github.com/pingcap/tidb/pull/2404)

* [Fix float truncate bug.](https://github.com/pingcap/tidb/pull/2405)

## Improved

* [Improve TiDB schema lease checker.](https://github.com/pingcap/tidb/pull/2327)

* [Speed up alter table add index statement.](https://github.com/pingcap/tidb/pull/2341)

* [Refactor system variable related code.](https://github.com/pingcap/tidb/pull/2359)

* Refactor built-in function, add function class and function signature: [#2361](https://github.com/pingcap/tidb/pull/2361), [#2384](https://github.com/pingcap/tidb/pull/2384), [#2385](https://github.com/pingcap/tidb/pull/2385), [#2389](https://github.com/pingcap/tidb/pull/2389), [#2391](https://github.com/pingcap/tidb/pull/2391), [#2399](https://github.com/pingcap/tidb/pull/2399), [#2410](https://github.com/pingcap/tidb/pull/2410)

* [Add unique key information into plan's schema](https://github.com/pingcap/tidb/pull/2376), this will be used for plan optimizing.

* [Fetch tso and compiling statement concurrently](https://github.com/pingcap/tidb/pull/2393): reduce the latency of small transaction.

* [Load data in a batch way](https://github.com/pingcap/tidb/pull/2394): make it easier for loading large data.

* [Extract a built-in function factory for date arithmetic operations.](https://github.com/pingcap/tidb/pull/2403)


## New contributor

* [idlesummerbreeze](https://github.com/idlesummerbreeze)


# Weekly update in TiKV

Last week, We landed [15 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2017-01-02..2017-01-08&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Support [pre vote](https://github.com/pingcap/tikv/pull/1444) feature for raft.

* Schedule replicas according to the [location of the stores](https://github.com/pingcap/pd/pull/462).

* Coprocessor support [`if`](https://github.com/pingcap/tikv/pull/1459), [`IsNull`](https://github.com/pingcap/tikv/pull/1460), [`IfNull`](https://github.com/pingcap/tikv/pull/1461) and [`NullIf`](https://github.com/pingcap/tikv/pull/1463).

## Fixed

* Check [cluseter ID](https://github.com/pingcap/pd/pull/456) to avoid PD joining different cluster.

## Improved

* Return [`StoreNotMatch`](https://github.com/pingcap/tikv/pull/1457) error when store ID doesn't match.

* Use [upper bound](https://github.com/pingcap/tikv/pull/1470) for scanner.

* [Unify logger](https://github.com/pingcap/tikv/pull/1476) to make the log format the same with TiDB/PD.

[原文链接](http://weekly.pingcap.com/2017/01/08/tidb-weekly/)

TiDB Weekly [2017.01.01]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1498 次浏览 • 2017-01-07 13:13 • 来自相关话题

## Weekly update in TiDB Last week, we landed [28 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr ...查看全部
## Weekly update in TiDB

Last week, we landed [28 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-12-26..2017-01-01%20) in the TiDB repositories.

## Added

* [Support the `CHAR_LENGTH` built-in function.](https://github.com/pingcap/tidb/pull/2323)

* [Support the `CRC32` built-in function.](https://github.com/pingcap/tidb/pull/2347)

* [Support the `LEAST` built-in function.](https://github.com/pingcap/tidb/pull/2360)

## Fixed

* [Fix a bug in Add Column with invalid default value.](https://github.com/pingcap/tidb/pull/2316)

* [Fix a bug about parsing string to float.](https://github.com/pingcap/tidb/pull/2337)

* [Fix a bug when using int and uint as join key.](https://github.com/pingcap/tidb/pull/2355)

* [Fix a bug in MySQL Protocol layer about prepared statement.](https://github.com/pingcap/tidb/pull/2351)

## Improved

* [Improve string to date parser.](https://github.com/pingcap/tidb/pull/2310)

* [Improve TiDB schema lease checker.](https://github.com/pingcap/tidb/pull/2137)

* Refactor optimizer: [#2321](https://github.com/pingcap/tidb/pull/2321), [#2322](https://github.com/pingcap/tidb/pull/2322)

* [Set a limitation on the quantity of the data in a single transaction.](https://github.com/pingcap/tidb/pull/2325)

* [Improve mocked tikv](https://github.com/pingcap/tidb/pull/2331): Split data in a table into multiple mocked regions. This will used in unit tests.

* [Speed up creating table](https://github.com/pingcap/tidb/pull/2332): In some cases, it is not necessary to wait a long time to create table.

* [Use MySQL standard error code when meeting incorrect function argument count error.](https://github.com/pingcap/tidb/pull/2335)

* [Find a better way to handle StoreNotMatch error.](https://github.com/pingcap/tidb/pull/2339)

* Refactor built-in function: [#2343](https://github.com/pingcap/tidb/pull/2343), [#2344](https://github.com/pingcap/tidb/pull/2344), [#2362](https://github.com/pingcap/tidb/pull/2362), [#2367](https://github.com/pingcap/tidb/pull/2367)

* Clean up tidb-server codes: [#2356](https://github.com/pingcap/tidb/pull/2356), [#2358](https://github.com/pingcap/tidb/pull/2358)

## New contributor

* [Zyguan](https://github.com/zyguan)

* [AndreMouche](https://github.com/AndreMouche)

# Weekly update in TiKV

Last week, We landed [19 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-12-26..2017-01-01&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Move `raw_get` to [thread pool](https://github.com/pingcap/tikv/pull/1434).

* Add `ResourceKind` for operator and don't resend duplicated `AddPeer` response when the peer is still pending, see [#449](https://github.com/pingcap/pd/pull/449).

* Add [member command](https://github.com/pingcap/pd/pull/455) for `pd-ctl`.

## Fixed

* Fix getting [valid float](https://github.com/pingcap/tikv/pull/1454).

## Improved

* Update default configuration to [speed up scheduler](https://github.com/pingcap/pd/pull/452).

* [Don't panic](https://github.com/pingcap/tikv/pull/1443) when receiving stale snapshot.

* Remove unnecessary [region cache](https://github.com/pingcap/pd/pull/457).

* Remove no used [constraint feature](https://github.com/pingcap/pd/pull/458).

* [Exit](https://github.com/pingcap/tikv/pull/1451) with error message if clusert ID mismatches directly.

* Add [replication section](https://github.com/pingcap/pd/pull/460) in configuration.

[原文链接](http://weekly.pingcap.com/2017/01/01/tidb-weekly/)

TiDB Weekly [2016.12.26]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1386 次浏览 • 2017-01-07 13:12 • 来自相关话题

# New Release [TiDB RC1](https://github.com/pingcap/tidb/releases/tag/rc1) is released! ## Weekly updat ...查看全部
# New Release

[TiDB RC1](https://github.com/pingcap/tidb/releases/tag/rc1) is released!

## Weekly update in TiDB

Last week, we landed [34 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-12-19..2016-12-25%20) in the TiDB repositories.

## Added

* [Support the `RPAD` built-in function.](https://github.com/pingcap/tidb/pull/2270).

* [Support the `show keys from table from database` statement.](https://github.com/pingcap/tidb/pull/2308)

## Fixed

* [Retry infinite times if the commit primary key times out.](https://github.com/pingcap/tidb/pull/2276)

* [Do not push aggregation down to the memory tables.](https://github.com/pingcap/tidb/pull/2296)

* [Fix a bug about the `alter table` statement](https://github.com/pingcap/tidb/pull/2297).

## Improved

* Refactor the time type related code: [#2259](https://github.com/pingcap/tidb/pull/2259), [#2280](https://github.com/pingcap/tidb/pull/2280), [#2284](https://github.com/pingcap/tidb/pull/2284), #2289, [#2292](https://github.com/pingcap/tidb/pull/2292)

* Refactor optimizer: [extract initialization related code into physical Initialization.](https://github.com/pingcap/tidb/pull/2263)

* [Speed up the DDL statement.](https://github.com/pingcap/tidb/pull/2268)

* [Avoid generating `parser.go` every time.](https://github.com/pingcap/tidb/pull/2281)

* [Skip the constraint check ](https://github.com/pingcap/tidb/pull/2288)for prewrite to improve the loading data speed.

* [Speed up the `add index` statement.](https://github.com/pingcap/tidb/pull/2309)

## New contributor

* [silentred](https://github.com/silentred)

# Weekly update in TiKV

Last week, We landed [14 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-12-18..2016-12-25&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

* Add configuration to control the [replica scheduling speed](https://github.com/pingcap/pd/pull/426).

* [Skip constraint check](https://github.com/pingcap/tikv/pull/1411) for prewrite to improve the loading data speed.

* Add [region and store commands](https://github.com/pingcap/pd/pull/439) to pd-ctl.

* Add [configuration](https://github.com/pingcap/tikv/pull/1422) to cache index and filter blocks in the block cache.

## Fixed

* [Report snapshot](https://github.com/pingcap/tikv/pull/1394) sending status reliably to fix [#1377](https://github.com/pingcap/tikv/issues/1377).

* Store [short value](https://github.com/pingcap/tikv/pull/1407) in write cf directly to save space and improve performance.

## Improved

* Remove unnecessary [admin operators](https://github.com/pingcap/pd/pull/427).

* Handle Raft ready [append log](https://github.com/pingcap/tikv/pull/1420)[ in one WriteBatch](https://github.com/pingcap/tikv/pull/1420) to reduce the CPU usage and improve performance.

* [Remove the down peer first](https://github.com/pingcap/pd/pull/446) when scheduling replicas.

[原文链接](http://weekly.pingcap.com/2016/12/26/tidb-weekly/)

TiDB Weekly [2016.12.19]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1541 次浏览 • 2016-12-25 13:46 • 来自相关话题

# Weekly update in TiDB Last week, we landed [32 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3A ...查看全部
# Weekly update in TiDB
Last week, we landed [32 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-12-12..2016-12-18) in the TiDB repositories.

## Added

+ [Add the `FlagIgnoreTruncate/FlagTruncateAsWarning` flag to control the behavior of truncated errors.](https://github.com/pingcap/tidb/pull/2212)

+ [Add the `prompt` text flag.](https://github.com/pingcap/tidb/pull/2227)

+ [Add the rawkv metrics](https://github.com/pingcap/tidb/pull/2228) to profile the rawkv API performance.

+ [Add a comparable varint encoding/decoding method](https://github.com/pingcap/tidb/pull/2236) to make encoded data smaller.

+ [Support the `timediff` built-in function.](https://github.com/pingcap/tidb/pull/2249)

+ [Support the following built-in functions](https://github.com/pingcap/tidb/pull/2258): ln(), log(), log2(), log10().

## Fixed

+ [A bug that ignores primary key’s unsigned attribute.](https://github.com/pingcap/tidb/pull/2222)

+ [A bug that ignores error in the `distsql` layer.](https://github.com/pingcap/tidb/pull/2226)

+ [Allow default value to be Null when the column has the `auto_increament` attribute](https://github.com/pingcap/tidb/pull/2230) to be compatible with MySQL 5.6.

+ [A bug in the `insert` statement that ignores error.](https://github.com/pingcap/tidb/pull/2241)

+ Fix bugs in the cost-based optimization framework: [#2243](https://github.com/pingcap/tidb/pull/2243)

## Improved

+ Refactor the time type related code: [#2185](https://github.com/pingcap/tidb/pull/2185), [#2190](https://github.com/pingcap/tidb/pull/2190), [#2206](https://github.com/pingcap/tidb/pull/2206), [#2233](https://github.com/pingcap/tidb/pull/2233), [#2261](https://github.com/pingcap/tidb/pull/2261)

+ [Remove the util/bytes package](https://github.com/pingcap/tidb/pull/2221) to clean up the code.

+ Refactor the code to remove the `evaluator.Eval()` method: [#2222](https://github.com/pingcap/tidb/pull/2222),

+ [Improve test coverage for the `util/segmentmap` package.](https://github.com/pingcap/tidb/pull/2235)

+ [Recover from panics caused by malformated mysql packet](https://github.com/pingcap/tidb/pull/2267) to make tidb-server more robust.

## New contributor

+ [Bai Yang](https://github.com/hamo)

# Weekly update in TiKV

Last week, we landed [11 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-12-11..2016-12-17&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

+ Add a configuration to [disable data sync](https://github.com/pingcap/tikv/pull/1369) to speed up loading data.

+ Filter [the pending peers](https://github.com/pingcap/pd/pull/421) for Placement Driver (PD) scheduler.

+ Add [`pd-ctl`](https://github.com/pingcap/pd/pull/431) to operate PD more easily.

## Fixed

+ Update [the advertise peer urls](https://github.com/pingcap/pd/pull/438) from etcd to fix [#435](https://github.com/pingcap/pd/issues/435).

## Improved

+ [Read and verify snapshot](https://github.com/pingcap/tikv/pull/1381) file in one step.

+ Use a [smaller interval](https://github.com/pingcap/tikv/pull/1401) to make Raft tick more accurate.

+ Clean up [the tombstone store](https://github.com/pingcap/pd/issues/428) to fix [#401](https://github.com/pingcap/pd/issues/401).

+ Use [`delete_file_in_range`](https://github.com/pingcap/tikv/pull/1409) when clean up the tombstone regions.

[原文链接](http://weekly.pingcap.com/2016/12/19/tidb-weekly/)

TiDB Weekly [2016.12.12]

Go开源项目qiuyesuifeng 发表了文章 • 0 个评论 • 1442 次浏览 • 2016-12-17 14:14 • 来自相关话题

# Weekly update in TiDB Last week, we landed [41 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr% ...查看全部
# Weekly update in TiDB

Last week, we landed [41 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-12-05..2016-12-11%20) in the TiDB repositories.

## Added

+ [Support the built-in function: `str_to_date`. ](https://github.com/pingcap/tidb/pull/2078)

+ [Support built-in function schema().](https://github.com/pingcap/tidb/pull/2173)

+ [Support pushing the `case-when` expression to TiKV.](https://github.com/pingcap/tidb/pull/2171)

+ [Support changing the type and name of a column.](https://github.com/pingcap/tidb/pull/2174)

+ [Add the `](https://github.com/pingcap/tidb/pull/2194)[session_variables` and `plugins` of memory table to infoschema.](https://github.com/pingcap/tidb/pull/2194)

+ [Make the `union all` operator run parallelly.](https://github.com/pingcap/tidb/pull/2195)

+ [Support explaining the `union` statement.](https://github.com/pingcap/tidb/pull/2216)

## Fixed

+ [A bug that causes infinite loop.](https://github.com/pingcap/tidb/pull/2163)

+ [A bug in the `on duplicate…` statement when updating the primary key (PK)](https://github.com/pingcap/tidb/pull/2179)

+ [Make the charset name case-insensitive.](https://github.com/pingcap/tidb/pull/2184)

+ [Forbid dropping columns with the `auto_inc` and `PK` attribute.](https://github.com/pingcap/tidb/pull/2203)

+ [Fix bugs in the parser.](https://github.com/pingcap/tidb/pull/2210)

## Improved

+ [Pass the filter to TiKV when scanning indexes.](https://github.com/pingcap/tidb/pull/2166)

+ [Allocate column/index IDs in the table space](https://github.com/pingcap/tidb/pull/2205) to make the IDs shorter.

# Weekly update in TiKV

Last week, we landed [34 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-12-04..2016-12-10&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

+ Support [online backup](https://github.com/pingcap/tikv/pull/1355) of the RocksDB data for `tikv-ctl` debugging.

+ Add [replication constraints](https://github.com/pingcap/pd/pull/402) to schedule replicas.

+ Support [`GrantLeaderScheduler`](https://github.com/pingcap/pd/pull/406) to transfer all leaders to one store.

+ Support [`ShuffleLeaderScheduler`](https://github.com/pingcap/pd/pull/409) to shuffle leaders in different stores.

+ Support Circle CI for [TiKV](https://github.com/pingcap/tikv/pull/1384) and [Placement Driver (PD)](https://github.com/pingcap/pd/pull/412).

+ Add the [state filter argument](https://github.com/pingcap/pd/pull/425) to get the stores API.

## Fixed

+ [Use `channel`](https://github.com/pingcap/tikv/pull/1375) to fix the possible stale snapshot state, issue [#1373](https://github.com/pingcap/tikv/issues/1373).

+ Report [`ServerIsBusy`](https://github.com/pingcap/tikv/pull/1390) and let [PD ignore the busy store](https://github.com/pingcap/pd/issues/420) when scheduling to fix [#414](https://github.com/pingcap/pd/issues/414).

## Improved

+ [Speed up the shutdown duration](https://github.com/pingcap/tikv/pull/1385) to reduce the close waiting time.

+ [Add retry](https://github.com/pingcap/pd/pull/417) when initializing the cluster ID.

+ Support [safe ConfChange](https://github.com/pingcap/tikv/pull/1398) to fix [#1366](https://github.com/pingcap/tikv/issues/1366).

+ [Report pending peers](https://github.com/pingcap/tikv/pull/1395) to PD to improve its scheduler.

+ [Bind ports lazily](https://github.com/pingcap/tikv/pull/1400) to avoid the message channel full error when starting up.

[原文链接](http://weekly.pingcap.com/2016/12/12/tidb-weekly/)

TiDB Weekly [2016.12.05]

文章分享qiuyesuifeng 发表了文章 • 0 个评论 • 1348 次浏览 • 2016-12-07 11:42 • 来自相关话题

# Weekly update in TiDB Last week, we landed [48 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr% ...查看全部
# Weekly update in TiDB

Last week, we landed [48 PRs](https://github.com/pingcap/tidb/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-11-28..2016-12-04) in the TiDB repositories, [6 PRs](https://github.com/pingcap/docs/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20merged%3A2016-11-28..2016-12-04%20) in the TiDB docs repositories.

## Added

+ [Support the built-in function: str\_to\_date. ](https://github.com/pingcap/tidb/pull/2078)

+ Refactor the time structure: [Introduce a `TimeInternal` interface to replace the go time representation.](https://github.com/pingcap/tidb/pull/2098)

+ [Add the raw Key-Value API ](https://github.com/pingcap/tidb/pull/2101) and make TiKV a raw Key-Value engine.

+ [Add a bench tool for the raw Key-Value ](https://github.com/pingcap/tidb/pull/2109)[API](https://github.com/pingcap/tidb/pull/2101).

+ [Support the PARTITION keyword](https://github.com/pingcap/tidb/pull/2115): parsed but ignored.

+ [Support the `Alter User` statement to change user’s password.](https://github.com/pingcap/tidb/pull/2144)

+ [Support the `set password = pwd;` statement.](https://github.com/pingcap/tidb/pull/2155)

+ [Use circleci.](https://github.com/pingcap/tidb/pull/2154)

## Fixed

+ [Check duplicate column names when adding index.](https://github.com/pingcap/tidb/pull/2103)

+ [Check duplicate index columns when creating table.](https://github.com/pingcap/tidb/pull/2123)

+ [Fix a bug when `join` exists in subquery.](https://github.com/pingcap/tidb/pull/2106)

+ [Make the `schema out of date` error retry automatically within TiDB.](https://github.com/pingcap/tidb/pull/2110)

+ [Fix a typo in the `INFORMATION_SCHEMA.COLUMNS` table.](https://github.com/pingcap/tidb/pull/2126)

+ [Fix the `date_format` type infer bug.](https://github.com/pingcap/tidb/pull/2152)

+ [Do not prune the `set variable` expression in projection.](https://github.com/pingcap/tidb/pull/2160)

## Improved

+ [Update the template for new issue.](https://github.com/pingcap/tidb/pull/2131)

+ [Use the statement context to handle truncated error.](https://github.com/pingcap/tidb/pull/2147)

+ [Push the aggregate operator down under union all.](https://github.com/pingcap/tidb/pull/2150)

## Document change

The following guides are updated:

* [TiDB Binary Deployment](https://github.com/pingcap/docs/blob/master/op-guide/binary-deployment.md)

* [TiDB Docker Deployment](https://github.com/pingcap/docs/blob/master/op-guide/docker-deployment.md)

* [Compatibility with MySQL](https://github.com/pingcap/docs/blob/master/op-guide/mysql-compatibility.md)

# Weekly update in TiKV

Last week, we landed [22 PRs](https://github.com/search?utf8=%E2%9C%93&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-11-27..2016-12-03&type=Issues&ref=searchresults) in the TiKV repositories.

## Added

+ Support [consistency check](https://github.com/pingcap/tikv/pull/1344) to find if data is corrupted or not dynamically.

+ Add configuration to control the [max running task count](https://github.com/pingcap/tikv/pull/1361).

+ Add the raw [Key-Value API](https://github.com/pingcap/tikv/pull/1364).

## Fixed

+ [Check the Placement Driver list](https://github.com/pingcap/tikv/pull/1201) to fix [#1186](https://github.com/pingcap/tikv/issues/1186).

+ Check whether the term is stale for the Raft command to fix [#1317](https://github.com/pingcap/tikv/issues/1317).

+ Schedule [the log Garbage Collection by size](https://github.com/pingcap/tikv/pull/1362) to fix [#1337](https://github.com/pingcap/tikv/issues/1337).

## Improved

+ Replace the score type with [resource kind](https://github.com/pingcap/pd/pull/393) to calculate the scores more easily.

+ Replace origin concept `balance` with [`schedule`](https://github.com/pingcap/tikv/pull/1344) and simplify configurations.

+ [Use coordinator](https://github.com/pingcap/pd/pull/398) to control the speed of different schedulers.

[原文链接](http://weekly.pingcap.com/2016/12/05/tidb-weekly/)

TiDB Weekly [2016.11.28]

文章分享qiuyesuifeng 发表了文章 • 0 个评论 • 1401 次浏览 • 2016-11-30 12:12 • 来自相关话题

# Weekly update in TiDB Last week, we landed [44 PRs](https://github.com/pingcap/tidb/pulls?utf8=✓&q=is%3Apr%20is%3Am ...查看全部
# Weekly update in TiDB

Last week, we landed [44 PRs](https://github.com/pingcap/tidb/pulls?utf8=✓&q=is%3Apr%20is%3Amerged%20merged%3A2016-11-21..2016-11-27%20) in the TiDB repositories, [3 PRs](https://github.com/pingcap/docs/pulls?utf8=✓&q=is%3Apr%20is%3Amerged%20merged%3A2016-11-21..2016-11-27%20) in the TiDB docs repositories.

## Added

+ [Support creating anonymous index.](https://github.com/pingcap/tidb/pull/2075)

+ [Add the mailing list for TiDB users.](https://github.com/pingcap/tidb/pull/2090)

+ [Support the `show events` syntax.](https://github.com/pingcap/tidb/pull/2099)

## Fixed

+ [Enlarge the ](https://github.com/pingcap/tidb/pull/2004)[Time To Live (TTL)](https://github.com/pingcap/tidb/pull/2004)[ for large transactions.](https://github.com/pingcap/tidb/pull/2004)

+ [Parse float literal using the decimal parser.](https://github.com/pingcap/tidb/pull/2044)

+ [Prevent panic for malformed packets.](https://github.com/pingcap/tidb/pull/2048)

+ [Fix the behavior in the aggregate operator:](https://github.com/pingcap/tidb/pull/2057) for the `select a, c from t groupby t.b` statement, `a` and `c` should use the first row in the group.

+[ Add sequence number in binlog](https://github.com/pingcap/tidb/pull/2060) to preserve the original mutation order.

+ [Reset the current database after dropping the current database.](https://github.com/pingcap/tidb/pull/2089)

## Improved

+ [Prevent loading schema by multiple threads.](https://github.com/pingcap/tidb/pull/1984)

+ [Make unit test run faster.](https://github.com/pingcap/tidb/pull/2010)

+ [Make explain result clearer.](https://github.com/pingcap/tidb/pull/2063)

+ [Remove `driver.go` from the TiDB project](https://github.com/pingcap/tidb/pull/2066) to enable users to use the MySQL official driver.

## Document change

Add the following new guides:

* [TiDB Cluster Troubleshooting Guide](https://github.com/pingcap/docs/blob/master/trouble-shooting.md)
* [TiKV Tuning Guide](https://github.com/pingcap/docs/blob/master/op-guide/tune-TiKV.md)

# Weekly update in TiKV

Last week, we landed [20 PRs](https://github.com/search?p=1&q=repo%3Apingcap%2Ftikv+repo%3Apingcap%2Fpd+is%3Apr+is%3Amerged+merged%3A2016-11-20..2016-11-26&ref=searchresults&type=Issues&utf8=%E2%9C%93) in the TiKV repositories.

## Added

+ Add and report the [store labels](https://github.com/pingcap/tikv/pull/1349) to Placement Driver (PD).

+ Add [pending task metrics](https://github.com/pingcap/tikv/pull/1354) for Worker.

+ Dump all the statistics about [Column Family compaction and Database](https://github.com/pingcap/tikv/pull/1357).

## Fixed

+ Use [`fs2`](https://github.com/pingcap/tikv/pull/1323) to get disk states to fix [#1318](https://github.com/pingcap/tikv/issues/1318)

+ Use the [monotonic clock time](https://github.com/pingcap/tikv/pull/1327) to improve the safety of leader lease read, to fix [#964](https://github.com/pingcap/tikv/issues/964).

+ Check the format of the [listening and advertise address](https://github.com/pingcap/tikv/pull/1335) to fix [#1332](https://github.com/pingcap/tikv/issues/1332).

+ [Get the first value](https://github.com/pingcap/tikv/pull/1339) no matter if it is Null in aggregation.

+ [Fix a Garbage Collection bug](https://github.com/pingcap/tikv/pull/1350) which deletes the latest deleted key before SafePoint.

+ [Stop attaching term to the MsgReadIndex message](https://github.com/pingcap/tikv/pull/1353) to fix [#1240](https://github.com/pingcap/tikv/issues/1240).

## Improved

+ Clean up the [command flags and configurations](https://github.com/pingcap/tikv/pull/1307) parsing.

+ [Abstract a Selector](https://github.com/pingcap/pd/pull/389) to schedule region peer.

+ Split the [`Raft Ready` handle](https://github.com/pingcap/pd/pull/1322) to two handles: `Append` and `Apply`.

+ Use [larger Heartbeat and Election timeout](https://github.com/pingcap/pd/pull/391) to reduce the network pressure.

+ [Ignore outdated tasks](https://github.com/pingcap/tikv/pull/1324) in Coprocessor to fix [#1305](https://github.com/pingcap/tikv/issues/1305).

[原文链接](http://weekly.pingcap.com/2016/11/28/tidb-weekly/)

Percolator 和 TiDB 事务算法

文章分享qiuyesuifeng 发表了文章 • 1 个评论 • 2442 次浏览 • 2016-11-24 11:24 • 来自相关话题

本文先概括的讲一下 Google Percolator 的大致流程。Percolator 是 Google 的上一代分布式事务解决方案,构建在 BigTable 之上,在 Google 内部 用于网页索引更新的业务,原始的论文[在此](http://resea ...查看全部
本文先概括的讲一下 Google Percolator 的大致流程。Percolator 是 Google 的上一代分布式事务解决方案,构建在 BigTable 之上,在 Google 内部 用于网页索引更新的业务,原始的论文[在此](http://research.google.com/pubs/pub36726.html)。原理比较简单,总体来说就是一个经过优化的二阶段提交的实现,进行了一个二级锁的优化。TiDB 的事务模型沿用了 Percolator 的事务模型。
总体的流程如下:

###读写事务

1) 事务提交前,在客户端 buffer 所有的 update/delete 操作。
2) Prewrite 阶段:

首先在所有行的写操作中选出一个作为 primary,其他的为 secondaries。

PrewritePrimary: 对 primaryRow 写入 L 列(上锁),L 列中记录本次事务的开始时间戳。写入 L 列前会检查:

1. 是否已经有别的客户端已经上锁 (Locking)。
2. 是否在本次事务开始时间之后,检查 W 列,是否有更新 [startTs, +Inf) 的写操作已经提交 (Conflict)。

在这两种种情况下会返回事务冲突。否则,就成功上锁。将行的内容写入 row 中,时间戳设置为 startTs。

将 primaryRow 的锁上好了以后,进行 secondaries 的 prewrite 流程:

1. 类似 primaryRow 的上锁流程,只不过锁的内容为事务开始时间及 primaryRow 的 Lock 的信息。
2. 检查的事项同 primaryRow 的一致。

当锁成功写入后,写入 row,时间戳设置为 startTs。

3) 以上 Prewrite 流程任何一步发生错误,都会进行回滚:删除 Lock,删除版本为 startTs 的数据。

4) 当 Prewrite 完成以后,进入 Commit 阶段,当前时间戳为 commitTs,且 commitTs> startTs :

1. commit primary:写入 W 列新数据,时间戳为 commitTs,内容为 startTs,表明数据的最新版本是 startTs 对应的数据。
2. 删除L列。

如果 primary row 提交失败的话,全事务回滚,回滚逻辑同 prewrite。如果 commit primary 成功,则可以异步的 commit secondaries, 流程和 commit primary 一致, 失败了也无所谓。

###事务中的读操作

1. 检查该行是否有 L 列,时间戳为 [0, startTs],如果有,表示目前有其他事务正占用此行,如果这个锁已经超时则尝试清除,否则等待超时或者其他事务主动解锁。注意此时不能直接返回老版本的数据,否则会发生幻读的问题。
2. 读取至 startTs 时该行最新的数据,方法是:读取 W 列,时间戳为 [0, startTs], 获取这一列的值,转化成时间戳 t, 然后读取此列于 t 版本的数据内容。

由于锁是分两级的,primary 和 seconary,只要 primary 的行锁去掉,就表示该事务已经成功 提交,这样的好处是 secondary 的 commit 是可以异步进行的,只是在异步提交进行的过程中 ,如果此时有读请求,可能会需要做一下锁的清理工作。

[原文链接](http://www.pingcap.com/blog-percolator-and-txn.html)

TiKV 的 MVCC(Multi-Version Concurrency Control)机制

文章分享qiuyesuifeng 发表了文章 • 0 个评论 • 1867 次浏览 • 2016-11-24 11:20 • 来自相关话题

### 并发控制简介 事务隔离在数据库系统中有着非常重要的作用,因为对于用户来说数据库必须提供这样一个“假象”:当前只有这么一个用户连接到了数据库中,这样可以减轻应用层的开发难度。但是,对于数据库系统来说,因为同一时间可能会存在很多用户连接,那么许 ...查看全部
### 并发控制简介
事务隔离在数据库系统中有着非常重要的作用,因为对于用户来说数据库必须提供这样一个“假象”:当前只有这么一个用户连接到了数据库中,这样可以减轻应用层的开发难度。但是,对于数据库系统来说,因为同一时间可能会存在很多用户连接,那么许多并发问题,比如数据竞争(data race),就必须解决。在这样的背景下,数据库管理系统(简称 DBMS)就必须保证并发操作产生的结果是安全的,通过可串行化(serializability)来保证。


虽然 Serilizability 是一个非常棒的概念,但是很难能够有效的实现。一个经典的方法就是使用一种[两段锁(2PL)][1]。通过 2PL,DBMS 可以维护读写锁来保证可能产生冲突的事务按照一个良好的次序(well-defined) 执行,这样就可以保证 Serializability。但是,这种通过锁的方式也有一些缺点:

1. 读锁和写锁会相互阻滞(block)。
2. 大部分事务都是只读(read-only)的,所以从事务序列(transaction-ordering)的角度来看是无害的。如果使用基于锁的隔离机制,而且如果有一段很长的读事务的话,在这段时间内这个对象就无法被改写,后面的事务就会被阻塞直到这个事务完成。这种机制对于并发性能来说影响很大。

*多版本并发控制(Multi-Version Concurrency Control, 以下简称 MVCC)*以一种优雅的方式来解决这个问题。在 MVCC 中,每当想要更改或者删除某个数据对象时,DBMS 不会在原地去删除或这修改这个已有的数据对象本身,而是创建一个该数据对象的新的版本,这样的话同时并发的读取操作仍旧可以读取老版本的数据,而写操作就可以同时进行。这个模式的好处在于,可以让读取操作不再阻塞,事实上根本就不需要锁。这是一种非常诱人的特型,以至于在很多主流的数据库中都采用了 MVCC 的实现,比如说 PostgreSQL,Oracle,Microsoft SQL Server 等。


### TiKV 中的 MVCC
让我们深入到 TiKV 中的 MVCC,了解 MVCC 在 TiKV 中是如何[实现](https://github.com/pingcap/tikv/tree/master/src/storage)的。


#### Timestamp Oracle(TSO)
因为`TiKV` 是一个分布式的储存系统,它需要一个全球性的授时服务,下文都称作 TSO(Timestamp Oracle),来分配一个单调递增的时间戳。 这样的功能在 TiKV 中是由 PD 提供的,在 Google 的 [Spanner](http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf) 中是由多个原子钟和 GPS 来提供的。


#### Storage
从源码结构上来看,想要深入理解 TiKV 中的 MVCC 部分,[src/storage](https://github.com/pingcap/tikv/blob/master/src/storage/mod.rs) 是一个非常好的入手点。 `Storage` 是实际上接受外部命令的结构体。
```rust
pub struct Storage {
engine: Box,
sendch: SendCh,
handle: Arc>,
}




impl Storage {
pub fn start(&mut self, config: &Config) -> Result<()> {
let mut handle = self.handle.lock().unwrap();
if handle.handle.is_some() {
return Err(box_err!("scheduler is already running"));
}




let engine = self.engine.clone();
let builder = thread::Builder::new().name(thd_name!("storage-scheduler"));
let mut el = handle.event_loop.take().unwrap();
let sched_concurrency = config.sched_concurrency;
let sched_worker_pool_size = config.sched_worker_pool_size;
let sched_too_busy_threshold = config.sched_too_busy_threshold;
let ch = self.sendch.clone();
let h = try!(builder.spawn(move || {
let mut sched = Scheduler::new(engine,
ch,
sched_concurrency,
sched_worker_pool_size,
sched_too_busy_threshold);
if let Err(e) = el.run(&mut sched) {
panic!("scheduler run err:{:?}", e);
}
info!("scheduler stopped");
}));
handle.handle = Some(h);




Ok(())
}
}
```
`start` 这个函数很好的解释了一个 storage 是怎么跑起来的。


#### Engine
首先是 [Engine](https://github.com/pingcap/tikv/blob/master/src/storage/engine/mod.rs#L44)。 `Engine` 是一个描述了在储存系统中接入的的实际上的数据库的接口,[raftkv](https://github.com/pingcap/tikv/blob/master/src/storage/engine/raftkv.rs#L91) 和 [Enginerocksdb](https://github.com/pingcap/tikv/blob/master/src/storage/engine/rocksdb.rs#L66) 分别实现了这个接口。


#### StorageHandle
`StorageHanle` 是处理从`sench` 接受到指令,通过 [mio](https://github.com/carllerche/mio) 来处理 IO。


接下来在`Storage`中实现了`async_get` 和`async_batch_get`等异步函数,这些函数中将对应的指令送到通道中,然后被调度器(scheduler)接收到并异步执行。


Ok,了解完`Storage` 结构体是如何实现的之后,我们终于可以接触到在`Scheduler` [被调用的 MVCC 层](https://github.com/pingcap/tikv/blob/master/src/storage/txn/scheduler.rs#L763)了。


当 storage 接收到从客户端来的指令后会将其传送到调度器中。然后调度器执行相应的过程或者调用相应的[异步函数](https://github.com/pingcap/tikv/blob/master/src/storage/txn/scheduler.rs#L643)。在调度器中有两种操作类型,读和写。读操作在 [MvccReader](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/reader.rs#L20) 中实现,这一部分很容易理解,暂且不表。写操作的部分是MVCC的核心。


#### MVCC
Ok,两段提交(2-Phase Commit,2PC)是在 MVCC 中实现的,整个 TiKV 事务模型的核心。在一段事务中,由两个阶段组成。


##### Prewrite
选择一个 row 作为 primary row, 余下的作为 secondary row。
对primary row [上锁](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L80). 在上锁之前,会检查[是否有其他同步的锁已经上到了这个 row 上](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L71) 或者是是否经有在 startTS 之后的提交操作。这两种情况都会导致冲突,一旦都冲突发生,就会[回滚(rollback)](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L115)。
对于 secondary row 重复以上操作。


##### Commit
[Rollback](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L115) 在`Prewrite` 过程中出现冲突的话就会被调用。


##### Garbage Collector
很容易发现,如果没有[垃圾收集器(Gabage Collector)](https://github.com/pingcap/tikv/blob/master/src/storage/mvcc/txn.rs#L143) 来移除无效的版本的话,数据库中就会存有越来越多的 MVCC 版本。但是我们又不能仅仅移除某个 safe point 之前的所有版本。因为对于某个 key 来说,有可能只存在一个版本,那么这个版本就必须被保存下来。在`TiKV`中,如果在 safe point 前存在`Put` 或者`Delete`,那么说明之后所有的 writes 都是可以被移除的,不然的话只有`Delete`,`Rollback`和`Lock` 会被删除。


## TiKV-Ctl for MVCC
在开发和 debug 的过程中,我们发现查询 MVCC 的版本信息是一件非常频繁并且重要的操作。因此我们开发了新的工具来查询 MVCC 信息。`TiKV` 将 Key-Value,Locks 和Writes 分别储存在`CF_DEFAULT`,`CF_LOCK`,`CF_WRITE`中。它们以这样的格式进行编码


| | default | lock | write |
| --- | --- | --- | --- |
| **key** | z{encoded_key}{start_ts(desc)} | z{encoded_key} | z{encoded_key}{commit_ts(desc)} |
| **value** | {value} | {flag}{primary_key}{start_ts(varint)} | {flag}{start_ts(varint)} |

Details can be found [here](https://github.com/pingcap/tikv/issues/1077).


因为所有的 MVCC 信息在 Rocksdb 中都是储存在 CF Key-Value 中,所以想要查询一个 Key 的版本信息,我们只需要将这些信息以不同的方式编码,随后在对应的 CF 中查询即可。CF Key-Values 的[表示形式](https://github.com/pingcap/tikv/blob/master/src/bin/tikv-ctl.rs#L210)。

[1]: https://en.wikipedia.org/wiki/Two-phase_locking

[原文链接](http://www.pingcap.com/blog-mvcc-in-tikv.html)