Make Raft Engine production ready for TiKV

[tabokie]

Development Task

As of now Raft Engine is already wired with TiKV, and can be enabled by TiKV's raft-engine configuration group. But we haven't announce this feature publicly, and do not encourage using it for production.

This task aims to change this situation and make raft engine ready for production. We will be tracking the progress here and the detailed design in this internal doc.

Staff: @tabokie @MrCroxx

Time estimation: currently shooting for late 2021

Task breakdown:

Interface

• API for storage access
  • Basic file system interface for creating IO wrapper: Introduce file system and implement streaming encoding #91
  • Strongly typed file system interface: replace file system with file builder #96
• Customized storage layer for TiKV: engine: provide customized file builder to raft engine tikv#10937
  • Adapt encryption for Raft Engine
    • encryption: support encryption on top of custom reader and writer tikv#10758
    • encryption: support seek for encryption writer tikv#10929
• API for customized thread pool (optional)

Observability

• Prometheus metering: add basic metrics #76
• Integration with TiKV Grafana: *: add metrics for Raft engine tikv#10637

Performance

• Fast recovery: 6x faster
  • New log format for recovering indexes: Optimize log file format for faster recovery #83
  • Parallel recovery: Implement parallel recovery #97
• Fast log batch encoding: 3x faster
  • Streaming encoding: Introduce file system and implement streaming encoding #91
  • Reduce allocation: reduce allocation of log batch compression #98
• Optimize file operation: 2.5x throughput, Add some micro optimizations #100
• Benchmark and best practice for usage within TiKV
  • Known issue: IOPS explosion, Fix: Implement write barrier for grouping multiple writers #116

Safety

• Improve handling of IO error: rollback to a consistent state after fsync error #131
• Unit test

Tools

• Standalone performance benchmark: Add stress test #84
• Data administration tool: Add administration tool for recovery from data corruption #99

Integration test

• Benchmark with TPC-C, sysbench
• High pressure workload with random failures

Misc

• Remove entry cache: Remove entry cache #77
• Update README
• Compile with stable Rust (optional)
• Build on Windows (optional)

[tabokie]

Scheduled to be released in v5.4.0 as an experimental feature. Closing this.