3.4 to 3.5 upgrade may panic

[gyuho]

Originally reported by @chaochn47 https://github.com/etcd-io/etcd/pull/11427/files#r633271675

We've added a new internal raft request field here #11427, but we did not have the test coverage where cluster version is already 3.5 but minority are still in 3.4. As a result, monitoring and persisting cluster versions using v3 crashes 3.4 nodes.

Looks like these two fields made v3.5 to v3.4 downgrade fail.

Can we make it an experimental API in v3.4 and add these two fields to make downgrade from v3.5 to v3.4 possible?

panic: not implemented

goroutine 214 [running]:
go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply(0xc000238468, 0xc0002b2460, 0x0)
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/etcdserver/apply.go:173 +0xf85
go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply(0xc000242000, 0xc0002b2460, 0x0)
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/etcdserver/apply_auth.go:60 +0xd2
go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal(0xc000290600, 0xc0002d14d8)
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/etcdserver/server.go:2230 +0x1fb
go.etcd.io/etcd/etcdserver.(*EtcdServer).apply(0xc000290600, 0xc000302a80, 0xa, 0xc, 0xc0001c8000, 0x0, 0xc000046000, 0xc00033f640)
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/etcdserver/server.go:2144 +0x579
go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries(0xc000290600, 0xc0001c8000, 0xc00002c000)
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/etcdserver/server.go:1396 +0xe5
go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll(0xc000290600, 0xc0001c8000, 0xc00002c000)
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/etcdserver/server.go:1120 +0x88
go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8(0x11b6e30, 0xc0005da040)
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/etcdserver/server.go:1065 +0x3c
go.etcd.io/etcd/pkg/schedule.(*fifo).run(0xc0003c0180)
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/pkg/schedule/schedule.go:157 +0xf3
created by go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler
        /home/chaochn/workplace/EKS-etcd/src/EKS-etcd/pkg/schedule/schedule.go:70 +0x13b

[gyuho]

@ptabor @hexfusion This is a bug, which I think we can fix and backport?

@chaochn47 will be working on making version monitoring v3 configurable and false by default, to give enough cycle to migrate to latest raft message proto buf.

[hexfusion]

@gyuho do you want to hold beta 0?

[gyuho]

@hexfusion Sure, how about wait for @chaochn47's fix? Should be ready by today.