bug: config_etcd.lua:100: readdir(): grpc: received message larger than max (4236894 vs. 4194304)

[heytong]

Issue description

[error] 13264#13264: *108 [lua] config_etcd.lua:448: failed to fetch data from etcd: /home/admin/apisix-2.0/apisix/core/etcd.lua:84: attempt to index field 'header' (a nil value)
stack traceback:
/home/admin/apisix-2.0/apisix/core/etcd.lua:84: in function 'get_format'
/home/admin/apisix-2.0/apisix/core/config_etcd.lua:98: in function 'readdir'
/home/admin/apisix-2.0/apisix/core/config_etcd.lua:176: in function 'sync_data'
/home/admin/apisix-2.0/apisix/core/config_etcd.lua:424: in function </home/admin/apisix-2.0/apisix/core/config_etcd.lua:414>
[C]: in function 'xpcall'
/home/admin/apisix-2.0/apisix/core/config_etcd.lua:414: in function </home/admin/apisix-2.0/apisix/core/config_etcd.lua:405>, etcd key: /apisix/routes, context: ngx.timer

Environment

• apisix version (2.0 & 2.1):
• OS: any

Minimal test code / Steps to reproduce the issue

1、debug : config_etcd.lua:100: readdir(): -=-=-=-=-={"code":8,"error":"grpc: received message larger than max (4236894 vs. 4194304)","message":"grpc: received message larger than max (4236894 vs. 4194304)"}, context: ngx.timer

What's the actual result? (including assertion message & call stack if applicable)

1、The amount of data on the "/apisix/routes" is too large, We really need so much data.

What's the expected result?

[tzssangglass]

this is the limit of the response body for etcd, and the related parameter grpc.MaxCallRecvMsgSize.
ref --> clientv3 grpc.MaxCallRecvMsgSize defaulting to 4MB (clients can only receive <4MB response).

take a look at etcd-io/etcd#9043

[heytong]

About Iterate over the routes' data, Can it be handled like this：

$keys = etcdctl get --keys-only --from-key /apisix/routes
foeach $i in $keys:
$val = etcdctl get --from-key $i

[tzssangglass]

About Iterate over the routes' data, Can it be handled like this：

$keys = etcdctl get --keys-only --from-key /apisix/routes
foeach $i in $keys:
$val = etcdctl get --from-key $i

this error is generated by etcd, first you should change the limits of etcd, but I don't know how to change it by lua.

[spacewander]

It seems to me there is no way to configure it via HTTP. I have submitted an issue: etcd-io/etcd#12576

[spacewander]

Fixed by etcd-io/etcd#13077 which is included in the https://github.com/etcd-io/etcd/releases/tag/v3.5.0