[voq][chassis][dnx] syncd crash seen on multi-asic line cards when polling for stats on fabric ports

[sanmalho-git]

Description

Issue with PR# 2522 in swss submodule - [voq][chassis]Add show fabric counters port/queue commands.
This PR enabled fabric port stats polling in orchagent. However, this is not supported in SAI 7.1.28.4 (libsai in 202205) and causing syncd to crash on the linecards.

Backtrace of the crash is below:

(gdb) bt
#0  0x00007f52e68c4ce1 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f52e68ae537 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f52ec19ffe6 in _brcm_sai_switch_assert () from /usr/lib/libsai.so.1
#3  0x00007f52ec275a50 in _brcm_sai_indexed_data_get () from /usr/lib/libsai.so.1
#4  0x00007f52ec333b21 in _brcm_sai_dnx_irpp_port_core_get () from /usr/lib/libsai.so.1
#5  0x00007f52ec3340ef in _brcm_sai_dnx_port_discard_stat_get () from /usr/lib/libsai.so.1
#6  0x00007f52ec169a0e in ?? () from /usr/lib/libsai.so.1
#7  0x000055fc75cf190f in syncd::VendorSai::getStats (this=0x55fc775e2060, object_type=SAI_OBJECT_TYPE_PORT, object_id=2203318223104, number_of_counters=1, counter_ids=0x7ffc94c756b4, counters=0x7ffc94c756b8)
    at VendorSai.cpp:435
#8  0x000055fc75d5816f in syncd::FlexCounter::getSupportedPortCounters (this=0x55fc93b43d10, portRid=2203318223104, supportedPortCounters=std::set with 2 elements = {...}) at FlexCounter.cpp:2489
#9  0x000055fc75d5832b in syncd::FlexCounter::updateSupportedPortCounters (this=0x55fc93b43d10, portRid=2203318223104, supportedPortCounters=std::set with 2 elements = {...}) at FlexCounter.cpp:2516
#10 0x000055fc75d65324 in syncd::FlexCounter::setPortCounterList (this=0x55fc93b43d10, portVid=<optimized out>, portId=2203318223104, counterIds=std::vector of length 8, capacity 8 = {...}) at FlexCounter.cpp:243
#11 0x000055fc75d66589 in syncd::FlexCounter::addCounter (this=0x55fc93b43d10, vid=vid@entry=281474976710780, rid=rid@entry=2203318223104, values=std::vector of length 1, capacity 1 = {...}) at FlexCounter.cpp:3206
#12 0x000055fc75d149b7 in syncd::FlexCounterManager::addCounter (this=0x55fc77603230, vid=281474976710780, rid=2203318223104, instanceId="FABRIC_PORT_STAT_COUNTER", values=std::vector of length 1, capacity 1 = {...})
    at /usr/include/c++/10/bits/shared_ptr_base.h:1324
#13 0x000055fc75cd81a8 in syncd::Syncd::processFlexCounterEvent (this=0x55fc775f8320, consumer=...) at /usr/include/c++/10/bits/shared_ptr_base.h:1324
#14 0x000055fc75ce331b in syncd::Syncd::run (this=this@entry=0x55fc775f8320) at Syncd.cpp:4620
#15 0x000055fc75cc98af in syncd_main (argc=argc@entry=12, argv=argv@entry=0x7ffc94c76018) at syncd_main.cpp:71
#16 0x000055fc75cc7e9f in main (argc=12, argv=0x7ffc94c76018) at main.cpp:9

This will require help from BRCM to support stats on the fabric ports.

BRCM CSP CS00012277618 has been raised to address this issue.

Steps to reproduce the issue:

1. Load the latest sonic-broadcom-dnx.bin on a voq chassis with multi-asic J2C+ based linecards.

Describe the results you received:

syncd cores will be seen in /var/core upon reboot.

Describe the results you expected:

There should be no syncd cores seen upon boot up.

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[arlakshm]

@sanmalho-git, updated the issue link to sonic-net/sonic-swss#2522 in the description.

[arlakshm]

@kenneth-arista @ysmanman for viz.

[rlhui]

brcm csp open