Handle ASIC/SDK health event

[stephenxs]

ASIC/SDK health event

A way for syncd to notify orchagent an ASIC/SDK health event before asking orchagent to shutdown is introduced in this document.

For most of ethernet switches, the switch ASIC is the core component in the system. It is very important to identify a switch ASIC is in a failure state and report such event to NOS.

Currently, such failure is detected by SDK/FW on most of platforms. A vendor SAI notifies orchagent to shutdown using switch_shutdown_request notification when it detects an ASIC/SDK internal error. Usually, the vendor SAI prints log message before calling shutdown API.

Orchagent can abort itself if a SAI API call fails, usually due to a bad arguments, and can not be recovered. From a customer's perspective of view, this can be distinguished from the ASIC/SDK health event only by analyzing the log message.

The current implementation has the following limitations:

• It is difficult for a customer to understand what occured on SAI and below or distinguish an SDK/FW internal error from a SAI API call. Even a customer can analyze the issue using the log message, it is not intuitive.
• It is unable notify an ASIC/FW/SDK event if the event is less serious to ask for shutdown.
• It is unable for telementry agent to collect such information.

In this design, we will introduce a new way to address the limitations.

Implementation PRs

Module	PR title	state
sonic-buildimage	Support ASIC/SDK health event
sonic-swss-common	Support ASIC/SDK health event
sonic-sairedis	Support ASIC/SDK health event
sonic-swss	Support ASIC/SDK health event
sonic-utilities	Support ASIC/SDK health event

Signed-off-by: Stephen Sun stephens@nvidia.com

[zhangyanzhao]

Community review recording https://zoom.us/rec/share/bkb8Ed5drPXoyM1Yl_7-B6obBYYqlyyoY-AEF4BnErHCGrIDCSpJLhp-Bwbs6AJE.N_n2CjI_Pvr7Y5_o

[zhangyanzhao]

If anyone want to be reviewer of this feature, plesae leave your comments here. Thanks.

[stephenxs]

Some comments from the HLD review meeting

1. A way to limit the number of the ASIC/SDK health event
2. A police generate dump by SONiC in case SAI doesn't collect dump
3. Notify SAI the folder to store the collected dump
4. Consider whether to leverage the event/alarm system
5. A way to disable the events for platforms who expose the capability bits but don't support collect dumps on receiving an event.

[stephenxs]

Some comments from the HLD review meeting

1. A way to limit the number of the ASIC/SDK health event

Provided a way to limit the number of events in the database but won't limit the rate at which the vendor SAI generates the events (which is vendor SAI's responsibility)

2. A police generate dump by SONiC in case SAI doesn't collect dump
3. Notify SAI the folder to store the collected dump

Won't address it for now.

4. Consider whether to leverage the event/alarm system

We already did but still need to keep the ASIC_SDK_HEALTH_EVENT table in STATE_DB

5. A way to disable the events for platforms who expose the capability bits but don't support collect dumps on receiving an event.

Vendor SAI should expose the capability only when it is supported completely.

[zhangyanzhao]

Can you please help to add the code PRs by referring to #806 ? @stephenxs

[prsunny]

@venkatmahalingam /@prvattem , please signoff

[stephenxs]

Can you please help to add the code PRs by referring to #806 ? @stephenxs

Done.

[liat-grozovik]

@venkatmahalingam /@prvattem kindly reminder. if not further comments i will just go and merge the HLD.