[201911] S5232f - Z9264f T0 Syncd crashes on reboot

[santhosh-kt]

Description
Both in S5232f and Z9264f - Syncd got crashed on reload after loading T0 topology in 201911 branch

Steps to reproduce the issue:

1. Load the S5232 or Z9264 with the latest 201911 in T0 topology
2. Reboot the node
3. We can see syncd crashes, once the switch comes up.

Describe the results you received:

Looks like SAI is crashing.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/syncd --diag -u -p /etc/sai.d/sai.profile'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f653e525fff in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f6548364240 (LWP 28))]
(gdb)
(gdb) bt
#0  0x00007f653e525fff in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f653e52742a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f653e563c00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f653e569fc6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f653e56a80e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f6541806d2d in CHECK_FREE_CLEAR_SIZE () from /usr/lib/libsai.so.1
#6  0x00007f65418271a9 in _brcm_sai_indexed_data_free2 ()
   from /usr/lib/libsai.so.1
#7  0x00007f65417e8d16 in _brcm_sai_free_hostif () from /usr/lib/libsai.so.1
#8  0x00007f654185194e in _brcm_sai_dm_fini () from /usr/lib/libsai.so.1
#9  0x00007f65417c53a3 in ?? () from /usr/lib/libsai.so.1
#10 0x000055ac4c69259a in ?? ()
#11 0x000055ac4c67f51e in ?? ()
#12 0x00007f653e5132e1 in __libc_start_main ()
   from /lib/x86_64-linux-gnu/libc.so.6
#13 0x000055ac4c6804aa in ?? ()
(gdb)

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

root@sonic:~# show version

SONiC Software Version: SONiC.HEAD.59-e12d2e8b
Distribution: Debian 9.12
Kernel: 4.9.0-11-2-amd64
Build commit: e12d2e8b
Build date: Wed Apr 29 09:52:06 UTC 2020
Built by: johnar@jenkins-worker-7

Platform: x86_64-dellemc_s5232f_c3538-r0
HwSKU: DellEMC-S5232f-C32
ASIC: broadcom
Serial Number: CN01WJVTCES0094A0012
Uptime: 22:52:47 up 7 min,  1 user,  load average: 3.55, 1.53, 0.67

Docker images:
REPOSITORY                    TAG                 IMAGE ID            SIZE
docker-syncd-brcm             HEAD.59-e12d2e8b    a180efcab00b        430MB
docker-syncd-brcm             latest              a180efcab00b        430MB
docker-platform-monitor       HEAD.59-e12d2e8b    386aa55a8508        335MB
docker-platform-monitor       latest              386aa55a8508        335MB
docker-router-advertiser      HEAD.59-e12d2e8b    9f4854585ee3        283MB
docker-router-advertiser      latest              9f4854585ee3        283MB
docker-fpm-frr                HEAD.59-e12d2e8b    3670c851b173        328MB
docker-fpm-frr                latest              3670c851b173        328MB
docker-sflow                  HEAD.59-e12d2e8b    023ab4968814        308MB
docker-sflow                  latest              023ab4968814        308MB
docker-lldp-sv2               HEAD.59-e12d2e8b    042f7f751f7a        305MB
docker-lldp-sv2               latest              042f7f751f7a        305MB
docker-orchagent              HEAD.59-e12d2e8b    87512310c295        326MB
docker-orchagent              latest              87512310c295        326MB
docker-dhcp-relay             HEAD.59-e12d2e8b    40456ac237c4        293MB
docker-dhcp-relay             latest              40456ac237c4        293MB
docker-database               HEAD.59-e12d2e8b    3aac1ffbac82        283MB
docker-database               latest              3aac1ffbac82        283MB
docker-snmp-sv2               HEAD.59-e12d2e8b    ca7db59dc9a2        340MB
docker-snmp-sv2               latest              ca7db59dc9a2        340MB
docker-teamd                  HEAD.59-e12d2e8b    6bf05ac5a892        307MB
docker-teamd                  latest              6bf05ac5a892        307MB
docker-nat                    HEAD.59-e12d2e8b    57fe398c336f        309MB
docker-nat                    latest              57fe398c336f        309MB
docker-sonic-mgmt-framework   HEAD.59-e12d2e8b    270430fb9afe        422MB
docker-sonic-mgmt-framework   latest              270430fb9afe        422MB
docker-sonic-telemetry        HEAD.59-e12d2e8b    fd97b5bcd946        345MB
docker-sonic-telemetry        latest              fd97b5bcd946        345MB

root@sonic:~# 

**Attach debug file `sudo generate_dump`:**

```

Attached logs for S5232f
Issues log.txt
syncd_log.txt

```

[santhosh-kt]

Attached show tech support dump in my dedicated repository since the file size is more than 25MB which Github doesn't allow to upload.

tech support dump

[xinliu-seattle]

@santhosh-kt , please test with SAI 3.7.4.2.

[santhosh-kt]

This issue is not seen in latest 201911 branch(#88) with updated SAI (version 3.7.5.1).
Hence, closing this issue.
Thank you team in resolving this.