Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[warmboot] swss crash due to translateVidToRid error for SAI_OBJECT_TYPE_BUFFER_POOL #8893

Closed
vaibhavhd opened this issue Oct 2, 2021 · 2 comments · Fixed by sonic-net/sonic-swss#1945
Closed

[warmboot] swss crash due to translateVidToRid error for SAI_OBJECT_TYPE_BUFFER_POOL #8893

vaibhavhd opened this issue Oct 2, 2021 · 2 comments · Fixed by sonic-net/sonic-swss#1945
Labels
Master Branch Quality warmboot 👢

Comments

@vaibhavhd
Copy link
Contributor

Description

Steps to reproduce the issue:

  1. Run test_warm_reboot
  2. The test will fail with one OA core.
  3. Check logs to see that it syncd initiated issue - translateVidToRid error on SAI_OBJECT_TYPE_BUFFER_POOL obj.

Describe the results you received:

Oct  1 13:18:13.900718 str2-7050cx3-acs-01 ERR syncd#syncd: :- translateVidToRid: unable to get RID for VID oid:0x18000000000b35
Oct  1 13:18:13.900894 str2-7050cx3-acs-01 ERR syncd#syncd: :- run: Runtime error: :- translateVidToRid: unable to get RID for VID oid:0x18000000000b35
Oct  1 13:18:13.900918 str2-7050cx3-acs-01 NOTICE syncd#syncd: :- sendShutdownRequest: sending switch_shutdown_request notification to OA for switch: oid:0x21000000000000
Oct  1 13:18:13.901086 str2-7050cx3-acs-01 NOTICE syncd#syncd: :- sendShutdownRequestAfterException: notification send successfully

Sai redis log:

2021-10-01.13:18:08.850368|c|SAI_OBJECT_TYPE_BUFFER_POOL:oid:0x18000000000b35|SAI_BUFFER_POOL_ATTR_THRESHOLD_MODE=SAI_BUFFER_POOL_THRESHOLD_MODE_STATIC|SAI_BUFFER_POOL_ATTR_SIZE=32599040|SAI_BUFFER_POOL_ATTR_TYPE=SAI_BUFFER_POOL_TYPE_EGRESS
2021-10-01.13:18:08.851519|c|SAI_OBJECT_TYPE_BUFFER_POOL:oid:0x18000000000b36|SAI_BUFFER_POOL_ATTR_THRESHOLD_MODE=SAI_BUFFER_POOL_THRESHOLD_MODE_DYNAMIC|SAI_BUFFER_POOL_ATTR_SIZE=24709632|SAI_BUFFER_POOL_ATTR_TYPE=SAI_BUFFER_POOL_TYPE_EGRESS
2021-10-01.13:18:08.852575|c|SAI_OBJECT_TYPE_BUFFER_POOL:oid:0x18000000000b37|SAI_BUFFER_POOL_ATTR_THRESHOLD_MODE=SAI_BUFFER_POOL_THRESHOLD_MODE_DYNAMIC|SAI_BUFFER_POOL_ATTR_SIZE=32732160|SAI_BUFFER_POOL_ATTR_TYPE=SAI_BUFFER_POOL_TYPE_INGRESS|SAI_BUFFER_POOL_ATTR_XOFF_SIZE=1622016

Describe the results you expected:

Output of show version:

            "", 
            "SONiC Software Version: SONiC.master.40145-144b9f158", 
            "Distribution: Debian 10.10", 
            "Kernel: 4.19.0-12-2-amd64", 
            "Build commit: 144b9f158", 
            "Build date: Tue Sep 28 14:17:29 UTC 2021", 
            "Built by: AzDevOps@sonic-build-workers-000QNG", 
            "",

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

Syslog and sairedis logs caputed at (look after warmboot test starts):

test_advanced_reboot (14).log
@judyjoseph
Copy link
Contributor

This happens in 202106 PR builds as well https://dev.azure.com/mssonic/build/_build/results?buildId=41603&view=logs&jobId=88ce9a53-729c-5fa9-7b6e-3d98f2488e3f&j=551384a3-3c3d-5d98-9a93-ab579ce10658

@kcudnik
Copy link
Contributor

kcudnik commented Oct 5, 2021

this should be addressing this issue sonic-net/sonic-sairedis#930

@vaibhavhd vaibhavhd mentioned this issue Oct 6, 2021
Merged
vaibhavhd added a commit to sonic-net/sonic-swss that referenced this issue Oct 6, 2021
@vaibhavhd
Revert "[buffer orch] Bugfix: Don't query counter SAI_BUFFER_POOL_STA…
94d2d44 
…T_XOFF_ROOM_WATERMARK_BYTES on a pool where it is not supported (#1857)" (#1945)

This reverts commit 3d6b1f0.

Fix sonic-net/sonic-buildimage#8893

What I did
This commit had earlier caused issue on master image warmboot - sonic-net/sonic-buildimage#8722

To fix this issue, this PR was created to retreat sonic-swss head on buildimage - sonic-net/sonic-buildimage#8732

Now, this commit was again pulled into sonic-buildimage as part of sonic-swss submodule advance:
sonic-net/sonic-buildimage#8839

And, warm-reboot again broke for the same reason.

This change is so that any other swss submodule update on buildimage will not fail warmboot again.
judyjoseph pushed a commit to sonic-net/sonic-swss that referenced this issue Oct 7, 2021
@vaibhavhd @judyjoseph
Revert "[buffer orch] Bugfix: Don't query counter SAI_BUFFER_POOL_STA…
6e895ad 
…T_XOFF_ROOM_WATERMARK_BYTES on a pool where it is not supported (#1857)" (#1945)

This reverts commit 3d6b1f0.

Fix sonic-net/sonic-buildimage#8893

What I did
This commit had earlier caused issue on master image warmboot - sonic-net/sonic-buildimage#8722

To fix this issue, this PR was created to retreat sonic-swss head on buildimage - sonic-net/sonic-buildimage#8732

Now, this commit was again pulled into sonic-buildimage as part of sonic-swss submodule advance:
sonic-net/sonic-buildimage#8839

And, warm-reboot again broke for the same reason.

This change is so that any other swss submodule update on buildimage will not fail warmboot again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Master Branch Quality warmboot 👢
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert "[buffer orch] Bugfix: Don't query counter SAI_BUFFER_POOL_STA… vaibhavhd/sonic-swss
3 participants
@kcudnik @vaibhavhd @judyjoseph