[202012] Fast reboot dataplane downtime degraded

[vaibhavhd]

Description

Steps to reproduce the issue:

1. Run test_fast_reboot with IO.
2. The test would fail with dataplane downtime >30s.
3. This issue is seen on most of the HWSKUs.

Describe the results you received:

Fast reboot test failed with:

RunAnsibleModuleFail: run module shell failed, Ansible Results => {     "changed": true,      "cmd": "ptf --test-dir ptftests advanced-reboot.ReloadTest --platform-dir ptftests --qlen=1000 --platform remote -t 'nexthop_ips=None;dut_password=u'"'"'password'"'"';portchannel_ports_file='"'"'/tmp/portchannel_interfaces.json'"'"';lo_v6_prefix='"'"'fc00:1::/64'"'"';preboot_files=None;preboot_oper=None;reboot_type='"'"'fast-reboot'"'"';bgp_v4_v6_time_diff=40;vlan_ip_range='"'"'{"Vlan1000": "192.168.0.0/21"}'"'"';sniff_time_incr=60;ports_file='"'"'/tmp/ports.json'"'"';allow_vlan_flooding=False;setup_fdb_before_test=True;arista_vms=[u'"'"'172.16.134.127'"'"', u'"'"'172.16.134.126'"'"', u'"'"'172.16.134.125'"'"', u'"'"'172.16.134.124'"'"'];reboot_limit_in_seconds=30;dut_username=u'"'"'admin'"'"';vnet_pkts='"'"''"'"';dut_mac=u'"'"'d4:af:f7:1e:a7:f4'"'"';inboot_oper=None;dut_hostname=u'"'"'10.3.146.137'"'"';vnet=False;vlan_ports_file='"'"'/tmp/vlan_interfaces.json'"'"';default_ip_range='"'"'192.168.64.0/18'"'"'' --relax --debug info --log-file /tmp/advanced-reboot.ReloadTest.log",      "delta": "0:05:21.312002",      "end": "2021-08-13 19:31:38.899603",      "failed": true,      "invocation": {         "module_args": {             "_raw_params": "ptf --test-dir ptftests advanced-reboot.ReloadTest --platform-dir ptftests --qlen=1000 --platform remote -t 'nexthop_ips=None;dut_password=u'"'"'password'"'"';portchannel_ports_file='"'"'/tmp/portchannel_interfaces.json'"'"';lo_v6_prefix='"'"'fc00:1::/64'"'"';preboot_files=None;preboot_oper=None;reboot_type='"'"'fast-reboot'"'"';bgp_v4_v6_time_diff=40;vlan_ip_range='"'"'{"Vlan1000": "192.168.0.0/21"}'"'"';sniff_time_incr=60;ports_file='"'"'/tmp/ports.json'"'"';allow_vlan_flooding=False;setup_fdb_before_test=True;arista_vms=[u'"'"'172.16.134.127'"'"', u'"'"'172.16.134.126'"'"', u'"'"'172.16.134.125'"'"', u'"'"'172.16.134.124'"'"'];reboot_limit_in_seconds=30;dut_username=u'"'"'admin'"'"';vnet_pkts='"'"''"'"';dut_mac=u'"'"'d4:af:f7:1e:a7:f4'"'"';inboot_oper=None;dut_hostname=u'"'"'10.3.146.137'"'"';vnet=False;vlan_ports_file='"'"'/tmp/vlan_interfaces.json'"'"';default_ip_range='"'"'192.168.64.0/18'"'"'' --relax --debug info --log-file /tmp/advanced-reboot.ReloadTest.log",              "_uses_shell": true,              "argv": null,              "chdir": "/root",              "creates": null,              "executable": null,              "removes": null,              "stdin": null,              "stdin_add_newline": true,              "strip_empty_ends": true,              "warn": true         }     },      "msg": "non-zero return code",      "rc": 1,      "start": "2021-08-13 19:26:17.587601",      "stderr": "WARNING: No route found for IPv6 destination :: (no default route?)\n/usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.\n  from cryptography.hazmat.backends import default_backend\nadvanced-reboot.ReloadTest ... FAIL\n\n======================================================================\nFAIL: advanced-reboot.ReloadTest\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File "ptftests/advanced-reboot.py", line 1211, in runTest\n    self.handle_post_reboot_test_reports()\n  File "ptftests/advanced-reboot.py", line 1161, in handle_post_reboot_test_reports\n    self.assertTrue(is_good, errors)\nAssertionError: \n\nSomething went wrong. Please check output below:\n\nFAILED:dut:Longest downtime period must be less then 30 seconds. It was 0:00:48.268142\n\n\n----------------------------------------------------------------------\nRan 1 test in 318.701s\n\nFAILED (failures=1)",      "stderr_lines": [         "WARN

Describe the results you expected:

Fast reboot downtime should be less than 30s.

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[vaibhavhd]

sonic-net/sonic-utilities#1774 fixes this issue.