Cold-boot: teammgrd PORTCHANNEL configuration flushed by swss docker start script swss.sh

[jipanyang]

Description
Upon cold boot, teamd service starts immediately after updategraph.service, teammgrd reads the portchannel configuration and push configurations like min_links, fallback, mtu down to appDB.

SWSS server starts after updategraph.service and interfaces-config.service, upon cold start, it will flushes appDB. All the configuration done teammgrd previously will be lost.

teammgrd started at Feb 26 05:11:52.768030

Feb 26 05:11:52.768030 vlab-01 NOTICE teamd#teammgrd: :- main:25: --- Starting teammrgd ---
 866 Feb 26 05:11:52.768030 vlab-01 NOTICE teamd#teammgrd: :- loadRedisScript:28: lua script loaded, sha: 88270a7c5c90583e56425aca8af8a4b8c39fe757
 867 Feb 26 05:11:52.868304 vlab-01 INFO teamd#supervisord: teammgrd Using team device "PortChannel0001".
 868 Feb 26 05:11:52.868304 vlab-01 INFO teamd#supervisord: teammgrd
 869 Feb 26 05:11:52.868304 vlab-01 INFO teamd#supervisord: teammgrd Using PID file "/var/run/teamd/PortChannel0001.pid"

teamd.log:Feb 26 05:11:52.939019 vlab-01 NOTICE teamd#teammgrd: :- setLagAdminStatus:368: Set port channel PortChannel0001 admin status to up
teamd.log:Feb 26 05:11:52.939019 vlab-01 NOTICE teamd#teammgrd: :- setLagMtu:418: Set port channel PortChannel0001 MTU to 9100
teamd.log:Feb 26 05:11:53.011069 vlab-01 NOTICE teamd#teammgrd: :- setLagAdminStatus:368: Set port channel PortChannel0002 admin status to up
teamd.log:Feb 26 05:11:53.038120 vlab-01 NOTICE teamd#teammgrd: :- setLagMtu:418: Set port channel PortChannel0002 MTU to 9100
teamd.log:Feb 26 05:11:53.089232 vlab-01 NOTICE teamd#teammgrd: :- setLagAdminStatus:368: Set port channel PortChannel0003 admin status to up
teamd.log:Feb 26 05:11:53.102062 vlab-01 NOTICE teamd#teammgrd: :- setLagMtu:418: Set port channel PortChannel0003 MTU to 9100
teamd.log:Feb 26 05:11:53.194389 vlab-01 NOTICE teamd#teammgrd: :- setLagAdminStatus:368: Set port channel PortChannel0004 admin status to up
teamd.log:Feb 26 05:11:53.213747 vlab-01 NOTICE teamd#teammgrd: :- setLagMtu:418: Set port channel PortChannel0004 MTU to 9100

swss.sh called at Feb 26 05:12:06.759357

Feb 26 05:12:06.759357 vlab-01 INFO swss.sh[2807]: OK
 2721 Feb 26 05:12:07.451206 vlab-01 INFO swss.sh[2807]: Warning: ignore interface 'fortyGigE0/24' in DEVICE_NEIGHBOR as it is not in the port_config.ini
 2722 Feb 26 05:12:07.451679 vlab-01 INFO swss.sh[2807]: Warning: ignore interface 'fortyGigE0/40' in DEVICE_NEIGHBOR as it is not in the port_config.ini

When creating router interface on the portchannel, it has default mtu 1492.

root@vlab-01:/var/log# grep "Create router interface" *
grep: lost+found: Is a directory
grep: swss: Is a directory
syslog:Feb 26 05:12:21.100838 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs: Create router interface Vlan1000 MTU 9100
syslog:Feb 26 05:12:21.683519 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs: Create router interface PortChannel0001 MTU 1492
syslog:Feb 26 05:12:21.684545 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs: Create router interface PortChannel0002 MTU 1492
syslog:Feb 26 05:12:21.692284 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs: Create router interface PortChannel0003 MTU 1492
syslog:Feb 26 05:12:21.701900 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs:477: Create router interface PortChannel0004 MTU 1492

configDB

127.0.0.1:6379[4]> hgetall "PORTCHANNEL|PortChannel0001"
 1) "mtu"
 2) "9100"
 3) "admin_status"
 4) "up"
 5) "members@"
 6) "Ethernet112"
 7) "port_offset"
 8) "0"
 9) "min_links"
10) "1"

appDB

127.0.0.1:6379> hgetall "LAG_TABLE:PortChannel0001"
1) "admin_status"
2) "up"
3) "oper_status"
4) "up"

At least "After=updategraph.service swss.service" is needed for teamd service.

[dawnbeauty]

another scenario, sudo systemctl restart swss without enable warm-boot, all the lag configuration in app db done by teammgrd also be flushed.

[jipanyang]

Actually it affects warm reboot too. After warm reboot, the correct mtu will be applied on portchannel interface and cause traffic disruption.

[stcheng]

I think the better fix is not pushing the configurations into application database; but reading configuration database instead of application database.

[yxieca]

We added service dependency to let teamd start after swss and that change addressed this issue.