Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[teamd]: Warm/Fast reboot: Send USR1/USR2 only to subscribers #30

Closed
wants to merge 1 commit into from

Conversation

nazariig
Copy link
Owner

Signed-off-by: Nazarii Hnydyn nazariig@nvidia.com

Before:

Sep 27 15:36:39.593423 sonic NOTICE admin: Pre-shutdown succeeded, state: pre-shutdown-succeeded ...
Sep 27 15:36:39.818878 sonic NOTICE admin: Backing up database ...
Sep 27 15:36:39.892730 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel0003' has been removed.
Sep 27 15:36:39.892826 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel0004' has been removed.
Sep 27 15:36:39.892877 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel0002' has been removed.
Sep 27 15:36:39.892899 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel0001' has been removed.
Sep 27 15:36:39.980706 sonic INFO database#/supervisord: redis 35:M 27 Sep 2021 15:36:39.980 * DB saved on disk
Sep 27 15:36:40.104994 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-etc-sonic.mount: Succeeded.
Sep 27 15:36:40.125423 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-usr-share-sonic-platform.mount: Succeeded.
Sep 27 15:36:40.156169 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-run-redis.mount: Succeeded.
Sep 27 15:36:40.182548 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-etc-resolv.conf.mount: Succeeded.
Sep 27 15:36:40.202195 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-etc-hostname.mount: Succeeded.
Sep 27 15:36:40.221990 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-etc-hosts.mount: Succeeded.
Sep 27 15:36:40.397449 sonic NOTICE admin: Stopping teamd ...
Sep 27 15:36:40.406095 sonic INFO systemd[1]: Stopping TEAMD container...
Sep 27 15:36:40.413385 sonic NOTICE admin: Stopping teamd service...
Sep 27 15:36:40.615055 sonic NOTICE admin: Warm boot flag: teamd true.
Sep 27 15:36:40.621085 sonic NOTICE admin: Fast boot flag: teamd false.
Sep 27 15:36:40.755013 sonic INFO teamd#supervisord 2021-09-27 15:36:40,754 INFO exited: tlm_teamd (terminated by SIGUSR1; not expected)
Sep 27 15:36:40.979849 sonic DEBUG /container: container_stop: BEGIN
Sep 27 15:36:40.980191 sonic DEBUG /container: read_data: config:True feature:teamd fields:[('set_owner', 'local'), ('no_fallback_to_local', False)] val:['local', False]
Sep 27 15:36:40.980440 sonic DEBUG /container: read_data: config:False feature:teamd fields:[('current_owner', 'none'), ('remote_state', 'none'), ('container_id', '')] val:['none', 'none', '']
Sep 27 15:36:40.980672 sonic DEBUG /container: container_stop: teamd: set_owner:local current_owner:none remote_state:none docker_id:teamd
Sep 27 15:36:41.758506 sonic INFO teamd#supervisord 2021-09-27 15:36:41,756 INFO reaped unknown pid 25 (exit status 0)
Sep 27 15:36:41.758506 sonic INFO teamd#supervisord 2021-09-27 15:36:41,756 INFO reaped unknown pid 34 (exit status 0)
Sep 27 15:36:41.758506 sonic INFO teamd#supervisord 2021-09-27 15:36:41,757 INFO reaped unknown pid 42 (exit status 0)
Sep 27 15:36:41.758506 sonic INFO teamd#supervisord 2021-09-27 15:36:41,757 INFO reaped unknown pid 50 (exit status 0)
Sep 27 15:36:41.766193 sonic INFO teamd#/supervisor-proc-exit-listener: Process 'tlm_teamd' exited unexpectedly. Terminating supervisor 'teamd'
Sep 27 15:36:41.767071 sonic INFO teamd#supervisord 2021-09-27 15:36:41,766 WARN received SIGTERM indicating exit request
Sep 27 15:36:41.767331 sonic INFO teamd#supervisord 2021-09-27 15:36:41,766 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd, teamsyncd to die
Sep 27 15:36:43.770027 sonic NOTICE teamd#teamsyncd: :- cleanTeamSync: Cleaning up LAG teamd resources ...
Sep 27 15:36:43.771247 sonic NOTICE teamd#teamsyncd: :- main: Received SIGTERM Exiting
Sep 27 15:36:44.416124 sonic INFO teamd#supervisord 2021-09-27 15:36:44,415 INFO stopped: teamsyncd (exit status 0)
Sep 27 15:36:45.417286 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: Cleaning up LAGs during shutdown...
Sep 27 15:36:45.418185 sonic INFO teamd#supervisord 2021-09-27 15:36:45,417 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd to die
Sep 27 15:36:45.421040 sonic INFO teamd#/supervisord: teammgrd cat: /var/run/teamd/PortChannel0001.pid: No such file or directory
Sep 27 15:36:45.421795 sonic ERR teamd#teammgrd: :- main: Runtime error: cat "/var/run/teamd/PortChannel0001.pid" :
Sep 27 15:36:46.271465 sonic INFO teamd#supervisord 2021-09-27 15:36:46,270 INFO stopped: teammgrd (exit status 1)
Sep 27 15:36:47.425981 sonic INFO containerd[486]: time="2021-09-27T15:36:47.424407649Z" level=info msg="shim reaped" id=90f942c9e7be6991ae0ce30bc71738de51f1f599773106caf1c70860cb87731a
Sep 27 15:36:47.436674 sonic INFO dockerd[726]: time="2021-09-27T15:36:47.434422086Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 27 15:36:47.446982 sonic INFO systemd[1]: var-lib-docker-containers-90f942c9e7be6991ae0ce30bc71738de51f1f599773106caf1c70860cb87731a-mounts-shm.mount: Succeeded.
Sep 27 15:36:47.469852 sonic INFO systemd[1]: var-lib-docker-overlay2-06a63c38bf0b38196702d168ba789fed9969900a0ac534ea6836a280b3b643d5-merged.mount: Succeeded.
Sep 27 15:36:47.497159 sonic INFO /container: docker cmd: wait for teamd
Sep 27 15:36:47.498422 sonic INFO /container: docker cmd: stop for teamd
Sep 27 15:36:47.498678 sonic DEBUG /container: container_stop: END
Sep 27 15:36:47.527123 sonic NOTICE admin: Stopped teamd service...
Sep 27 15:36:47.529288 sonic INFO systemd[1]: teamd.service: Succeeded.
Sep 27 15:36:47.529910 sonic INFO systemd[1]: Stopped TEAMD container.
Sep 27 15:36:47.535760 sonic NOTICE admin: Stopped  teamd ...

After:

Sep 27 15:43:53.121517 sonic NOTICE admin: Pre-shutdown succeeded, state: pre-shutdown-succeeded ...
Sep 27 15:43:53.355894 sonic NOTICE admin: Backing up database ...
Sep 27 15:43:53.429249 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel0003' has been removed.
Sep 27 15:43:53.429249 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel0002' has been removed.
Sep 27 15:43:53.429249 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel0004' has been removed.
Sep 27 15:43:53.429368 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel0001' has been removed.
Sep 27 15:43:53.523066 sonic INFO database#/supervisord: redis 36:M 27 Sep 2021 15:43:53.522 * DB saved on disk
Sep 27 15:43:53.647846 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-etc-sonic.mount: Succeeded.
Sep 27 15:43:53.673307 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-usr-share-sonic-platform.mount: Succeeded.
Sep 27 15:43:53.706143 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-run-redis.mount: Succeeded.
Sep 27 15:43:53.753142 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-etc-resolv.conf.mount: Succeeded.
Sep 27 15:43:53.776188 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-etc-hostname.mount: Succeeded.
Sep 27 15:43:53.799224 sonic INFO systemd[1]: var-lib-docker-overlay2-669b2b0b164bc3da2be6b7bed86437d1a47a9235297a62c71e1f159a292ba970-merged-etc-hosts.mount: Succeeded.
Sep 27 15:43:53.956416 sonic NOTICE admin: Stopping teamd ...
Sep 27 15:43:53.965053 sonic INFO systemd[1]: Stopping TEAMD container...
Sep 27 15:43:53.973066 sonic NOTICE admin: Stopping teamd service...
Sep 27 15:43:54.179522 sonic NOTICE admin: Warm boot flag: teamd true.
Sep 27 15:43:54.186392 sonic NOTICE admin: Fast boot flag: teamd false.
Sep 27 15:43:54.540820 sonic DEBUG /container: container_stop: BEGIN
Sep 27 15:43:54.541168 sonic DEBUG /container: read_data: config:True feature:teamd fields:[('set_owner', 'local'), ('no_fallback_to_local', False)] val:['local', False]
Sep 27 15:43:54.541496 sonic DEBUG /container: read_data: config:False feature:teamd fields:[('current_owner', 'none'), ('remote_state', 'none'), ('container_id', '')] val:['none', 'none', '']
Sep 27 15:43:54.541765 sonic DEBUG /container: container_stop: teamd: set_owner:local current_owner:none remote_state:none docker_id:teamd
Sep 27 15:43:55.103875 sonic INFO teamd#supervisord 2021-09-27 15:43:55,101 INFO reaped unknown pid 26 (exit status 0)
Sep 27 15:43:55.103875 sonic INFO teamd#supervisord 2021-09-27 15:43:55,102 INFO reaped unknown pid 34 (exit status 0)
Sep 27 15:43:55.103875 sonic INFO teamd#supervisord 2021-09-27 15:43:55,102 INFO reaped unknown pid 42 (exit status 0)
Sep 27 15:43:55.103875 sonic INFO teamd#supervisord 2021-09-27 15:43:55,102 INFO reaped unknown pid 50 (exit status 0)
Sep 27 15:43:56.104759 sonic INFO teamd#supervisord 2021-09-27 15:43:56,104 WARN received SIGTERM indicating exit request
Sep 27 15:43:56.105595 sonic INFO teamd#supervisord 2021-09-27 15:43:56,104 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd, teamsyncd, tlm_teamd to die
Sep 27 15:43:57.105912 sonic NOTICE teamd#tlm_teamd: :- main: Exiting
Sep 27 15:43:57.817711 sonic INFO teamd#supervisord 2021-09-27 15:43:57,817 INFO stopped: tlm_teamd (exit status 0)
Sep 27 15:43:58.818834 sonic NOTICE teamd#teamsyncd: :- cleanTeamSync: Cleaning up LAG teamd resources ...
Sep 27 15:43:58.820140 sonic NOTICE teamd#teamsyncd: :- main: Received SIGTERM Exiting
Sep 27 15:43:58.948445 sonic INFO teamd#supervisord 2021-09-27 15:43:58,947 INFO stopped: teamsyncd (exit status 0)
Sep 27 15:43:59.949582 sonic NOTICE teamd#teammgrd: :- cleanTeamProcesses: Cleaning up LAGs during shutdown...
Sep 27 15:43:59.950284 sonic INFO teamd#supervisord 2021-09-27 15:43:59,949 INFO waiting for supervisor-proc-exit-listener, rsyslogd, teammgrd to die
Sep 27 15:43:59.953287 sonic INFO teamd#/supervisord: teammgrd cat: /var/run/teamd/PortChannel0001.pid: No such file or directory
Sep 27 15:43:59.954101 sonic ERR teamd#teammgrd: :- main: Runtime error: cat "/var/run/teamd/PortChannel0001.pid" :
Sep 27 15:44:00.790477 sonic INFO teamd#supervisord 2021-09-27 15:44:00,789 INFO stopped: teammgrd (exit status 1)
Sep 27 15:44:01.939112 sonic INFO containerd[486]: time="2021-09-27T15:44:01.936948014Z" level=info msg="shim reaped" id=90f942c9e7be6991ae0ce30bc71738de51f1f599773106caf1c70860cb87731a
Sep 27 15:44:01.949213 sonic INFO dockerd[675]: time="2021-09-27T15:44:01.947031450Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 27 15:44:01.958717 sonic INFO systemd[1]: var-lib-docker-containers-90f942c9e7be6991ae0ce30bc71738de51f1f599773106caf1c70860cb87731a-mounts-shm.mount: Succeeded.
Sep 27 15:44:01.982581 sonic INFO systemd[1]: var-lib-docker-overlay2-06a63c38bf0b38196702d168ba789fed9969900a0ac534ea6836a280b3b643d5-merged.mount: Succeeded.
Sep 27 15:44:02.026851 sonic INFO /container: docker cmd: wait for teamd
Sep 27 15:44:02.028477 sonic INFO /container: docker cmd: stop for teamd
Sep 27 15:44:02.028767 sonic DEBUG /container: container_stop: END
Sep 27 15:44:02.057161 sonic NOTICE admin: Stopped teamd service...
Sep 27 15:44:02.060093 sonic INFO systemd[1]: teamd.service: Succeeded.
Sep 27 15:44:02.061628 sonic INFO systemd[1]: Stopped TEAMD container.

Why I did it

  • To fix teamd signal handling:
Sep 27 15:36:40.755013 sonic INFO teamd#supervisord 2021-09-27 15:36:40,754 INFO exited: tlm_teamd (terminated by SIGUSR1; not expected)
Sep 27 15:36:41.766193 sonic INFO teamd#/supervisor-proc-exit-listener: Process 'tlm_teamd' exited unexpectedly. Terminating supervisor 'teamd'

How I did it

  • N/A

How to verify it

  • N/A

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106

Description for the changelog

  • N/A

A picture of a cute animal (not mandatory but encouraged)

  • N/A

Signed-off-by: Nazarii Hnydyn <nazariig@nvidia.com>
@nazariig nazariig closed this Sep 29, 2021
nazariig pushed a commit that referenced this pull request Feb 9, 2022
Updating ztp submodule to include the below commits
f7dd3c5 [sonic-ztp]Fixing build failure after bullseye integration (#30)
9218e16 Replace swsssdk.ConfigDBConnector and SonicV2Connector with swsscommon(#28)

Signed-off-by: Sudharsan Dhamal Gopalarathnam <sudharsand@nvidia.com>
nazariig pushed a commit that referenced this pull request Feb 9, 2022
* [BFN] Updated platform APIs impl

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

* Extended BFN platform SFP APIs implementation

* Update sfp.py

* [BFN] Extended SFP platform plugin implementation

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

* [BFN] Extended Fans platform plugin implementation

* [BFN] divided classes Fan and  FanDrawer into 2 files

* Signed-off-by: Vadym Yashchenko <vadymx.yashchenko@intel.com>

What I did
	Add get_model() function
	Add get_low_critical_threshold() function
	Change __get(...) function.
How I did it
	Differnece from previous implementation of __get(...) function is return real value or -9999.9 if value is not provided by thrift API

* Add get_presence() function and revised __get() function

Signed-off-by: Vadym Yashchenko <vadymx.yashchenko@intel.com>

* [BFN] Updated PSU platform APIs impl

Signed-off-by: Dmytro Lytvynenko <dmytrox.lytvynenko@intel.com>

* Added BFN PSU cache (#9)

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

* [BFN]  Fans and Fantray platform APIs update (#7)

* [BFN] Updated SFP platform APIs (#10)

Signed-off-by: Volodymyr Boyko <volodymyrx.boiko@intel.com>

* [BFN] Updated platform API for thermal (#8)

* Signed-off-by: Vadym Yashchenko <vadymx.yashchenko@intel.com>

* Revert "[BFN]  Fans and Fantray platform APIs update (#7)" (#11)

This reverts commit c62a733.

* Add support health monitor system (#15)

Signed-off-by: Petro Bratash <petrox.bratash@intel.com>

* Update chassis.py

* [BFN] Updated FANs and FAN Tray platform API (#14)

* Fix fix_alignment (#17)

Signed-off-by: Petro Bratash <petrox.bratash@intel.com>

* [BFN] Improvement show environment (#16)

* Added PSU temperature skip into platform.json (#18)

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

* Do not skip psud on Newport

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

* [BFN] fix fan status from Not OK to Ok (#19)

* [BFN] Updated SFP platform plugin (#13)

Signed-off-by: Volodymyr Boyko <volodymyrx.boiko@intel.com>

* [DPB] Fix typo for Ethernet0 2x200G[100G,40G] breakout mode (#21)

Signed-off-by: Mykola Gerasymenko <mykolax.gerasymenko@intel.com>

* [barefoot] Tmp fix vendor_rev (#22)

Signed-off-by: Volodymyr Boyko <volodymyrx.boiko@intel.com>

* Fixed python issues in sonic_platform/fan_drawer.py

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

* Updated fan_drawer.py

* Fixing trailing white spaces in fan_drawer.py

* [BFN] Fix thrift for SFPs API

Signed-off-by: Volodymyr Boyko <volodymyrx.boiko@intel.com>

* In platform.json, replaced 'false' with '0' to workaround ast.literal_eval() issue

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

* [Newport] Thermal manager  (#23)

* Signed-off-by: Vadym Yashchenko <vadymx.yashchenko@intel.com>

* Revert "In platform.json, replaced 'false' with '0' to workaround ast.literal_eval() issue"

This reverts commit 1e73127.

* Removed 'controllable' options from platform.json to fix factory default config generation

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

* Update thermal_manager.py

* Migrated SFP plugin to sonic_xcvr API (#30)

Signed-off-by: Andriy Kokhan <andriyx.kokhan@intel.com>

Co-authored-by: KostiantynYarovyiBf <kostiantynx.yarovyi@intel.com>
Co-authored-by: Vadym Yashchenko <vadymx.yashchenko@intel.com>
Co-authored-by: Dmytro Lytvynenko <dmytrox.lytvynenko@intel.com>
Co-authored-by: Volodymyr Boiko <volodymyrx.boiko@intel.com>
Co-authored-by: Petro Bratash <petrox.bratash@intel.com>
Co-authored-by: Mykola Gerasymenko <mykolax.gerasymenko@intel.com>
nazariig pushed a commit that referenced this pull request Mar 30, 2022
ce72b0d Longxiang Lyu Thu Feb 24 06:05:12 2022 Put handler member functions as virtual in base (#30)
ef59e4f Jing Zhang Fri Feb 25 11:38:28 2022 Incrementing tolerance on mux state inconsistency (#27)
2d12892 Longxiang Lyu Wed Feb 16 03:32:06 2022 Rename LinkManagerStateMachine to ActiveStandbyStateMachine (#26)
f38634c Jing Zhang Thu Feb 17 17:23:56 2022 Update log level for mux probing and mux state chance (#23)
a8434dd Jing Zhang Thu Feb 17 17:21:01 2022 Handle xcvrd crashing scenarios (#22)
2ebdb2b Longxiang Lyu Mon Feb 14 13:26:07 2022 [make] Enable make extra includes (#24)
nazariig pushed a commit that referenced this pull request May 16, 2023
67a3bdf show counters wrong cli output fixed (#36)
5b3eea1 Update package cache, and bail on the first error (#35)
1d221b0 dhcpv6 relay UT code coverage improve (#32)
514b084 dhcpv6 packet handling code refine (#30)

Signed-off-by: Vivek Reddy <vkarri@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants