👩‍🌾 ReplayStep test regression in Windows

[chapulina]

This test used to pass consistently, but started failing since 2020-10-23 for ign-transport8:

https://build.osrfoundation.org/job/ignition_transport-ci-ign-transport8-windows7-amd64/28/testReport/(root)/playback/ReplayStep/history/

It passes on ign-transport9 though:

https://build.osrfoundation.org/job/ignition_transport-ci-ign-transport9-windows7-amd64/1/testReport/(root)/playback/ReplayStep/history/

Failure log:

79: [ RUN      ] playback.ReplayStep
79: Chirping [100] times on [3] topics:
79:  -- /foo
79:  -- /bar
79:  -- /baz
79: Stepping playback...
79: D:\Jenkins\workspace\ignition_transport-ci-ign-transport8-windows7-amd64\ws\ign-transport\log\test\integration\playback.cc(660): error: Value of: MessagesAreEqual(secondMessageData, thirdMessageData)
79:   Actual: true
79: Expected: false
79: Waiting to for playback to finish...
79:  Done waiting...
79: Playback finished!
79: [  FAILED  ] playback.ReplayStep (4688 ms)

[Crola1702]

I entered the test regression in buildfarmer database and found out that this issue happened less than 25 times. It has happened in the following jobs:

Job	Last time seen	First time seen (Since 2021)
ign_transport-ign-8-win	2023-01-24	2021-05-01
ign_transport-gz-12-win	2022-12-01	2022-08-31
ign_transport-ign-11-win	2022-09-14	2022-05-25
ign_transport-ci-win	2022-08-02	2022-08-02

This means that this issue is happening on Citadel, Fortress and Garden

[Crola1702]

This regression is still happening flaky on transport-11 and transport-13

Flaky ratio in the last 30 days:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
ign_transport-ign-11-win	2023-10-20	2023-10-20	7	1	14.29
ign_transport-gz-13-win	2023-10-17	2023-10-08	10	2	20.0

FYI @azeey

[Crola1702]

Flaky ration in the last 30 days:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
gz_transport-ign-transport11-win	2024-01-07	2023-12-18	7	2	28.57
gz_transport-ign-transport8-win	2024-01-03	2024-01-03	7	1	14.29
gz_transport-12-win	2023-12-30	2023-12-17	7	3	42.86
gz_transport-13-win	2023-12-26	2023-12-12	8	2	25.0

[Crola1702]

This issue seems to be happening more often this week.

Flaky ratio in the last 15 days:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
gz_transport-main-win	2024-08-03	2024-07-30	4	2	50.0
gz_transport-ign-transport8-win	2024-08-03	2024-08-03	4	1	25.0
gz_transport-12-win	2024-07-25	2024-07-25	4	1	25.0

[Crola1702]

Log output is almost the same as before:

83: [ RUN      ] playback.ReplayStep
83: Stepping playback...
83: C:\J\workspace\gz_transport-main-win\ws\gz-transport\log\test\integration\playback.cc(652): error: Value of: MessagesAreEqual(firstMessageData, secondMessageData)
83:   Actual: true
83: Expected: false
83: Waiting to for playback to finish...
83:  Done waiting...
83: Playback finished!
83: [  FAILED  ] playback.ReplayStep (11265 ms)