flutter_integration_test is very flaky [iOS] only

[marandaneto]

Description

sentry-dart/.github/workflows/flutter_integration_test.yml

Lines 88 to 89 in 62de927

	- name: run ios integration test
	run: flutter test integration_test/integration_test.dart --verbose

Run flutter test integration_test/integration_test.dart --verbose
[ +3 ms] executing: [/Users/runner/hostedtoolcache/flutter/stable-3.10.0-x64/] git -c log.showSignature=false log -n 1 --pretty=format:%H
[ +66 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[ ] 84a1e904f44f9b0e9c4510138010edcc653163f8
[ ] executing: [/Users/runner/hostedtoolcache/flutter/stable-3.10.0-x64/] git tag --points-at 84a1e904f44f9b0e9c4510138010edcc653163f8
[ +40 ms] Exit code 0 from: git tag --points-at 84a1e904f44f9b0e9c4510138010edcc653163f8
[ ] 3.10.0
[ +4 ms] executing: [/Users/runner/hostedtoolcache/flutter/stable-3.10.0-x64/] git rev-parse --abbrev-ref HEAD
[ +14 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[ ] stable
[ +112 ms] executing: [/Users/runner/hostedtoolcache/flutter/stable-3.10.0-x64/] git rev-parse --abbrev-ref --symbolic @{upstream}
[ +19 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{upstream}
[ ] origin/stable
[ +37 ms] executing: [/Users/runner/hostedtoolcache/flutter/stable-3.10.0-x64/] git ls-remote --get-url origin
[ +87 ms] Exit code 0 from: git ls-remote --get-url origin
[ +91 ms] https://github.com/flutter/flutter.git
[ +24 ms] executing: sysctl hw.optional.arm64
[ +10 ms] Exit code 1 from: sysctl hw.optional.arm64
[ +1 ms] sysctl: unknown oid 'hw.optional.arm64'
[ +152 ms] Found 1 files which will be executed as Integration Tests.
[ +15 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[ ] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[ ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[ ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[ +95 ms] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[ +1 ms] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[ ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[ ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[ ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[ ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[ ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[ +558 ms] executing: /Users/runner/Library/Android/sdk/platform-tools/adb devices -l
[ +145 ms] executing: sysctl hw.optional.arm64
[ +17 ms] Exit code 1 from: sysctl hw.optional.arm64
[ ] sysctl: unknown oid 'hw.optional.arm64'
[ ] executing: xcrun xcodebuild -version
[+4648 ms] Exit code 0 from: xcrun xcodebuild -version
[ ] Xcode 14.2
Build version 14C18
[ +9 ms] executing: xcrun xcdevice list --timeout 2
[ +6 ms] xcrun simctl list devices booted iOS --json
[ ] executing: xcrun simctl list devices booted iOS --json
[ +28 ms] executing: xcrun simctl list devices booted
[+1683 ms] Exit code 0 from: xcrun simctl list devices booted

Source: https://github.com/getsentry/sentry-dart/actions/runs/4979670694/jobs/8911566016?pr=1447

It works locally, its either the emulator not booting or the tests timing out.

[denrase]

Maybe creating and booting simulator with xcrun is more stable. https://medium.com/xcblog/simctl-control-ios-simulators-from-command-line-78b9006a20dc

[denrase]

The action is also using xcrun, so this might solve flakyness.

https://github.com/futureware-tech/simulator-action/blob/736249ceb5ed7916224ed6b6d3c3582fd8049deb/src/xcrun.ts#L23

[marandaneto]

Sometimes I also see this:

✅ setup sentry and render app
Error: The operation was canceled.

This means after the first test is executed, it gets stuck, and times out.

[denrase]

Just tried to create & boot a new simulator with xcrun, but i'm seeing the same behaviour. The first test is run and then the action is stuck. At least we know that it's not the action we have used to launch simulators.

[denrase]

@marandaneto Can we close this?