Skip to content

Commit

Permalink
Record video using simctl
Browse files Browse the repository at this point in the history
  • Loading branch information
@RainNapper
RainNapper committed May 19, 2020
1 parent 89010ca commit e48eaeb
Show file tree
Hide file tree
Showing 11 changed files with 282 additions and 59 deletions.
69 changes: 35 additions & 34 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,41 +54,42 @@ $ bluepill -c config.json
A full list supported options are listed here.


| Config Arguments | Command Line Arguments | Explanation | Required | Default value |
|:----------------------:|:----------------------:|-------------------------------------------------------------------------------------|:--------:|:----------------:|
| app | -a | The path to the host application to execute (your `.app`) | N | n/a |
| xctestrun-path | | The path to the `.xctestrun` file that xcode leaves when you `build-for-testing`. | Y | n/a |
| Config Arguments | Command Line Arguments | Explanation | Required | Default value |
|:----------------------:|:----------------------:|----------------------------------------------------------------------------------------------|:--------:|:----------------:|
| app | -a | The path to the host application to execute (your `.app`) | N | n/a |
| xctestrun-path | | The path to the `.xctestrun` file that xcode leaves when you `build-for-testing`. | Y | n/a |
| test-plan-path | | The path of a json file which describes the test plan. It is equivalent to the `.xctestrun` file generated by Xcode, but it can be generated by a different build system, e.g. Bazel | Y | n/a |
| output-dir | -o | Directory where to put output log files. **(bluepill only)** | Y | n/a |
| config | -c | Read options from the specified configuration file instead of the command line. | N | n/a |
| device | -d | On which device to run the app. | N | iPhone 8 |
| exclude | -x | Exclude a testcase in the set of tests to run (takes priority over `include`). | N | empty |
| headless | -H | Run in headless mode (no GUI). | N | off |
| clone-simulator | -L | Spawn simulator by clone from simulator template. | N | off |
| xcode-path | -X | Path to xcode. | N | xcode-select -p |
| include | -i | Include a testcase in the set of tests to run (unless specified in `exclude`). | N | all tests |
| list-tests | -l | Only list tests and exit without executing tests. | N | false |
| num-sims | -n | Number of simulators to run in parallel. **(bluepill only)** | N | 4 |
| printf-config | -P | Print a configuration file suitable for passing back using the `-c` option. | N | n/a |
| error-retries | -R | Number of times to recover from simulator/app crashing/hanging and continue running.| N | 4 |
| failure-tolerance | -f | Number of times to retry on test failures | N | 0 |
| only-retry-failed | -F | Only retry failed tests instead of all. Also retry test that timed-out/crashed. | N | false |
| runtime | -r | What runtime to use. | N | iOS 13.3 |
| stuck-timeout | -S | Timeout in seconds for a test that seems stuck (no output). | N | 300s |
| test-timeout | -T | Timeout in seconds for a test that is producing output. | N | 300s |
| test-bundle-path | -t | The path to the test bundle to execute (single `.xctest`). | N | n/a |
| additional-unit-xctests| n/a | Additional XCTest bundles that is not Plugin folder | N | n/a |
| additional-ui-xctests | n/a | Additional XCTUITest bundles that is not Plugin folder | N | n/a |
| repeat-count | -C | Number of times we'll run the entire test suite (used for load testing). | N | 1 |
| no-split | -N | Test bundles you don't want to be packed into different groups to run in parallel. | N | n/a |
| quiet | -q | Turn off all output except fatal errors. | N | YES |
| diagnostics | n/a | Enable collection of diagnostics in output directory in case of test failures. | N | NO |
| help | -h | Help. | N | n/a |
| runner-app-path | -u | The test runner for UI tests. | N | n/a |
| screenshots-directory | n/a | Directory where simulator screenshots for failed ui tests will be stored. | N | n/a |
| video-paths | -V | A list of videos that will be saved in the simulators. | N | n/a |
| image-paths | -I | A list of images that will be saved in the simulators. | N | n/a |
| unsafe-skip-xcode-version-check | | Skip Xcode version check | N | NO |
| output-dir | -o | Directory where to put output log files. **(bluepill only)** | Y | n/a |
| config | -c | Read options from the specified configuration file instead of the command line. | N | n/a |
| device | -d | On which device to run the app. | N | iPhone 8 |
| exclude | -x | Exclude a testcase in the set of tests to run (takes priority over `include`). | N | empty |
| headless | -H | Run in headless mode (no GUI). | N | off |
| clone-simulator | -L | Spawn simulator by clone from simulator template. | N | off |
| xcode-path | -X | Path to xcode. | N | xcode-select -p |
| include | -i | Include a testcase in the set of tests to run (unless specified in `exclude`). | N | all tests |
| list-tests | -l | Only list tests and exit without executing tests. | N | false |
| num-sims | -n | Number of simulators to run in parallel. **(bluepill only)** | N | 4 |
| printf-config | -P | Print a configuration file suitable for passing back using the `-c` option. | N | n/a |
| error-retries | -R | Number of times to recover from simulator/app crashing/hanging and continue running. | N | 4 |
| failure-tolerance | -f | Number of times to retry on test failures | N | 0 |
| only-retry-failed | -F | Only retry failed tests instead of all. Also retry test that timed-out/crashed. | N | false |
| runtime | -r | What runtime to use. | N | iOS 13.3 |
| stuck-timeout | -S | Timeout in seconds for a test that seems stuck (no output). | N | 300s |
| test-timeout | -T | Timeout in seconds for a test that is producing output. | N | 300s |
| test-bundle-path | -t | The path to the test bundle to execute (single `.xctest`). | N | n/a |
| additional-unit-xctests| n/a | Additional XCTest bundles that is not Plugin folder | N | n/a |
| additional-ui-xctests | n/a | Additional XCTUITest bundles that is not Plugin folder | N | n/a |
| repeat-count | -C | Number of times we'll run the entire test suite (used for load testing). | N | 1 |
| no-split | -N | Test bundles you don't want to be packed into different groups to run in parallel. | N | n/a |
| quiet | -q | Turn off all output except fatal errors. | N | YES |
| diagnostics | n/a | Enable collection of diagnostics in output directory in case of test failures. | N | NO |
| help | -h | Help. | N | n/a |
| runner-app-path | -u | The test runner for UI tests. | N | n/a |
| screenshots-directory | n/a | Directory where simulator screenshots for failed ui tests will be stored. | N | n/a |
| videos-directory | n/a | Directory where videos of test runs will be saved. If not provided, videos are not recorded. | N | n/a |
| video-paths | -V | A list of videos that will be saved in the simulators. | N | n/a |
| image-paths | -I | A list of images that will be saved in the simulators. | N | n/a |
| unsafe-skip-xcode-version-check | | Skip Xcode version check | N | NO |


## Exit Status
Expand Down
67 changes: 66 additions & 1 deletion bluepill/tests/BPIntegrationTests.m
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ @interface BPIntegrationTests : XCTestCase

@implementation BPIntegrationTests

- (BPConfiguration *)generateConfig {
- (BPConfiguration *)generateConfigWithVideoDir:(NSString *)videoDir {
NSString *hostApplicationPath = [BPTestHelper sampleAppPath];
NSString *testBundlePath = [BPTestHelper sampleAppBalancingTestsBundlePath];
BPConfiguration *config = [[BPConfiguration alloc] initWithProgram:BP_MASTER];
Expand All @@ -36,9 +36,16 @@ - (BPConfiguration *)generateConfig {
config.deviceType = @BP_DEFAULT_DEVICE_TYPE;
config.headlessMode = YES;
config.quiet = [BPUtils isBuildScript];
if (videoDir != nil) {
config.videosDirectory = videoDir;
}
return config;
}

- (BPConfiguration *)generateConfig {
return [self generateConfigWithVideoDir: nil];
}

- (void)setUp {
[super setUp];
// Put setup code here. This method is called before the invocation of each test method in the class.
Expand Down Expand Up @@ -215,4 +222,62 @@ - (void)writeTestPlan {
[jsonData writeToFile:[BPTestHelper testPlanPath] atomically:YES];
}

- (void)testOutputFiles {
NSString *path = [BPUtils mkdtemp:@"bpout" withError:nil];
NSString *testFilePath = [path stringByAppendingPathComponent:@"test.txt"];
[[NSFileManager defaultManager] createFileAtPath:testFilePath contents:[@"datadata" dataUsingEncoding:NSUTF8StringEncoding] attributes:nil];

NSArray *directoryContent = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:path error:nil];
XCTAssertNotNil(directoryContent);
XCTAssertEqual(directoryContent.count, 1);

NSSet *filenameSet = [NSSet setWithArray: directoryContent];
XCTAssertEqual(filenameSet.count, 1);
BOOL hasFile = [filenameSet containsObject: @"test.txt"];
XCTAssertTrue(hasFile);
}

- (void)testTwoBPInstancesWithVideo {
NSError *mkdtempError;
NSString *path = [BPUtils mkdtemp:@"bpout" withError:&mkdtempError];
XCTAssertNil(mkdtempError);

NSString* videoDirName = @"my_videos";
NSString *videoPath = [path stringByAppendingPathComponent:videoDirName];
BPConfiguration *config = [self generateConfigWithVideoDir:videoPath];
config.numSims = @2;
config.errorRetriesCount = @1;
config.failureTolerance = @0;
// This looks backwards but we want the main app to be the runner
// and the sampleApp is launched from the callback.
config.testBundlePath = [BPTestHelper sampleAppUITestBundlePath];
config.testRunnerAppPath = [BPTestHelper sampleAppPath];
config.appBundlePath = [BPTestHelper sampleAppUITestRunnerPath];

NSError *err;
BPApp *app = [BPApp appWithConfig:config
withError:&err];

NSString *bpPath = [BPTestHelper bpExecutablePath];
BPRunner *runner = [BPRunner BPRunnerWithConfig:config withBpPath:bpPath];
XCTAssert(runner != nil);
int rc = [runner runWithBPXCTestFiles:app.testBundles];
XCTAssert(rc == 0);
XCTAssert([runner.nsTaskList count] == 0);

NSError *dirContentsError;
NSArray *directoryContent = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:videoPath error:&dirContentsError];
XCTAssertNil(dirContentsError);
XCTAssertNotNil(directoryContent);
XCTAssertEqual(directoryContent.count, 2);

NSString *testClass = @"BPSampleAppUITests";
NSSet *filenameSet = [NSSet setWithArray: directoryContent];
XCTAssertEqual(filenameSet.count, 2);
BOOL hasTest1 = [filenameSet containsObject: [NSString stringWithFormat:@"%@__%@__1.mp4", testClass, @"testExample"]];
XCTAssertTrue(hasTest1);
BOOL hasTest2 = [filenameSet containsObject: [NSString stringWithFormat:@"%@__%@__1.mp4", testClass, @"testExample2"]];
XCTAssertTrue(hasTest2);
}

@end
1 change: 1 addition & 0 deletions bp/src/BPConfiguration.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ typedef NS_ENUM(NSInteger, BPProgram) {
@property (nonatomic, strong) NSString *outputDirectory;
@property (nonatomic, strong) NSString *testTimeEstimatesJsonFile;
@property (nonatomic, strong) NSString *screenshotsDirectory;
@property (nonatomic, strong) NSString *videosDirectory;
@property (nonatomic, strong) NSString *simulatorPreferencesFile;
@property (nonatomic, strong) NSString *scriptFilePath;
@property (nonatomic) BOOL headlessMode;
Expand Down
22 changes: 20 additions & 2 deletions bp/src/BPConfiguration.m
View file
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,9 @@ typedef NS_OPTIONS(NSUInteger, BPOptionType) {
{364, "test-plan-path", BP_MASTER | BP_SLAVE, NO, NO, required_argument, NULL, BP_VALUE | BP_PATH, "testPlanPath",
"The path of a json file which describes the test plan. It is equivalent to the .xctestrun file generated by Xcode, but it can be generated by a different build system, e.g. Bazel"},
{365, "unsafe-skip-xcode-version-check", BP_MASTER | BP_SLAVE, NO, NO, no_argument, "Off", BP_VALUE | BP_BOOL , "unsafeSkipXcodeVersionCheck",
" "},

" "},
{366, "videos-directory", BP_MASTER | BP_SLAVE, NO, NO, required_argument, NULL, BP_VALUE | BP_PATH, "videosDirectory",
"Directory where videos of test runs will be saved. If not provided, videos are not recorded."},
{0, 0, 0, 0, 0, 0, 0}
};

Expand Down Expand Up @@ -772,6 +773,23 @@ - (BOOL)validateConfigWithError:(NSError *__autoreleasing *)errPtr {
}
}

if (self.videosDirectory) {
if ([[NSFileManager defaultManager] fileExistsAtPath:self.videosDirectory isDirectory:&isdir]) {
if (!isdir) {
BP_SET_ERROR(errPtr, @"%@ is not a directory.", self.videosDirectory);
return NO;
}
} else {
// create the directory
if (![[NSFileManager defaultManager] createDirectoryAtPath:self.videosDirectory
withIntermediateDirectories:YES
attributes:nil
error:errPtr]) {
return NO;
}
}
}

if (self.simulatorPreferencesFile) {
if ([[NSFileManager defaultManager] fileExistsAtPath:self.simulatorPreferencesFile isDirectory:&isdir]) {
if (isdir) {
Expand Down
4 changes: 3 additions & 1 deletion bp/src/BPTestBundleConnection.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
//

#import <Foundation/Foundation.h>
#import "BPExecutionContext.h"
#import "BPSimulator.h"

// This is a small subset of XCTestManager_IDEInterface protocol
Expand All @@ -16,9 +17,10 @@

@interface BPTestBundleConnection : NSObject
@property (nonatomic, strong) BPConfiguration *config;
@property (nonatomic, strong) BPExecutionContext *context;
@property (nonatomic, strong) BPSimulator *simulator;
@property (nonatomic, copy) void (^completionBlock)(NSError *, pid_t);
- (instancetype)initWithDevice:(BPSimulator *)device andInterface:(id<BPTestBundleConnectionDelegate>)interface;
- (instancetype)initWithContext:(BPExecutionContext *)context andInterface:(id<BPTestBundleConnectionDelegate>)interface;
- (void)connectWithTimeout:(NSTimeInterval)timeout;
- (void)startTestPlan;
@end
Expand Down
Loading

0 comments on commit e48eaeb

Please sign in to comment.