Xcode/XCTest Runner Integration #454
Conversation
This inverts the usual Catch behavior; we register stub XCTestCase classes that allow Xcode's test runner to invoke Catch for each individual test case.
|
Hello! I am interested in having Xcode support for Catch as well. Why is this PR still open after all this time but no comments have been made? Can we merge it? |
|
@philsquared is there an ETA on when this will be addressed? Would like to know if this will be integrated or not given how long this has been open. |
|
I've been using this patch in my local repo and it's been working pretty well! |
There was a problem hiding this comment.
This patch works damned well, but I do have a couple of gripes:
- You only see one XCTest listed per TEST_CASE/SCENARIO block, but not per SECTION/WHEN block, limiting the Catch styling options you can use if you want it to give you a clear readout
- You need to relaunch XCode if you add new tests, so that it regenerates the test list with the new tests included
Would it be possible to fix either of these issues?
|
Nevermind on point #1:
:) This should really be merged in so others can gain its benefit. |
|
Sorry all - I've been swamped over the last year or two and got so far behind on issues and PRs that I've been missing a lot of them before I even get to read them - including this one. So, looking at this issue my first thought is: people are still using Catch with Objective-C? I'll have a closer look at the PR itself and see if I can adopt it shortly. |
|
Yes. Using catch with objc. |
|
👍 |
|
@philsquared thanks for taking the time. In my case I have some developers using XCode as a C++ IDE on OSX, and having Catch integrated is extremely helpful |
|
@landonf For some weird reason, only after the first execution, the absolute last testcase to be listed (alphabetically!) appears in Test Navigator for a brief moment, and then quickly disappears, whether or not it failed and there's no code highlighting telling you there was a problem. The first time you execute, everything is visible, but the second time, that last test disappears. This can be worked around by creating a single pointless test called "zzzzzIgnoreMe" (or something to ensure its the last test to be listed) in XCTestRunner.mm file, but its super hacky. |
|
👍 |
|
Any news on this issue? |
|
Another vote for this one. This would be extra handy for IDE integration with C++ code. Objective-C and Swift support would be icing. There are already some tutorials (https://accu.org/index.php/journals/1851) for IDE integration of Catch with Visual Studio. Matching that on the Xcode side would allow me to push for universal adoption within our org. Without that it won't fly. There are those who will push back against non-portable tests for portable units, and those that will push back against anything that does not have IDE integration (since almost all IDEs have something integrated now). |
|
This is very much still on my radar - and creeping up my list. In the meantime, I don't know if it's of interest, but test runner integration with Catch is already there, now, in Visual Studio if you use ReSharper C++ (as of 2016.3), and will be in the next releases of CLion and AppCode (2017.1). |
|
Thanks for the update phil. Unfortunately it will be the xcode integration, specifically, that will hold us back. I'll keep an eye out for updates. |
|
I've pulled this PR and spent some time improving it. First of all, test names are no longer reduced to just digits and latin letters. All the names are now the same as they're supposed to be, including spaces, punctuation and other languages. It even supports emojis, just in case anybody cares 😂. Having a lot of test cases with but a single method each feels like a waste. On top of that, classes and methods having duplicate names, makes things even more confusing. So the next obvious step was to group them. As for those without a class, they can be grouped by source file name. As unit tests usually have debug info available, source file mapping is not really a problem. And since they're usually located under the single So the Test Navigator can actually looks like this: Of course, there's still a long way to go. For example, jumping from Test Navigator to the succeeded tests is broken, and Xcode also refuses to show green/red dots in the editor next to the test methods. I'm not even sure if it can be fixed at all… I'd be happy to share it with everyone, but filing a PR for another PR from 2 years ago feels like… ugh. So it would be nice if the author tells if he Is interested in this. |
|
@pieceofsummer I don't do OS X, so it will be up to @philsquared to review and merge it. Generally though, we are interested in IDE integration, as long as the changes needed to support it are reasonable. Of course, what is reasonable is in the eye of a beholder 😄 |
|
@pieceofsummer I'm very much interested and looking to include this very soon, so please raise that revised PR! |
|
Are there any news on this Xcode integration? It would be super awesome to have this working in some way since many of us use Xcode on macOS as their main C++ IDE. |
|
This would be super helpful for myself and my peers as well. |
|
What's the state of this? Is there documentation? |
|
I suspect @pieceofsummer has moved on to other things. So it falls to me to look at the original PR and see how it can be applied to the current code-base. I probably won't get to that in the short term - but, of course, if anyone else would like to pick it up I'd be very grateful (and will try to prioritise reviewing any subsequent PR). |
|
If the tests pass shouldn't it be a question of merging this? |
|
Can we please merge this? I still want Xcode integration with Catch. Thx! |
|
Its been roughly a year now, is there still no way to integrate catch2 with Xcode? @philsquared |
|
I'm interested in this too because XCode test runner seems to have code coverage options (we're using catch for C++ unittests). |
|
Just stumbled upon this masterpiece after spending some really horrible time trying to run Catch tests in an iOS app. Any plans to merge this and make our Xcode tests green? |
|
I'm interested in this too 👍 |
|
I had a quick look at the PR, but it's not quite clear to me how to integrate it in the current code base: could someone provide insights on the new structure of the project? Where should I start to understand how it works? Besides this PR, I think some documentation on this would facilitate contributions. |
|
I've made it work for my ios-testing projects. Used a bit of cmake magic, this entire patch went into my codebase, without touching catch code. But now it works and runs catch tests on the simulator. |
|
Are you using it with the current version of Catch2 or with Catch 1.x? I suppose it could run tests on a real device, couldn't it? Anyway, do you have a public project I could look at? |
|
I'm using it with Catch 2.1; it should just the same run tests on real device - all it needs is a IOS_SIMULATOR env var to set up. Unfortunately, this is not a part of public project but I'll try to extract a standalone public sample. |
|
Ok, I managed to make it work in Catch 2.2.2. I need to thank you for the suggestion to make the PR a part of my code, not part of Catch2... simply I didn't think at this possibility. For now I tested a super simple setup, in which I simply added catch.hpp and XCTestRunner.mm to my tests directory, and now I can run some simple Catch2 tests - will try more real-life tests in the next days. Still, I needed a slight modification in Catch2 to make this work, to correct what I believe is a bug in Catch2: --- a/single_include/catch.hpp
+++ b/single_include/catch.hpp
@@ -2928,8 +2928,9 @@ namespace Catch {
std::string name = Detail::getAnnotation( cls, "Name", testCaseName );
std::string desc = Detail::getAnnotation( cls, "Description", testCaseName );
const char* className = class_getName( cls );
- getMutableRegistryHub().registerTest( makeTestCase( new OcMethod( cls, selector ), className, name.c_str(), desc.c_str(), SourceLineInfo("",0) ) );
+ NameAndTags nameAndTags(name);
+ getMutableRegistryHub().registerTest( makeTestCase( new OcMethod( cls, selector ), className, nameAndTags, SourceLineInfo("",0) ) );
noTestMethods++;
}
}Now, if this modification is ok and will be applied, I could update this PR to make it work with the current version of Catch: is this something @philsquared is still interested in? |
|
Just noticed that the needed bugfix in Catch has been committed: 788f812 |
|
Sounds like this PR is good to go! |
|
@davnat Im really interested in your update of this PR to work with Catch2. |
|
I'm a bit busy at the moment, but as soon as I'll have some spare time I'll create an updated PR. |
|
I am also trying to update this PR to the latest Catch2 master. My work-in-progress branch is https://github.com/ankch/Catch2/tree/catch2-objc-registration-support-reverted-singleton and the bulk of my changes are in antcz@361a947. However, I've hit a roadblock where the Catch2/projects/runners/XCTestRunner/XCTestRunner.mm Lines 366 to 373 in 56a6704 I am working around it for now by reverting the registry hub singleton patch in my branch. Does anyone have suggestions? |
|
That change seems to state that the use of RegistryHub class as the internal registry hub is a private implementation detail, and not something that client software such as this PR can change. Every solution I can imagine requires a change in the design of how the registry hub is created and handled, in order to allow the change of the type to be used internally as the registry hub. What is the position of @philsquared and @horenmar on this? |
|
any plans about furthr support for XCTest Runner integration? |
|
Here's a cleaned up version with the @pieceofsummer features. #define CATCH_CONFIG_RUNNER
#include <catch.hpp>
#include <XCTest/XCTest.h>
#include <inttypes.h>
#include <objc/message.h>
@interface CatchXCTest : NSObject
+ (XCTestSuite*)testSuiteForClass:(Class)klass;
+ (void)storeSuite:(XCTestSuite*)suite forClass:(Class)klass;
+ (Catch::TestCase&)testCaseForSelector:(SEL)selector;
+ (void)storeTestCase:(const Catch::TestCase&)test forSelector:(SEL)selector;
@end
namespace Catch {
class XCTestReporter : public StreamingReporterBase<XCTestReporter> {
public:
XCTestReporter (ReporterConfig const &config) : StreamingReporterBase<XCTestReporter>(config) {}
virtual ~XCTestReporter () {}
static std::string getDescription() {
return "Reports test results via Xcode's XCTest interface";
}
XCTestCase *xcTestCase;
private:
virtual void assertionStarting(AssertionInfo const &) {}
virtual bool assertionEnded(AssertionStats const &assertionStats) {
AssertionStats as = assertionStats;
auto result = as.assertionResult;
if (result.getResultType() == ResultWas::Ok || (result.getResultType() == ResultWas::ExpressionFailed && result.isOk()))
return true;
bool expected = true;
std::string description = "", cause = "";
switch (result.getResultType()) {
case ResultWas::Ok: assert(0 && "should not happen");
case ResultWas::ExpressionFailed:
if (as.infoMessages.size() == 1) {
cause = "expression failed with message";
} else if (as.infoMessages.size() > 1) {
cause = "expression failed with messages";
} else {
cause = "expression did not evaluate to true";
}
break;
case ResultWas::ExplicitFailure:
cause = "failed";
break;
case ResultWas::DidntThrowException:
cause = "the expected expression was not thrown";
break;
case ResultWas::ThrewException:
cause = "an unexpected exception was thrown";
expected = false;
break;
case ResultWas::FatalErrorCondition:
cause = "an unexpected error occured";
expected = false;
break;
case ResultWas::Info:
cause = "info";
break;
case ResultWas::Warning:
cause = "warning";
break;
case ResultWas::Unknown:
case ResultWas::FailureBit:
case ResultWas::Exception:
cause = "internal error";
expected = false;
break;
}
if (result.hasExpression()) {
description.append(result.getExpression());
if (result.hasExpandedExpression())
description.append(" (expands to: " + result.getExpandedExpression() + ")");
}
description.append(": " + cause);
if (!as.infoMessages.size())
as.infoMessages.emplace_back((result.hasMessage() ? result.getMessage() : ""), result.getSourceInfo(), result.getResultType());
for (auto&& info : as.infoMessages) {
auto msg = description;
if (info.message.size())
msg.append(": " + info.message);
if (msg[msg.size() - 1] != '.')
msg.append(".");
NSString* desc = [NSString stringWithUTF8String:msg.c_str()];
NSString* file = [NSString stringWithUTF8String:info.lineInfo.file];
[xcTestCase recordFailureWithDescription:desc inFile:file atLine:info.lineInfo.line expected:expected];
}
return true;
}
virtual ReporterPreferences getPreferences() const {
ReporterPreferences prefs;
prefs.shouldRedirectStdOut = false;
return prefs;
}
};
class XCTestRegistryHub : public IMutableRegistryHub, IMutableEnumValuesRegistry {
public:
XCTestRegistryHub () {}
virtual void registerReporter( std::string const&, IReporterFactoryPtr const& ) {}
virtual void registerListener( IReporterFactoryPtr const& ) {}
virtual void registerTranslator( const IExceptionTranslator* ) {}
virtual void registerTagAlias( std::string const&, std::string const&, SourceLineInfo const& ) {};
virtual void registerStartupException() noexcept {}
virtual IMutableEnumValuesRegistry& getMutableEnumValuesRegistry() { return *this; }
virtual Detail::EnumInfo const& registerEnum( StringRef, StringRef, std::vector<int> const& ) {
static struct Detail::EnumInfo info;
return info;
};
virtual void registerTest( TestCase const& testInfo ) {
// Running inside Xcode
if ([[[NSProcessInfo processInfo] arguments][0] containsString:@"Xcode/Agents/xctest"]) {
// Skip test if there's a forks tag
// Tests run in Xcode doens't like fork
for (const auto &tag : testInfo.tags) if (tag == "forks") return;
}
// at some point clang will get gcc's -fmacro-prefix-map compiler option that can be also used instead of FILE_PREFIX
NSString* filename = [[NSString stringWithUTF8String:testInfo.lineInfo.file] stringByReplacingOccurrencesOfString:@FILE_PREFIX withString:@""];
NSString* clsName = (testInfo.className.size() ? [NSString stringWithUTF8String:testInfo.className.c_str()] : filename);
clsName = [clsName stringByReplacingOccurrencesOfString:@"." withString:@"․"]; // Xcode doesn't like dots
Class cls;
if (!(cls = NSClassFromString(clsName))) {
cls = objc_allocateClassPair([XCTestCase class], [clsName UTF8String], 0);
assert(cls);
// +defaultTestSuite
[CatchXCTest storeSuite:[XCTestSuite testSuiteWithName:clsName] forClass:cls];
Method m = class_getInstanceMethod([[XCTestCase class] class], @selector(defaultTestSuite));
class_addMethod(object_getClass(cls), @selector(defaultTestSuite), (IMP)_methodImpl_defaultTestSuite, method_getTypeEncoding(m));
// -name
m = class_getInstanceMethod([XCTestCase class], @selector(name));
class_addMethod(cls, @selector(name), (IMP)_methodImpl_name, method_getTypeEncoding(m));
objc_registerClassPair(cls);
}
// -xsmTestSection
SEL testSel = NSSelectorFromString([[NSString stringWithUTF8String:testInfo.name.c_str()] stringByReplacingOccurrencesOfString:@"." withString:@"․"]);
NSString *typeEnc = [NSString stringWithFormat:@"%s%s%s", @encode(void), @encode(id), @encode(SEL)];
class_addMethod(cls, testSel, (IMP)_methodImpl_runTest, [typeEnc UTF8String]);
XCTestCase *tc = [(XCTestCase*)[cls alloc] initWithSelector:testSel];
[[cls defaultTestSuite] addTest:tc];
[CatchXCTest storeTestCase:testInfo forSelector:testSel];
}
private:
static XCTestSuite *_methodImpl_defaultTestSuite(Class self, SEL) {
return [CatchXCTest testSuiteForClass:self];
}
static NSString *_methodImpl_name(XCTestCase *self, SEL) {
auto &testInfo = [CatchXCTest testCaseForSelector:self.invocation.selector];
return [NSString stringWithUTF8String:testInfo.name.c_str()];
}
static void _methodImpl_runTest(XCTestCase *self, SEL) {
auto &testInfo = [CatchXCTest testCaseForSelector:self.invocation.selector];
ConfigData data;
data.testsOrTags.push_back(testInfo.name);
std::shared_ptr<IConfig> config(new Config(data));
std::unique_ptr<XCTestReporter> reporter(new XCTestReporter(ReporterConfig(config)));
reporter->xcTestCase = self;
RunContext runner(config, std::move(reporter));
Totals totals;
runner.testGroupStarting(testInfo.name, 1, 1);
totals += runner.runTest(testInfo);
runner.testGroupEnded(testInfo.name, totals, 1, 1);
}
};
}
@implementation CatchXCTest {
Catch::TestCase *ptr;
}
static NSMutableDictionary *CatchXCTestDict;
+ (XCTestSuite*)testSuiteForClass:(Class)klass {
return CatchXCTestDict[klass];
}
+ (void)storeSuite:(XCTestSuite*)suite forClass:(Class)klass {
if (!CatchXCTestDict) CatchXCTestDict = [NSMutableDictionary new];
CatchXCTestDict[klass] = suite;
}
+ (Catch::TestCase&)testCaseForSelector:(SEL)selector {
return *((CatchXCTest*)CatchXCTestDict[NSStringFromSelector(selector)])->ptr;
}
+ (void)storeTestCase:(const Catch::TestCase&)test forSelector:(SEL)selector {
if (!CatchXCTestDict) CatchXCTestDict = [NSMutableDictionary new];
CatchXCTestDict[NSStringFromSelector(selector)] = [CatchXCTest withPointer:new Catch::TestCase(test)];
}
+ (CatchXCTest*)withPointer:(Catch::TestCase*)ptr {
CatchXCTest *c = [self new];
c->ptr = ptr;
return c;
}
- (void)dealloc {
delete ptr;
}
+ (void)load {
auto &hub = Catch::getMutableRegistryHub();
new (&hub) Catch::XCTestRegistryHub();
}
@end |
|
If you want to integrate the above with catch2's ctest integration, you can compile following binary: #import <Foundation/Foundation.h>
#import <XCTest/XCTest.h>
#include <err.h>
static XCTest*
find_test(NSString *name, XCTestSuite *suite) {
for (XCTest *test in suite.tests) {
XCTest *tc;
if ([test isKindOfClass:[XCTestSuite class]] && (tc = find_test(name, (XCTestSuite*)test)))
return tc;
if (![test.name isEqualToString:name])
continue;
return test;
}
return nil;
}
int
main(int argc, const char *argv[]) {
if (argc < 2) errx(EXIT_FAILURE, "usage: xctest.bundle [test]");
@autoreleasepool {
NSString *path = [NSString stringWithUTF8String:argv[1]];
for (int i = 0; i < 4 && ![[path lastPathComponent] containsString:@".xctest"]; ++i)
path = [path stringByDeletingLastPathComponent];
if (![[NSBundle bundleWithPath:path] load])
errx(EXIT_FAILURE, "couldn't load: %s\n", argv[1]);
XCTest *tc = [XCTestSuite defaultTestSuite];
if (argc > 2) {
// Not sure why catch2's cmake escapes these..
if (!(tc = find_test([[NSString stringWithUTF8String:argv[2]] stringByReplacingOccurrencesOfString:@"\\," withString:@","], (XCTestSuite*)tc)))
errx(EXIT_FAILURE, "couldn't find test: %s\n", argv[2]);
}
[tc runTest];
if (!tc.testRun.testCaseCount) return 1;
return !tc.testRun.hasSucceeded;
}
}And do something like this in cmake: find_package(XCTest REQUIRED)
add_executable(xctest xctest.m)
target_link_libraries(xctest ${XCTest_LIBRARIES})
target_include_directories(xctest PRIVATE ${XCTest_INCLUDE_DIRS})
xctest_add_bundle(catch2 testee main.mm ${tests_src})
add_dependencies(catch2 xctest)
include(ParseAndAddCatchTests)
set(PARSE_CATCH_TESTS_ADD_TO_CONFIGURE_DEPENDS ON)
set(PARSE_CATCH_TESTS_ADD_TARGET_IN_TEST_NAME OFF)
set(OptionalCatchTestLauncher $<TARGET_FILE:xctest>)
ParseAndAddCatchTests(catch2) |
|
The above also will work for Write this wrapper shell script (assumes cmake again): #!/bin/sh
# Magic to run iphonesimulator platform unit tests without simulator
bin="$1"; test="$2"; shift 2
EFFECTIVE_PLATFORM_NAME=-iphonesimulator
bin=$(eval "printf -- '%s' $bin") # unwrap EFFECTIVE_PLATFORM_NAME
test=$(eval "printf -- '%s' $test") # unwrap EFFECTIVE_PLATFORM_NAME
sdkroot=$(xcrun --sdk iphoneos --show-sdk-path)
DYLD_ROOT_PATH="$sdkroot/../../../Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot" \
"$bin" "$test" "$@"And in cmake slightly change the build options: set_xcode_property(catch2 FRAMEWORK_SEARCH_PATHS "$(SDKROOT)/../../Library/Frameworks")
set_xcode_property(xctest FRAMEWORK_SEARCH_PATHS "$(SDKROOT)/../../Library/Frameworks")
target_link_options(xctest PRIVATE -rpath $(SDKROOT)/../../Library/Frameworks)
set(OptionalCatchTestLauncher sh "${CMAKE_CURRENT_SOURCE_DIR}/xctest.sim.sh" $<TARGET_FILE:xctest>)The nice side effect of this is that no simulator will be launched at all. |
|
@philsquared @horenmar is there any currently supported way to run Catch2 tests when cross-compiling for iOS? My codebase is C++, not Objective-C, but Apple's SDKs don't allow you to build regular command line executables for iOS. CMake offers some XCTest integration, but it would be great if this could be handled by Catch. |
|
@benthevining, I know this is an old thread but this seems to be an under documented subject and I think I've got something somewhat working. |
|
Looks awesome! I think this can be done from CMake using a crosscompiling emulator script (CMAKE_CROSSCOMPILING_EMULATOR) |
This patch adds support for XCTest integration by inverting the usual Catch behavior; rather than using Catch as the test runner, we dynamically register XCTestCase classes with the Objective-C runtime to allow Xcode's test runner to execute (and report on) Catch-defined test cases.
To use this implementation, a project need only include "XCTestRunner.mm" in their unit test build -- the
+[XCTestCaseCatchRegistry load]method will insert an XCTestRegistryHub instance that automatically registers XCTestCase classes for any Catch test cases defined within the image.+loadmethods by definition will run prior to C++ static initializers in the same image, ensuring that we insert our registry prior to any TestCase instances registering themselves.Currently, our granularity is limited to test cases -- we can't report on section execution without first registering the section as a test instance in the XCTestCase, and this would require static access to the list of sections/subsections defined within a TestCase or Section.
This approach is based on the XCTest registration code I wrote for XSmallTest: https://github.com/landonf/XSmallTest