Deferred compilation on Android #4027

kraenhansen · 2021-10-14T11:09:43Z

What, How & Why?

The React Native project doesn't promise ABI / API stability of any shared object libraries shipped via the React Native package. In addition to this the end-user determines the NDK (hence the shared C++ STL) used when building the their app.

In principle this means that we can only publish a prebuilt shared object library file under the following assumptions:

Our prebuilt binary is installed alongside a specific subset of React Native versions (any version ABI compatible with the version we used when compiling our prebuilt binary).
The end-users app is compiled with the same NDK version that we used when building our prebuilt binary. This ensures ABI stability with the shared object C++ STL shipped with the app.

In practice however:

The breakage of the ABI might happen on the next release or ten releases from now, nobody really knows. We might be putting in a lot of work to build resilience towards a scenario that in practice rarely happens. Historically we've been "lucky" that we've mostly been integrating with ABIs of the Android platform and haven't had deep integrations with ABIs exposed by React Native. With the upcoming changes to support Hermes via JSI, we are not allowed that luxury.
We can always upgrade the React Native version, recompile and publish a new version of our library as a version of React Native ships with an ABI breaking change. If this happens a lot, it would be a bummer for our users (and our documentation and support crew) that the have to constantly know (possibly by NPM peerDepenency specifications or an NPM post-install script) what versions of Realm JS are compatible with which versions of React Native.

The goal of this PR is to increase resilience towards breaking ABI & API changes of the React Native JavaScript Interface (JSI) on Android. I'm putting this into draft for the time being, as it does complicated our build process and developer experience significantly and puts extra requirements on the end-users setup:

An expected version CMake via the SDK manager.
~~Building the app with an alpha release of the Android Gradle plugin. Although I still need to verify that this is the case when building Realm Core from prebuilt binaries.~~ This doesn't seem to be an issue anymore (see "Bump Gradle version to 7.2, Bump Kotlin version to 1.5.31 (9ae3367431 by @svbutko)" in the React Native Changelog)

☑️ ToDos

📝 Changelog entry
📝 Compatibility label is updated or copied from previous entry
🚦 Tests
📝 Public documentation PR created or is not necessary
💥 Breaking label has been applied or is not necessary

If this PR adds or changes public API's:

typescript definitions file is updated
jsdoc files updated
Chrome debug API is updated if API is available on React Native

This script is no longer needed as we're deferring compilation.

fealebenpae

The motivation for defending against ABI breakage is sound. This was an amazingly annoying problem for Node.js pre-NAPI and here we don't even have the infrastructure like nan and node-pre-gyp to soften the blow.

However, one thing to note is that in order for us to be the most resilient against ABI breakage we can't rely on prebuilt Core binaries. There's simply no guarantee that a hypothetical future React Native release just won't pull in a different NDK version than the one Core used at the time of making a release. We might need to look into building everything on the user's machine, which might be just too annoying for the user. And even then we wouldn't have a hard guarantee that the user's environment is configured with the same NDK that Facebook used when making a specific React Native release. We might just need to hope that ABI changes in libc++ between NDK versions are more rare than in JSI.

I realize why deferring compilation is important, but I'm still saddened that we effectively have to introduce a regression in our build system to support it. We won't be able to utilize link-time optimization to make our .so smaller. And on iOS we'll need to again support Xcode and CocoaPods as build systems of the actual SDK.

fealebenpae · 2021-10-14T14:54:39Z

CMakeLists.txt

@@ -39,6 +39,21 @@ set(CMAKE_MSVC_RUNTIME_LIBRARY "MultiThreaded$<$<CONFIG:Debug>:Debug>")

 list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")

+set(PACKAGE_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR})


I think this is just equivalent to CMAKE_SOURCE_DIR so we can just use that instead of PACKAGE_ROOT_DIR.

fealebenpae · 2021-10-14T14:56:26Z

CMakeLists.txt

+
+    set(REALM_CORE_PLATFORM Android-${ANDROID_ABI})
+    # TODO: Revising the "devel" part of the following line
+    set(REALM_CORE_FILENAME realm-${CMAKE_BUILD_TYPE}-v${REALM_CORE_VERSION}-${REALM_CORE_PLATFORM}-devel)


I think we might want to always use Release here, instead of using CMAKE_BUILD_TYPE, because we might not want debug builds of user applications to bring in the debug build of Core.

fealebenpae · 2021-10-14T15:07:11Z

CMakeLists.txt

+    else()
+        set(REALM_CORE_ARCHIVE_PATH ${CMAKE_BINARY_DIR}/${REALM_CORE_FILENAME}.tar.gz)
+        message(STATUS "Downloading Realm Core prebuilt binaries")
+        file(DOWNLOAD "https://static.realm.io/downloads/core/${REALM_CORE_FILENAME}.tar.gz" "${REALM_CORE_ARCHIVE_PATH}" STATUS DOWNLOAD_STATUS)


Suggested change

file(DOWNLOAD "https://static.realm.io/downloads/core/${REALM_CORE_FILENAME}.tar.gz" "${REALM_CORE_ARCHIVE_PATH}" STATUS DOWNLOAD_STATUS)

file(DOWNLOAD "https://static.realm.io/downloads/core/v${REALM_CORE_VERSION}/android/${ANDROID_ABI}/Release/${REALM_CORE_FILENAME}.tar.gz" "${REALM_CORE_ARCHIVE_PATH}" STATUS DOWNLOAD_STATUS)

We currently publish to both locations now, but the previous one is effectively deprecated and at some point we'll just stop publishing there, because dumping a bunch of files every release in the same folder over years doesn't scale.

fealebenpae · 2021-10-14T15:18:24Z

Jenkinsfile

@@ -92,7 +92,6 @@ stage('build') {
    parallelExecutors["Windows ia32 NAPI ${nodeTestVersion}"] = buildWindows(nodeTestVersion, 'ia32')
    parallelExecutors["Windows x64 NAPI ${nodeTestVersion}"] = buildWindows(nodeTestVersion, 'x64')

-    parallelExecutors["Android RN"] = buildAndroid()


It might still be nice to have a CI task that builds the native libraries for all ABIs just to make sure they compile and link on every platform, not just the one we run tests on during CI. I would keep the Android build paths in Jenkins and Actions.

fealebenpae · 2021-10-14T15:21:50Z

react-native/android/build.gradle

+                arguments '-DANDROID_STL=c++_shared', '-DREALM_JS_BUILD_CORE_FROM_SOURCE=off'
+                targets 'realm-js-android'
+                cppFlags ''
+                abiFilters 'x86', 'x86_64'


Maybe add a TODO so we don't forget about these?

fealebenpae · 2021-10-14T15:53:03Z

src/android/CMakeLists.txt

@@ -1,3 +1,10 @@
+cmake_minimum_required(VERSION 3.18.1)


This needs to be called in the top-level CMakeLists before the project() command, otherwise it means nothing.

kraenhansen · 2024-06-10T09:29:32Z

Closing this as #6650 is realizing the same goal 🎉

kraenhansen added 3 commits October 14, 2021 10:25

Adopting to changes in Realm Core

7f99ce7

WIP

41f70ee

Removed all mentions of the build-android.js script.

a542a56

This script is no longer needed as we're deferring compilation.

kraenhansen self-assigned this Oct 14, 2021

cla-bot bot added the cla: yes label Oct 14, 2021

fealebenpae reviewed Oct 14, 2021

View reviewed changes

kraenhansen mentioned this pull request May 18, 2022

Upgrade integration-tests/environment/react-native to React Native 0.68.2 #4583

Merged

7 tasks

kraenhansen mentioned this pull request Jun 17, 2022

Upgrade to NDK r23c #4628

Closed

1 task

kraenhansen mentioned this pull request Jul 11, 2022

Feedback on Hermes support #3940

Closed

kneth mentioned this pull request Mar 30, 2023

[Solved] Error: Exception in HostObject::get for prop 'Realm': java.lang.UnsatisfiedLinkError: couldn't find DSO to load: librealm.so caused by: dlopen failed: cannot locate symbol "__emutls_get_address" #5361

Closed

kraenhansen closed this Jun 10, 2024

kraenhansen deleted the kh/deferred-android-compilation branch June 10, 2024 09:29

kraenhansen mentioned this pull request Jun 10, 2024

RJS-2101: "Build optimizations" for iOS and Android #6650

Merged

9 tasks

github-actions bot locked as resolved and limited conversation to collaborators Jul 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deferred compilation on Android #4027

Deferred compilation on Android #4027

kraenhansen commented Oct 14, 2021 •

edited

Loading

fealebenpae left a comment

fealebenpae Oct 14, 2021

fealebenpae Oct 14, 2021

fealebenpae Oct 14, 2021

fealebenpae Oct 14, 2021

fealebenpae Oct 14, 2021

fealebenpae Oct 14, 2021

kraenhansen commented Jun 10, 2024

		@@ -39,6 +39,21 @@ set(CMAKE_MSVC_RUNTIME_LIBRARY "MultiThreaded$<$<CONFIG:Debug>:Debug>")

		list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")

		set(PACKAGE_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR})

	file(DOWNLOAD "https://static.realm.io/downloads/core/${REALM_CORE_FILENAME}.tar.gz" "${REALM_CORE_ARCHIVE_PATH}" STATUS DOWNLOAD_STATUS)
	file(DOWNLOAD "https://static.realm.io/downloads/core/v${REALM_CORE_VERSION}/android/${ANDROID_ABI}/Release/${REALM_CORE_FILENAME}.tar.gz" "${REALM_CORE_ARCHIVE_PATH}" STATUS DOWNLOAD_STATUS)

Deferred compilation on Android #4027

Deferred compilation on Android #4027

Conversation

kraenhansen commented Oct 14, 2021 • edited Loading

What, How & Why?

☑️ ToDos

fealebenpae left a comment

Choose a reason for hiding this comment

fealebenpae Oct 14, 2021

Choose a reason for hiding this comment

fealebenpae Oct 14, 2021

Choose a reason for hiding this comment

fealebenpae Oct 14, 2021

Choose a reason for hiding this comment

fealebenpae Oct 14, 2021

Choose a reason for hiding this comment

fealebenpae Oct 14, 2021

Choose a reason for hiding this comment

fealebenpae Oct 14, 2021

Choose a reason for hiding this comment

kraenhansen commented Jun 10, 2024

kraenhansen commented Oct 14, 2021 •

edited

Loading