Add darwin-aarch64 support

[tresf]

Since the announcement of Apple Silicon (arm64/aarch64), hid4java (and it's dependent library, JNA) doesn't have native support and can't be used unless run through emulation using Rosetta 2.

Note, this topic was split off from #93, some historical context may be there.

Related upstream java-native-access/jna#1238.

At the time of posting this, there are two aarch64 JDKs available natively for macOS Apple Silicon:

• Azul: https://www.azul.com/downloads/zulu-community/?os=macos&architecture=arm-64-bit&package=jdk
• Microsoft (that's not a typo): https://github.com/microsoft/openjdk-aarch64/releases/tag/16-ea%2B10-macos

[andriy-gerasika]

update: JNA 5.7 w/ Mac M1 support is out

[gary-rowe]

Thanks - I'll look into incorporating this ASAP.

[tresf]

@gary-rowe friendly bump. 🍻

[martinpaljak]

Any changes on this? Looking at last commits, all happened in 2020 (2 years ago), are there more recent versions of this project somewhere?

[tresf]

@martinpaljak the following has happened since an update was provided...

• JNA (as well as FFI) have accepted the aarch64 code changes as mentioned in this comment: Add darwin-aarch64 support  #108 (comment).
• Using the PR at Improvements to MacOS builds #116, I've built my own SNAPSHOT binary. A mirror of it is here if interested.

Some notes about the PR:

• It builds upon -- and thus includes -- the code for Windows ARM64 support (Add win32-aarch64 support #93), so it has commits in it that may seem out of place.
• It assumes JNA 5.8, however any newer version of JNA will suffice
• Getting these changes into JNA required convincing the JNA team that MacOS binaries should NOT be fat (multi-arch) binaries but rather use the same arch suffix that other platforms use (backwards compatibility with fat binaries offered so everything doesn't break).

  - # darwin - OS X 64-bit
  + # darwin-x86-64 - OS X 64-bit
  + # darwin-aarch64 - OS X 64-bit ARM

In regards to the reliability of the SNAPSHOT build, due to the slow adoption of native M1 apps, I can't offer that, but I can say that the native M1 version of the software that I use it with has been downloaded about 6,000 times since we've added it.

[martinpaljak]

Right. If there's a branch to build, I'd use it, maybe even prebuilt. I also thought of trying the "pure" java hidapi but that seems too much lower towards HID internals than I'm wishing to dive (To fix martinpaljak/FIDO2#8). Should we establish a newer branch of hid4java somewhere?

[tresf]

The #116 PR branch should build just fine if you have a macOS system to tinker with. Alternately, the snapshot linked in my previous comment has this precompiled.

I use both the purjavahidapi as well as this project (wrote a wrapper) to be able to switch in the event of a critical bug. This one (gary-rowe) is my recommendation due to better feature parity.

I'm in communication with Gary here too. He has every intention of merging these enhancements. If you'd prefer to operate from an official fork, I can do that too, but I have very limited time for this due to personal reasons, so I'd need a co-maintainer.

[gary-rowe]

Just to chime in with some good news. I've finally managed to get hold of an M1 machine for compiling new versions of the library. I'm still super busy with unrelated projects but I'm trying to prioritise time for hid4java to get another release out with as many bug fixes as I can.

[DaPutzy]

• Using the PR at Improvements to MacOS builds #116

I can not get anything to work with that branch:

Exception in thread "main" org.hid4java.HidException: Hidapi did not initialise: Unable to load library 'hidapi':
dlopen(libhidapi.dylib, 0x0009): tried: ..... libhidapi.dylib' (no such file)
dlopen(libhidapi.dylib, 0x0009): tried: ..... libhidapi.dylib' (no such file)
dlopen(.....): tried: .....
	at org.hid4java.HidDeviceManager.<init>(HidDeviceManager.java:87)
	at org.hid4java.HidServices.<init>(HidServices.java:81)
	at org.hid4java.HidManager.getHidServices(HidManager.java:72)
	at org.hid4java.examples.UsbHidEnumerationExample.executeExample(UsbHidEnumerationExample.java:54)
	at org.hid4java.examples.UsbHidEnumerationExample.main(UsbHidEnumerationExample.java:40)
Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'hidapi':
dlopen(libhidapi.dylib, 0x0009): tried: ..... libhidapi.dylib' (no such file)
dlopen(libhidapi.dylib, 0x0009): tried: ..... libhidapi.dylib' (no such file)
dlopen(.....): tried: .....
	at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:301)
	at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:461)
	at com.sun.jna.Library$Handler.<init>(Library.java:192)
	at com.sun.jna.Native.load(Native.java:596)
	at com.sun.jna.Native.load(Native.java:570)
	at org.hid4java.jna.HidrawHidApiLibrary.<clinit>(HidrawHidApiLibrary.java:37)
	at org.hid4java.jna.HidApi.init(HidApi.java:114)
	at org.hid4java.HidDeviceManager.<init>(HidDeviceManager.java:84)
	... 4 more
	Suppressed: java.lang.UnsatisfiedLinkError: dlopen(.....)
		at com.sun.jna.Native.open(Native Method)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
		... 11 more
	Suppressed: java.lang.UnsatisfiedLinkError: dlopen(.....)
		at com.sun.jna.Native.open(Native Method)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
		... 11 more
	Suppressed: java.lang.UnsatisfiedLinkError: dlopen(.....)
		at com.sun.jna.Native.open(Native Method)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:248)
		... 11 more
	Suppressed: java.lang.UnsatisfiedLinkError: dlopen(.....)
		at com.sun.jna.Native.open(Native Method)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:248)
		... 11 more
	Suppressed: java.lang.UnsatisfiedLinkError: dlopen(.....)
		at com.sun.jna.Native.open(Native Method)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:248)
		... 11 more
	Suppressed: java.io.IOException: Native library (darwin-aarch64/libhidapi.dylib) not found in resource path .....
		at com.sun.jna.Native.extractFromResourcePath(Native.java:1119)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:275)
		... 11 more

What am i missing here? o.O

[tresf]

Hmm... assuming you're on a version of JNA that supports the darwin-aarch64 resource path (the logs suggest you are), it should work. Did you confirm that the build system built a libhidapi.dylib file that shows as ARM64 when you do file libhidapi.dylib ?

[DaPutzy]

I just figured out that the PR does not contain the actual files and they have to be build first. To save me the trouble of doing that i just stole them from the SNAPSHOT jar you linked and now it works as intended!

[tresf]

I just figured out that the PR does not contain the actual files and they have to be build first. To save me the trouble of doing that i just stole them from the SNAPSHOT jar you linked and now it works as intended!

That's correct. For security reasons, @gary-rowe prefers to build his own binaries, so they were omitted from the PR.

[martinpaljak]

Is it possible to set up github action to deploy snapshots of what you are talking about? Or do any of you publish those snapshots? Anything else I could do to help bring this forward (I'd think testing would be one thing, but the whole docker stuff setup seems a bit too much for my skill level)

[DaPutzy]

Is it possible to set up github action to deploy snapshots of what you are talking about?

AFAIK github actions are not capable of that.

Or do any of you publish those snapshots?

tresf has here.

Anything else I could do to help bring this forward (I'd think testing would be one thing, but the whole docker stuff setup seems a bit too much for my skill level)

I dont think so.

We are just waiting for gary-rowe to get his ass up. /s

[tresf]

the whole docker stuff setup seems a bit too much for my skill level

It's a bit to setup and when things go wrong it's not immediately obvious how to fix them. Some environments just don't work at all but I digress. For MacOS, it's actually not using docker at all, but rather building in an assumed directory structure. This means binaries are easier to produce, but it's much less obvious as to the backwards compatibility they'll yield. This is probably worth putting together a YouTube tutorial for as much of the details have since slipped my mind.

Is it possible to set up github action to deploy snapshots of what you are talking about?

GitHub actions can probably do it. Not sure if Gary would entertain this, he's historically always built the artifacts on his own machine for security reasons.

[DaPutzy]

GitHub actions can probably do it. Not sure if Gary would entertain this, he's historically always built the artifacts on his own machine for security reasons.

Can you build a darwin-aarch64 binary on anything other than the actual hardware?
The only way I can see this working with GitHub Actions would be with a self-hosted runner 🤔

[tresf]

Can you build a darwin-aarch64 binary on anything other than the actual hardware?

Yes you can. According to Apple, XCode 12 or higher can produce darwin-aarch64 binaries. The unit tests will need to be skipped. According to my own notes in #120, it's the "11.0 SDK" that's bundled with XCode 12.

The only way I can see this working with GitHub Actions would be with a self-hosted runner 🤔

I haven't worked a lot with GitHub actions , but artifact->release should be entirely possible. Here's one example. This doesn't solve the rube-goldberg problem of the fat jars containing all platforms and architectures though, so I'm uncertain as to its real-world usefulness.

This means binaries are easier to produce, but it's much less obvious as to the backwards compatibility they'll yield.

Slight update, it appears the #116 PR addresses that as well, quoting:

• e40bdfe - Add hints for SDKROOT, MACOSX_DEPLOYMENT_TARGET (Closes Building for MacOS should encourage setting an SDK version #120)

[DaPutzy]

Can you build a darwin-aarch64 binary on anything other than the actual hardware?
The only way I can see this working with GitHub Actions would be with a self-hosted runner 🤔

I haven't worked a lot with GitHub actions , but artifact->release should be entirely possible. Here's one example. This doesn't solve the rube-goldberg problem of the fat jars containing all platforms and architectures though, so I'm uncertain as to its real-world usefulness.

That was more of a combined statement :) If you can build both darwin artifacts on x86 than the GitHub-hosted runners should be fine. Releasing should not be an issue, i am sure there is a way to collect multiple artifacts and release them at last as one fat jar.

Would probably require quite some work though and i have not yet worked with GitHub Actions at all.

[tresf]

This doesn't solve the rube-goldberg problem of the fat jars containing all platforms and architectures though, so I'm uncertain as to its real-world usefulness.

you can build both darwin artifacts on x86 than the GitHub-hosted runners should be fine. Releasing should not be an issue, i am sure there is a way to collect multiple artifacts and release them at last as one fat jar.

Absolutely. Here's a JNI project that's done it, if interested. java-native/jssc#102

[DaPutzy]

Absolutely. Here's a JNI project that's done it, if interested. java-native/jssc#102

Very interesting read, thanks!

[gary-rowe]

Ongoing support for this issue will be the develop branch as #108 has a working darwin-aarch64 dylib in place. To avoid clutter I'll close this issue now.