macOS packaging for beats #6344

adriansr · 2018-02-09T15:23:23Z

Added support to create signed PKG installers, bundled in a DMG file with a rudimentary uninstaller in it.

~~Currently disconnected from the packaging flow, must be run by hand after make package~~

Will be run by dev-tools/deploy given the environment variables KEYCHAIN and KEYCHAN_PASS point to a keychain file with the code-signing certificates and its password, respectively.

andrewkroh · 2018-02-09T15:50:47Z

dev-tools/packer/platforms/darwin/pkg/templates/launchd-daemon.plist.j2

+            <string>--path.config</string>
+            <string>/etc/{{.beat_name}}</string>
+            <string>--path.data</string>
+            <string>{{.install_path}}/{{.beat_vendor}}/{{.beat_name}}</string>


Would it be OK to set path.data to /var/lib/{{.beat_name}} similar to Linux?

beats/dev-tools/packer/platforms/centos/beatname.sh.j2

Line 9 in 545938e

-path.data /var/lib/{{.beat_name}} \

andrewkroh · 2018-02-09T15:58:00Z

dev-tools/packer/platforms/darwin/pkg/internal_build.sh

+    -srcfolder dmg \
+    -ov \
+    "$DMG_NAME"
+


Should we sign the the dmg too? (e.g. codesign -s <identity> <disk-image>)

Sure. I didn't know signing DMGs was a thing.

adriansr · 2018-02-16T12:43:47Z

Pending tasks:

Add KEYCHAIN* parameters to jenkins jobs (elastic/infra#3938)
do not add icon to DMG. It only works locally.
Add icon to uninstaller (artwork pending).
Include uninstaller along the beat binaries.
preference pane.

andrewkroh · 2018-02-19T14:21:16Z

This looks really good!

Include uninstaller along the beat binaries.

What do you think about creating an /Applications/Elastic directory that contains an "Uninstall Filebeat" link to the uninstall app?

Is there anything I can help with?

I'm thinking the next step should be to get this building as part of https://beats-ci.elastic.co/job/elastic+beats+master+package/. We can get the key information stored into Vault then load it from there. This should make us a bit more prepared to integrate with the Release Manager.

adriansr · 2018-02-19T21:10:52Z

What do you think about creating an /Applications/Elastic directory that contains an "Uninstall Filebeat" link to the uninstall app?

My plan was to add an Uninstall button to each beat in the Preference Pane.

I'm thinking the next step should be to get this building as part of https://beats-ci.elastic.co/job/elastic+beats+master+package/. We can get the key information stored into Vault then load it from there. This should make us a bit more prepared to integrate with the Release Manager.

Yes, that is already in progress. The certificates are deployed to the macOS images and the password is in Vault, now I'm waiting for infra to make the password and certificate path available to the build job via env vars (I added a link to the issue above).

adriansr · 2018-02-20T10:08:03Z

jenkins, test it

adriansr · 2018-02-20T11:15:28Z

jenkins, package it

andrewkroh · 2018-02-21T16:37:11Z

dev-tools/packer/platforms/darwin/build.sh

@@ -17,3 +17,6 @@ docker run --rm -v ${BUILD_DIR}:/build \
    tudorg/fpm /build/run-$runid.sh

 rm ${BUILD_DIR}/settings-$runid.yml ${BUILD_DIR}/run-$runid.sh
+
+echo 'Running macOS packager'
+${BASEDIR}/pkg/build.sh


I think there should be a conditional preventing this from error'ing out if we aren't on Darwin.

Or it might be better to separate this into it's own build target such that you can run make packages on any machine with Docker and can run make macos-dmgs (naming TBD) afterwards to build the the DMGs (and it only works on macOS).

adriansr · 2018-03-26T09:41:20Z

jenkins, test it

andrewkroh · 2018-03-27T22:28:06Z

dev-tools/osx-deploy

+def main():
+    parser = argparse.ArgumentParser(
+        description="Builds OSX packages for Beats")
+    args = parser.parse_args()


Probably this should have the same option as dev-tools/deploy?

beats/dev-tools/deploy

Lines 9 to 10 in 96b5aee

parser.add_argument("--no-snapshot", action="store_true",

help="Don't append -SNAPSHOT to the version.")

andrewkroh

LGTM. I think you should probably squash this one down yourself first in order to cleanup the commit message.

andrewkroh

LGTM. Can you please squash.

andrewkroh

Wrong PR ~~I built (unsigned) and installed BeatsPrefPane.pkg locally. It all went well. I'll test again after having some Beat packages installed too.~~

adriansr · 2018-05-31T12:40:12Z

~~Please do not merge until further notice~~

This adds a new make target, `osx-package`, that will build a PKG installer for each beat supported in macOS. Also features a DMG file wrapping the installer and uninstaller. Installer and binaries need to be signed. Valid Apple-issued certificates and their associated keys need to be available in an unlocked keychain: - `Developer ID application` certificate, for code-signing. - `Developer ID installer` certificate, for signing .pkg installers.

The installer package includes the Beats preference pane and installs it along with the Beat.

Now `dev-tools/osx-deploy` builds two sets of packages per Beat: - beatname-oss-version.{pkg,dmg} packages without X-Pack content. - beatname-version.{pkg,dmg} packages with X-Pack content The main Makefile target has been renamed to `osx-package-all` to match the existing `package-all` target. From inside a Beat it is possible to use the targets `osx-package-oss` and `osx-package-elastic` to build only one set of packages.

When Docker is run in Mac under Virtualbox, volumes in docker don't have the expected uid=0,gid=0 permissions, but those that Docker is using inside it's VM (uid=1000(docker) gid=50(staff). This caused files in the TAR archives to have wrong permissions, triggering a packaging test error later on.

adriansr · 2018-06-19T05:45:02Z

Ready to rebase and merge

andrewkroh · 2018-06-30T05:59:04Z

@adriansr With KEYCHAIN (and KEYCHAIN_PASS) not being used, can they be removed from the Jenkins job configuration?

It looks like Ansible installs the keychain-db file to Jenkins, but does not add it to the keychain search list or unlock it. Was that done manually or am I missing something?

It looks like Ansible could be made do the following and this would put the worker in a ready state. WDYT?

security list-keychains -s <keychain>
security -v unlock-keychain <keychain> < vault read -field=password

adriansr · 2018-07-02T14:55:40Z

@andrewkroh

The problem that prompted me to get rid of KEYCHAIN is that, in macOS 10.12 and later, there is no reliable way of handling keychains from launchd services, due to issues with session isolation.

The only reliable way to perform the code-signing from Jenkins is to copy the certificates into the System keychain. So we will need Ansible to do something like this https://apple.stackexchange.com/questions/80623/import-certificates-into-the-system-keychain-via-the-command-line

andrewkroh · 2018-07-03T15:34:34Z

Added via #7481.

adriansr added enhancement in progress Pull request is currently in progress. discuss Issue needs further discussion. review Packaging labels Feb 9, 2018

andrewkroh reviewed Feb 9, 2018

View reviewed changes

ph mentioned this pull request Feb 12, 2018

Update to Golang 1.9.4 #6326

Merged

andrewkroh reviewed Feb 21, 2018

View reviewed changes

adriansr force-pushed the osxpkg branch from 2efefd9 to eae39d1 Compare February 24, 2018 00:28

adriansr changed the title ~~[WIP] macOS packaging for beats~~ macOS packaging for beats Mar 23, 2018

adriansr removed discuss Issue needs further discussion. in progress Pull request is currently in progress. labels Mar 23, 2018

adriansr force-pushed the osxpkg branch from 67f13d5 to 72795f8 Compare March 23, 2018 13:46

andrewkroh reviewed Mar 27, 2018

View reviewed changes

andrewkroh approved these changes Mar 27, 2018

View reviewed changes

andrewkroh approved these changes Mar 28, 2018

View reviewed changes

adriansr force-pushed the osxpkg branch from b653a59 to 8c8901a Compare April 3, 2018 14:22

adriansr mentioned this pull request Apr 23, 2018

Beats preference pane for macOS #6923

Merged

adriansr force-pushed the osxpkg branch from e66fc15 to d1aea3a Compare April 30, 2018 16:38

adriansr force-pushed the osxpkg branch from d1aea3a to efabdcb Compare May 30, 2018 05:30

andrewkroh approved these changes May 31, 2018

View reviewed changes

adriansr force-pushed the osxpkg branch 4 times, most recently from dd234de to 633ddae Compare June 7, 2018 10:51

adriansr added 4 commits June 19, 2018 12:42

Install preference pane along with Beats

3951924

The installer package includes the Beats preference pane and installs it along with the Beat.

adriansr force-pushed the osxpkg branch from 633ddae to b9112f3 Compare June 19, 2018 05:43

andrewkroh closed this Jul 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macOS packaging for beats #6344

macOS packaging for beats #6344

adriansr commented Feb 9, 2018 •

edited

Loading

andrewkroh Feb 9, 2018

adriansr Feb 12, 2018

andrewkroh Feb 9, 2018

adriansr Feb 12, 2018

adriansr commented Feb 16, 2018 •

edited

Loading

andrewkroh commented Feb 19, 2018

adriansr commented Feb 19, 2018

adriansr commented Feb 20, 2018

adriansr commented Feb 20, 2018

andrewkroh Feb 21, 2018 •

edited

Loading

adriansr commented Mar 26, 2018

andrewkroh Mar 27, 2018

andrewkroh left a comment

andrewkroh left a comment

andrewkroh left a comment •

edited

Loading

adriansr commented May 31, 2018 •

edited

Loading

adriansr commented Jun 19, 2018 •

edited

Loading

andrewkroh commented Jun 30, 2018

adriansr commented Jul 2, 2018

andrewkroh commented Jul 3, 2018

	parser.add_argument("--no-snapshot", action="store_true",
	help="Don't append -SNAPSHOT to the version.")

macOS packaging for beats #6344

macOS packaging for beats #6344

Conversation

adriansr commented Feb 9, 2018 • edited Loading

andrewkroh Feb 9, 2018

Choose a reason for hiding this comment

adriansr Feb 12, 2018

Choose a reason for hiding this comment

andrewkroh Feb 9, 2018

Choose a reason for hiding this comment

adriansr Feb 12, 2018

Choose a reason for hiding this comment

adriansr commented Feb 16, 2018 • edited Loading

andrewkroh commented Feb 19, 2018

adriansr commented Feb 19, 2018

adriansr commented Feb 20, 2018

adriansr commented Feb 20, 2018

andrewkroh Feb 21, 2018 • edited Loading

Choose a reason for hiding this comment

adriansr commented Mar 26, 2018

andrewkroh Mar 27, 2018

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

adriansr commented May 31, 2018 • edited Loading

adriansr commented Jun 19, 2018 • edited Loading

andrewkroh commented Jun 30, 2018

adriansr commented Jul 2, 2018

andrewkroh commented Jul 3, 2018

adriansr commented Feb 9, 2018 •

edited

Loading

adriansr commented Feb 16, 2018 •

edited

Loading

andrewkroh Feb 21, 2018 •

edited

Loading

andrewkroh left a comment •

edited

Loading

adriansr commented May 31, 2018 •

edited

Loading

adriansr commented Jun 19, 2018 •

edited

Loading