Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ingest Manager] Support for linux/arm64 #23479

Merged
merged 8 commits into from
Jan 15, 2021
Merged

[Ingest Manager] Support for linux/arm64 #23479

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e3717f3
arm
michalpristas Jan 12, 2021
69d4462
arch updated
michalpristas Jan 12, 2021
816c393
changelog
michalpristas Jan 13, 2021
c6fd118
Merge branch 'master' of github.com:elastic/beats into agent-arm
michalpristas Jan 13, 2021
5cd10ac
runtime
michalpristas Jan 14, 2021
41cb372
update
michalpristas Jan 14, 2021
29de112
tests
michalpristas Jan 14, 2021
ac1e08d
unit test endpoint constraints
michalpristas Jan 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions x-pack/elastic-agent/CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
- Fix shell wrapper for deb/rpm packaging {pull}23038[23038]
- Fixed parsing of npipe URI {pull}22978[22978]
- Remove artifacts on transient download errors {pull}23235[23235]
- Support for linux/arm64 {pull}23479[23479]
- Skip top level files when unziping archive during upgrade {pull}23456[23456]
- Do not take ownership of Endpoint log path {pull}23444[23444]

Expand Down
4 changes: 4 additions & 0 deletions x-pack/elastic-agent/pkg/agent/application/global_config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
package application

import (
"runtime"

"github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/application/paths"

"github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/errors"
Expand All @@ -31,5 +33,7 @@ func agentGlobalConfig() map[string]interface{} {
"home": paths.Home(),
"logs": paths.Logs(),
},
"runtime.os": runtime.GOOS,
"runtime.arch": runtime.GOARCH,
}
}
15 changes: 15 additions & 0 deletions x-pack/elastic-agent/pkg/agent/program/program.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,21 @@ func detectPrograms(agentInfo transpiler.AgentInfo, singleConfig *transpiler.AST
programs := make([]Program, 0)
for _, spec := range Supported {
specificAST := singleConfig.Clone()
if len(spec.Constraints) > 0 {
constraints, err := eql.New(spec.Constraints)
if err != nil {
return nil, err
}
ok, err := constraints.Eval(specificAST)
ph marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return nil, err
}

if !ok {
continue
}
}

err := spec.Rules.Apply(agentInfo, specificAST)
if err != nil {
return nil, err
Expand Down
3 changes: 3 additions & 0 deletions x-pack/elastic-agent/pkg/agent/program/program_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,9 @@ func TestConfiguration(t *testing.T) {
"endpoint_unknown_output": {
expected: 0,
},
"endpoint_arm": {
expected: 0,
},
}

for name, test := range testcases {
Expand Down
1 change: 1 addition & 0 deletions x-pack/elastic-agent/pkg/agent/program/spec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ type Spec struct {
PostInstallSteps *transpiler.StepList `yaml:"post_install"`
PreUninstallSteps *transpiler.StepList `yaml:"pre_uninstall"`
When string `yaml:"when"`
Constraints string `yaml:"constraints"`
Copy link
Contributor Author

@michalpristas michalpristas Jan 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i added contraints section which is basically a when section eql expression, evaluated before transformations to save some time.
runtime.os/arch were injected into a tree beforehand, so far as a separate keys not a map, we need to add support for accessing dict elements in eql expression
cc @blakerouse as you already approved the PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small question, we are adding a new key "constraints", could this be handled already by the "when" key?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it could but it would mean we do the transformation which are not even necessary and then filter out all of the results because precondition failed. i wanted us to spare compute cycles

}

// ReadSpecs reads all the specs that match the provided globbing path.
Expand Down
4 changes: 3 additions & 1 deletion x-pack/elastic-agent/pkg/agent/program/spec_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@ func TestSerialization(t *testing.T) {
PreUninstallSteps: transpiler.NewStepList(
transpiler.ExecFile(30, "app", "uninstall", "--force"),
),
When: "1 == 1",
When: "1 == 1",
Constraints: "2 == 2",
}
yml := `name: hello
cmd: hellocmd
Expand Down Expand Up @@ -118,6 +119,7 @@ pre_uninstall:
- --force
timeout: 30
when: 1 == 1
constraints: 2 == 2
`
t.Run("serialization", func(t *testing.T) {
b, err := yaml.Marshal(spec)
Expand Down
2 changes: 1 addition & 1 deletion x-pack/elastic-agent/pkg/agent/program/supported.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

115 changes: 115 additions & 0 deletions x-pack/elastic-agent/pkg/agent/program/testdata/endpoint_arm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
revision: 5
name: Endpoint Host
fleet:
agent:
id: fleet-agent-id
host:
id: host-agent-id
access_api_key: VuaCfGcBCdbkQm-e5aOx:ui2lp2axTNmsyakw9tvNnw
kibana:
protocol: https
host: localhost:5601
timeout: 30s

outputs:
default:
type: elasticsearch
hosts: [127.0.0.1:9200, 127.0.0.1:9300]
username: elastic
password: changeme
api_key: TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA
ca_sha256: 7HIpactkIAq2Y49orFOOQKurWxmmSFZhBCoQYcRhJ3Y=

inputs:
- id: endpoint-id
type: endpoint
name: endpoint-1
enabled: true
package:
name: endpoint
version: 0.3.0
data_stream:
namespace: default
artifact_manifest:
schema_version: v22
manifest_version: v21
artifacts:
- endpoint-allowlist-windows:
sha256: 1234
size: 2
url: /relative/path/to/endpoint-allowlist-windows
- endpoint-allowlist-macos:
sha256: 1234
size: 2
url: /relative/path/to/endpoint-allowlist-macos
- endpoint-allowlist-linux:
sha256: 1234
size: 2
url: /relative/path/to/endpoint-allowlist-linux
policy:
linux:
advanced:
free-form: free-form-value
indices:
network: logs-endpoint.events.network-default
file: logs-endpoint.events.file-default
process: logs-endpoint.events.process-default
metadata: metrics-endpoint.metadata-default
policy: metrics-endpoint.policy-default
telemetry: metrics-endpoint.telemetry-default
logging:
file: info
stdout: debug
events:
process: true
file: true
network: true
windows:
malware:
mode: prevent
advanced:
free-form: free-form-value
indices:
network: logs-endpoint.events.network-default
file: logs-endpoint.events.file-default
registry: logs-endpoint.events.registry-default
process: logs-endpoint.events.process-default
driver: logs-endpoint.events.driver-default
library: logs-endpoint.events.library-default
alerts: logs-endpoint.alerts-default
metadata: metrics-endpoint.metadata-default
policy: metrics-endpoint.policy-default
telemetry: metrics-endpoint.telemetry-default
logging:
file: info
stdout: debug
events:
registry: true
process: true
security: true
file: true
dns: false
dll_and_driver_load: false
network: true
mac:
malware:
mode: prevent
advanced:
free-form: free-form-value
indices:
network: logs-endpoint.events.network-default
file: logs-endpoint.events.file-default
process: logs-endpoint.events.process-default
alerts: logs-endpoint.alerts-default
metadata: metrics-endpoint.metadata-default
policy: metrics-endpoint.policy-default
telemetry: metrics-endpoint.telemetry-default
logging:
file: info
stdout: debug
events:
process: true
file: true
network: true
runtime:
arch: arm64
2 changes: 2 additions & 0 deletions x-pack/elastic-agent/pkg/agent/program/testdata/endpoint_basic.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,3 +111,5 @@ inputs:
process: true
file: true
network: true
runtime:
arch: x86_64
13 changes: 7 additions & 6 deletions x-pack/elastic-agent/pkg/artifact/artifact.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,13 @@ import (
)

var packageArchMap = map[string]string{
"linux-binary-32": "linux-x86.tar.gz",
"linux-binary-64": "linux-x86_64.tar.gz",
"windows-binary-32": "windows-x86.zip",
"windows-binary-64": "windows-x86_64.zip",
"darwin-binary-32": "darwin-x86_64.tar.gz",
"darwin-binary-64": "darwin-x86_64.tar.gz",
"linux-binary-32": "linux-x86.tar.gz",
"linux-binary-64": "linux-x86_64.tar.gz",
"linux-binary-arm64": "linux-arm64.tar.gz",
"windows-binary-32": "windows-x86.zip",
"windows-binary-64": "windows-x86_64.zip",
"darwin-binary-32": "darwin-x86_64.tar.gz",
"darwin-binary-64": "darwin-x86_64.tar.gz",
}

// GetArtifactName constructs a path to a downloaded artifact
Expand Down
4 changes: 3 additions & 1 deletion x-pack/elastic-agent/pkg/artifact/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,9 @@ func (c *Config) Arch() string {
}

arch := "32"
if strings.Contains(runtime.GOARCH, "64") {
if strings.Contains(runtime.GOARCH, "arm64") {
arch = "arm64"
} else if strings.Contains(runtime.GOARCH, "64") {
arch = "64"
}

Expand Down
2 changes: 2 additions & 0 deletions x-pack/elastic-agent/pkg/artifact/download/http/elastic_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ func getTestCases() []testCase {
return []testCase{
{"linux", "32"},
{"linux", "64"},
{"linux", "arm64"},
{"darwin", "32"},
{"darwin", "64"},
{"windows", "32"},
Expand Down Expand Up @@ -164,6 +165,7 @@ func getElasticCoClient() http.Client {
fmt.Sprintf("%s-%s-%s", beatSpec.Cmd, version, "i686.rpm"): struct{}{},
fmt.Sprintf("%s-%s-%s", beatSpec.Cmd, version, "x86_64.rpm"): struct{}{},
fmt.Sprintf("%s-%s-%s", beatSpec.Cmd, version, "linux-x86.tar.gz"): struct{}{},
fmt.Sprintf("%s-%s-%s", beatSpec.Cmd, version, "linux-arm64.tar.gz"): struct{}{},
fmt.Sprintf("%s-%s-%s", beatSpec.Cmd, version, "linux-x86_64.tar.gz"): struct{}{},
fmt.Sprintf("%s-%s-%s", beatSpec.Cmd, version, "windows-x86.zip"): struct{}{},
fmt.Sprintf("%s-%s-%s", beatSpec.Cmd, version, "windows-x86_64.zip"): struct{}{},
Expand Down
1 change: 1 addition & 0 deletions x-pack/elastic-agent/spec/endpoint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,4 @@ rules:
- revision

when: length(${fleet}) > 0 and length(${inputs}) > 0 and hasKey(${output}, 'elasticsearch')
constraints: ${runtime.arch} != 'arm64'