Skip to content

Latest commit

 

History

History
263 lines (208 loc) · 10.4 KB

server.md

File metadata and controls

263 lines (208 loc) · 10.4 KB
Raw

Server

A centralized server running configurable services collecting events from clients and can output them to sinks.

Table of contents

Usage

Server requires a config file.

Usage:
  xatu server [flags]

Flags:
      --config string   config file (default is server.yaml) (default "server.yaml")
  -h, --help            help for server

Requirements

Configuration

Server requires a single yaml config file. An example file can be found here

Name Type Default Description
logging string warn Log level (panic, fatal, warn, info, debug, trace)
metricsAddr string :9090 The address the metrics server will listen on
pprofAddr string The address the pprof server will listen on. When ommited, the pprof server will not be started
addr string :8080 The grpc address for services
labels object A key value map of labels to append to every sentry event
ntpServer string pool.ntp.org NTP server to calculate clock drift for events
persistence.enabled bool false Enable persistence
persistence.driverName string postgres Persistence driver name (postgres)
persistence.connectionString string Connection string for the persistence driver
persistence.maxIdleConns int 2 The maximum number of connections in the idle connection pool. 0 means no idle connections are retained.
persistence.maxOpenConns int 0 The maximum number of open connections to the database. 0 means unlimited connections.
store.type string memory Type of store (memory, redis-server, redis-cluster)
store.config object Store type configuration redis-server/redis-cluster
geoip.enabled bool false Enable the geoip provider
geoip.type string maxmind Type of store (maxmind)
geoip.config object GeoIP Provider type configuration maxmind
services object Services to run
services.coordinator object Coordinator service
services.coordinator.enabled bool false Enable the coordinator service
services.coordinator.nodeRecord.maxQueueSize int 51200 The maximum queue size to buffer node records for delayed processing. If the queue gets full it drops the items
services.coordinator.nodeRecord.batchTimeout string 5s The maximum duration for constructing a batch. Processor forcefully sends available node records when timeout is reached
services.coordinator.nodeRecord.exportTimeout string 30s The maximum duration for exporting node records. If the timeout is reached, the export will be cancelled
services.coordinator.nodeRecord.maxExportBatchSize int 512 MaxExportBatchSize is the maximum number of node records to process in a single batch. If there are more than one batch worth of items then it processes multiple batches of items one batch after the other without any delay
services.eventIngester object Event Ingester service
services.eventIngester.enabled bool false Enable the event ingester service
services.eventIngester.outputs array List exampleone batch after the other without any delay

Store redis-server configuration

Store configuration for Redis server.

Name Type Default Description
store.config.address string The address of the redis server. Details on the address format
store.config.prefix string xatu The redis key prefix to use

Store redis-cluster configuration

Store configuration for Redis cluster.

Name Type Default Description
store.config.address string The address of the redis cluster. Details on the address format
store.config.prefix string xatu The redis key prefix to use

GeoIP maxmind configuration

GeoIP configuration for MaxMind.

Name Type Default Description
geoip.config.database.city string The path to the MaxMind GeoLite2 City database file
geoip.config.database.asn string The path to the MaxMind GeoLite2 ASN database file

Simple Example

Simple example that runs on default ports and enables the Coordinator and Event Ingester services.

services:
  coordinator:
    enabled: true
    persistence:
      driverName: postgres
      connectionString: postgres://postgres:admin@localhost:5432/xatu?sslmode=disable
  eventIngester:
    enabled: true
    outputs:
    - name: stdout
      type: stdout

Only run the event exporter example

services:
  eventIngester:
    enabled: true
    outputs:
    - name: http-sink
      type: http
      config:
        address: http://localhost:8081
        headers:
          authorization: "Basic Someb64Value"

Complex example

logging: "info"
addr: ":8080"
metricsAddr: ":9090"
pprofAddr: ":6060"

labels:
  ethpandaops: rocks

ntpServer: time.google.com

persistence:
  enabled: true
  driverName: postgres
  connectionString: postgres://postgres:password@localhost:5432/xatu?sslmode=disable
  maxIdleConns: 2 # 0 = no idle connections are retained
  maxOpenConns: 0 # 0 = unlimited

store:
  type: redis-server
  config:
    address: "redis://localhost:6379"

geoip:
  enabled: true
  type: maxmind
  config:
    database:
      city: ./GeoLite2-City.mmdb
      asn: ./GeoLite2-ASN.mmdb

services:
  coordinator:
    enabled: true
    nodeRecord:
      maxQueueSize: 51200
      batchTimeout: 5s
      exportTimeout: 30s
      maxExportBatchSize: 512
  eventIngester:
    enabled: true
    outputs:
    - name: logs
      type: stdout
    - name: http-sink
      type: http
      config:
        address: http://localhost:8081
        headers:
          authorization: "Basic Someb64Value"
    - name: kafka-sink
      type: kafka
      config:
        brokers: localhost:19092
        topic: events

Services

Xatu server runs a number of services that can be configured to run or not. Each service has its own configuration.

Coordinator

The coordinator service is responsible for;

Persistence is required for the coordinator service.

Event Ingester

The event ingester service is responsible for receiving events from clients (sentries), validating and then forwarding them to sinks.

Persistence

The server uses persistence layer to store data from various services. Currently only PostgreSQL is supported.

Migrations

Xatu server uses the golange-migrate tool for simple manual file based migrations. Xatu tries to follow the Best practices: How to write migrations when adding new migrations.

An example to run migrations locally against a local postgres instance;

migrate -database "postgres://postgres:password@localhost:5432/xatu?sslmode=disable" -path ./migrations/postgres up

Running locally

# docker
docker run -d --name xatu-server -v $HOST_DIR_CHANGE_ME/config.yaml:/opt/xatu/config.yaml -p 9090:9090 -p 8080:8080 -it ethpandaops/xatu:latest server --config /opt/xatu/config.yaml
# build
go build -o dist/xatu main.go
./dist/xatu server --config server.yaml
# dev
go run main.go server --config server.yaml

Running locally with persistence (native)

Requires postgres and golange-migrate to be installed locally.

# Create the xatu database
psql -h localhost -U postgres -W -c "create database xatu;"
# Run migrations (requires https://github.com/golang-migrate/migrate tool)
migrate -database "postgres://postgres:password@localhost:5432/xatu?sslmode=disable" -path ./migrations/postgres up
# Run the server
go run main.go server --config server.yaml

Running locally with persistence (docker)

Requires docker to be installed locally.

# Start a postgres instance
docker run -p 5432:5432 --name xatupostgres -e POSTGRES_PASSWORD=password -d postgres
# Create the xatu database. "password" is the password
docker exec -it xatupostgres psql -U postgres -W -c "create database xatu;"
# Run migrations
docker run -v "$(pwd)/migrations/postgres":/migrations --network host migrate/migrate -path=/migrations/ -database "postgres://postgres:password@localhost:5432/xatu?sslmode=disable" up
# Run the server
go run main.go server --config server.yaml