From d0aaedc5b30ae5c8888a21176b1b2df238d9997f Mon Sep 17 00:00:00 2001 From: Alvar Penning Date: Tue, 25 Jun 2024 16:05:17 +0200 Subject: [PATCH] doc: Initialize documentation Both structure and huge parts are copied from Icinga DB and were altered afterwards. The already existing Channel Plugin documentation was extended. --- doc/01-About.md | 35 +++++++ doc/02-Installation.md | 121 ++++++++++++++++++++++ doc/02-Installation.md.d/From-Source.md | 22 ++++ doc/03-Configuration.md | 117 +++++++++++++++++++++ doc/04-Upgrading.md | 4 + doc/05-Distributed-Setups.md | 3 + doc/{channel-plugin.md => 06-Channels.md} | 21 +++- doc/07-HTTP-API.md | 47 +++++++++ 8 files changed, 368 insertions(+), 2 deletions(-) create mode 100644 doc/01-About.md create mode 100644 doc/02-Installation.md create mode 100644 doc/02-Installation.md.d/From-Source.md create mode 100644 doc/03-Configuration.md create mode 100644 doc/04-Upgrading.md create mode 100644 doc/05-Distributed-Setups.md rename doc/{channel-plugin.md => 06-Channels.md} (71%) create mode 100644 doc/07-HTTP-API.md diff --git a/doc/01-About.md b/doc/01-About.md new file mode 100644 index 000000000..108dfd29f --- /dev/null +++ b/doc/01-About.md @@ -0,0 +1,35 @@ +# Icinga Notifications + +Icinga Notifications is a set of components that processes received events from various sources, manages incidents and +forwards notifications to predefined contacts, consisting of: + +* The Icinga Notifications daemon (this project), which receives events and sends notifications. +* An [Icinga Web module](https://github.com/Icinga/icinga-notifications-web) that provides graphical configuration and further processing of the data collected by the daemon. +* And Icinga 2 and other custom sources that propagate state updates and acknowledgement events to the daemon. + +## Big Picture + +Because Icinga Notifications consists of several components, +this section tries to help understand how these components relate. + +First, the Icinga Notifications configuration resides in a SQL database. +It can be conveniently tweaked via Icinga Notifications Web directly from a web browser. +The Icinga Notifications daemon uses this database to read the current configuration. + +As in any Icinga setup, all host and service checks are defined in Icinga 2. +By querying the Icinga 2 API, the Icinga Notifications daemon retrieves state changes, acknowledgements, and other events. +These events are stored in the database and are available for further inspection in the Icinga Notifications Web. + +Depending on its configuration, the daemon will take further action on these events. +This optionally includes escalations that are sent through a channel plugin. +Each such channel plugin implements a domain-specific transport, e.g., the `email` channel sends emails via SMTP. +When configured, Icinga Notifications will use channel plugins to notify end users or talk to other APIs. + +## Installation + +To install Icinga Notifications see [Installation](02-Installation.md). + +## License + +Icinga Notifications and the Icinga Notifications documentation are licensed under the terms of the +GNU General Public License Version 2. diff --git a/doc/02-Installation.md b/doc/02-Installation.md new file mode 100644 index 000000000..48cca4007 --- /dev/null +++ b/doc/02-Installation.md @@ -0,0 +1,121 @@ + +# Installing Icinga Notifications + +The recommended way to install Icinga Notifications is to use prebuilt packages +for all supported platforms from our official release repository. +Please follow the steps listed for your target operating system, +which guide you through setting up the repository and installing Icinga Notifications. + +To upgrade an existing Icinga Notifications installation to a newer version, +see the [Upgrading](04-Upgrading.md) documentation for the necessary steps. + + + +## Setting up the Database + +A MySQL (≥5.5), MariaDB (≥10.1), or PostgreSQL (≥9.6) database is required to run Icinga Notifications. +Please follow the steps listed for your target database, +which guide you through setting up the database and user and importing the schema. + +### Setting up a MySQL or MariaDB Database + +If you use a version of MySQL < 5.7 or MariaDB < 10.2, the following server options must be set: + +``` +innodb_file_format=barracuda +innodb_file_per_table=1 +innodb_large_prefix=1 +``` + +Set up a MySQL database for Icinga Notifications: + +``` +# mysql -u root -p + +CREATE DATABASE notifications; +CREATE USER 'notifications'@'localhost' IDENTIFIED BY 'CHANGEME'; +GRANT ALL ON notifications.* TO 'notifications'@'localhost'; +``` + +After creating the database, import the Icinga Notifications schema using the following command: + +``` +mysql -u root -p notifications < /usr/share/icinga-notifications/schema/mysql/schema.sql +``` + +### Setting up a PostgreSQL Database + +Set up a PostgreSQL database for Icinga Notifications: + +``` +# su -l postgres + +createuser -P notifications +createdb -E UTF8 --locale en_US.UTF-8 -T template0 -O notifications notifications +echo 'CREATE EXTENSION IF NOT EXISTS citext;' | psql notifications +``` + +The `CREATE EXTENSION` command requires the `postgresql-contrib` package. + +Edit `pg_hba.conf`, insert the following before everything else: + +``` +local all notifications md5 +host all notifications 0.0.0.0/0 md5 +host all notifications ::/0 md5 +``` + +To apply these changes, run `systemctl reload postgresql`. + +After creating the database, import the Icinga Notifications schema using the following command: + +``` +psql -U icinga-notifications notifications < /usr/share/icinga-notifications/schema/pgsql/schema.sql +``` + +## Configuring Icinga Notifications + +Icinga Notifications installs its configuration file to `/etc/icinga-notifications/config.yml`, +pre-populating most of the settings for a local setup. Before running Icinga Notifications, +adjust the database credentials and the Icinga Web 2 URL. +The configuration file explains general settings. +All available settings can be found under [Configuration](03-Configuration.md). + +## Configuring Icinga 2 + +Icinga Notifications utilizes the [Icinga 2 API](https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/) +to fetch events from an Icinga 2 instance. +Thus, if the Icinga 2 API is not already configured, +please follow the [API setup guide](https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#setting-up-the-api) now. +Furthermore, an [`ApiUser`](https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#objecttype-apiuser) +with [permissions](https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#icinga2-api-permissions) +as listed in the example below is required. + +``` +object ApiUser "notifications" { + password = "insecure" + permissions = [ "events/*", "objects/query/*" ] +} +``` + +After completing setting up Icinga Notifications Web, the Icinga 2 API can be configured with this `ApiUser` as a source. + +## Running Icinga Notifications + +The `icinga-notifications` package automatically installs the necessary systemd unit files to run Icinga Notifications. +Please run the following command to enable and start its service: + +``` +systemctl enable --now icinga-notifications +``` + +## Installing Icinga Notifications Web + +With Icinga 2, Icinga Notifications and the database fully set up, it is now time to install Icinga Notifications Web, +which connects to the database and allows configuring Icinga Notifications. + +The Icinga Notifications Web package is also included in the Icinga repository, and since it is already set up, +you have completed the instructions here and can proceed to +[install Icinga Notifications Web](https://icinga.com/docs/icinga-notifications-web/latest/doc/02-Installation/). + + diff --git a/doc/02-Installation.md.d/From-Source.md b/doc/02-Installation.md.d/From-Source.md new file mode 100644 index 000000000..032ffe919 --- /dev/null +++ b/doc/02-Installation.md.d/From-Source.md @@ -0,0 +1,22 @@ +# Installing Icinga Notifications from Source + +To install Icinga Notifications and get started, you first need to clone the git repository. +```bash +git clone https://github.com/Icinga/icinga-notifications.git +cd icinga-notifications +``` + +Now you can build the daemon and all its channels with the `Makefile` and you will find all build artifacts in the `build` directory. +```bash +make +``` + +Finally, the `Makefile` also allows you to perform an installation on your system. +To specify destinations, consult the variables on top of the `Makefile` which can be overwritten by your environment. +This command will need extended permissions, so it might need to be executed by the `root` user, through `sudo` or a similar mechanism. +```bash +sudo make install +``` + + + \ No newline at end of file diff --git a/doc/03-Configuration.md b/doc/03-Configuration.md new file mode 100644 index 000000000..fdbb74121 --- /dev/null +++ b/doc/03-Configuration.md @@ -0,0 +1,117 @@ +# Configuration + +The configuration for Icinga Notifications is twofold. +The main configuration resides in the database, +shared between the Icinga Notifications daemon and Icinga Notifications Web. +However, as the Icinga Notifications daemon needs to know how to access this database and some further settings, +it needs its own configuration file as well. + +This configuration is stored in `/etc/icinga-notifications/config.yml`. +See [config.example.yml](../config.example.yml) for an example configuration. + +## Top Level Configuration + +### HTTP API Configuration + +The HTTP API listener can be used both for submission and for debugging purposes. + +| Option | Description | +|----------------|----------------------------------------------------------------------| +| listen | Address to bind to, port included. (Example: `localhost:5680`) | +| debug-password | Password expected via HTTP Basic Authentication for debug endpoints. | + +### Icinga Web 2 + +The `icingaweb2-url` is expected to point to the base directory of your Icinga Web 2 installation, +i.e., `https://example.com/icingaweb2/`, to be used for URL creation. + +### Channel Plugin Directory + +All available Icinga Notification Channel Plugins should resident in the `channel-plugin-dir`. +For a package install, this directory is `/usr/libexec/icinga-notifications/channels` and should not be altered. + +## Database Configuration + +Connection configuration for the database to which Icinga Notifications synchronizes monitoring data. +This is also the database used in Icinga Notifications Web to view and work with the data. + +| Option | Description | +|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| +| type | **Optional.** Either `mysql` (default) or `pgsql`. | +| host | **Required.** Database host or absolute Unix socket path. | +| port | **Optional.** Database port. By default, the MySQL or PostgreSQL port, depending on the database type. | +| database | **Required.** Database name. | +| user | **Required.** Database username. | +| password | **Optional.** Database password. | +| tls | **Optional.** Whether to use TLS. | +| cert | **Optional.** Path to TLS client certificate. | +| key | **Optional.** Path to TLS private key. | +| ca | **Optional.** Path to TLS CA certificate. | +| insecure | **Optional.** Whether not to verify the peer. | +| options | **Optional.** List of low-level [database options](#database-options) that can be set to influence some Icinga Notifications internal default behaviours. | + +### Database Options + +Each of these configuration options are highly technical with thoroughly considered and tested default values that you +should only change when you exactly know what you are doing. You can use these options to influence the Icinga Notifications default +behaviour, how it interacts with databases, thus the defaults are usually sufficient for most users and do not need any +manual adjustments. + +!!! important + + Do not change the defaults if you do not have to! + +| Option | Description | +|--------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| +| max_connections | **Optional.** Maximum number of database connections Icinga Notifications is allowed to open in parallel if necessary. Defaults to `16`. | +| max_connections_per_table | **Optional.** Maximum number of queries Icinga Notifications is allowed to execute on a single table concurrently. Defaults to `8`. | +| max_placeholders_per_statement | **Optional.** Maximum number of placeholders Icinga Notifications is allowed to use for a single SQL statement. Defaults to `8192`. | +| max_rows_per_transaction | **Optional.** Maximum number of rows Icinga Notifications is allowed to `SELECT`,`DELETE`,`UPDATE` or `INSERT` in a single transaction. Defaults to `8192`. | +| wsrep_sync_wait | **Optional.** Enforce [Galera cluster](#galera-cluster) nodes to perform strict cluster-wide causality checks. Defaults to `7`. | + +## Logging Configuration + +Configuration of the logging component used by Icinga Notifications. + +| Option | Description | +|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| level | **Optional.** Specifies the default logging level. Can be set to `fatal`, `error`, `warn`, `info` or `debug`. Defaults to `info`. | +| output | **Optional.** Configures the logging output. Can be set to `console` (stderr) or `systemd-journald`. If not set, logs to systemd-journald when running under systemd, otherwise stderr. | +| interval | **Optional.** Interval for periodic logging defined as [duration string](#duration-string). Defaults to `"20s"`. | +| options | **Optional.** Map of component name to logging level in order to set a different logging level for each component instead of the default one. See [logging components](#logging-components) for details. | + +### Logging Components + +| Component | Description | +|-----------------|---------------------------------------------------------------------------| +| channel | Notification channels, their configuration and output. | +| database | Database connection status and queries. | +| icinga2 | Icinga 2 API communications, including the Event Stream. | +| incident | Incident management and changes. | +| listener | HTTP listener for event submission and debugging. | +| runtime-updates | Configuration changes through Icinga Notifications Web from the database. | + +## Appendix + +### Duration String + +A duration string is a sequence of decimal numbers and a unit suffix, such as `"20s"`. +Valid units are `"ms"`, `"s"`, `"m"` and `"h"`. + +### Galera Cluster + +Icinga Notifications expects a more consistent behaviour from its database than a +[Galera cluster](https://mariadb.com/kb/en/what-is-mariadb-galera-cluster/) provides by default. To accommodate this, +Icinga Notifications sets the [wsrep_sync_wait](https://mariadb.com/kb/en/galera-cluster-system-variables/#wsrep_sync_wait) system +variable for all its database connections. Consequently, strict cluster-wide causality checks are enforced before +executing specific SQL queries, which are determined by the value set in the `wsrep_sync_wait` system variable. +By default, Icinga Notifications sets this to `7`, which includes `READ, UPDATE, DELETE, INSERT, REPLACE` query types and is +usually sufficient. Unfortunately, this also has the downside that every single Icinga Notifications query will be blocked until +the cluster nodes resynchronise their states after each executed query, and may result in degraded performance. + +However, this does not necessarily have to be the case if, for instance, Icinga Notifications is only allowed to connect to a +single cluster node at a time. This is the case when a load balancer does not randomly route connections to all the +nodes evenly, but always to the same node until it fails, or if your database cluster nodes have a virtual IP address +fail over assigned. In such situations, you can set the `wsrep_sync_wait` system variable to `0` in the +`/etc/icinga-notifications/config.yml` file to disable it entirely, as Icinga Notifications doesn't have to wait for cluster +synchronisation then. diff --git a/doc/04-Upgrading.md b/doc/04-Upgrading.md new file mode 100644 index 000000000..7df38a670 --- /dev/null +++ b/doc/04-Upgrading.md @@ -0,0 +1,4 @@ +# Upgrading Icinga Notifications + +Specific version upgrades are described below. Please note that version upgrades are incremental. +If you are upgrading across multiple versions, make sure to follow the steps for each of them. diff --git a/doc/05-Distributed-Setups.md b/doc/05-Distributed-Setups.md new file mode 100644 index 000000000..4afca60d9 --- /dev/null +++ b/doc/05-Distributed-Setups.md @@ -0,0 +1,3 @@ +# Distributed Setups + +To be done. \ No newline at end of file diff --git a/doc/channel-plugin.md b/doc/06-Channels.md similarity index 71% rename from doc/channel-plugin.md rename to doc/06-Channels.md index a40dec106..319a45daf 100644 --- a/doc/channel-plugin.md +++ b/doc/06-Channels.md @@ -1,4 +1,21 @@ -## Channel Plugin +# Channels + +After Icinga Notifications decides to send a notification of any kind, it will be passed to a channel plugin. +Such a channel plugin submits the notification event to a channel, e.g., email or a chat client. + +Icinga Notifications comes packed with channel plugins, but also enables you to develop your own plugins. + +To make those plugins available to Icinga Notifications, they must be placed in the [plugin channel directory](03-Configuration.md#channel-plugin-directory), +which is being done automatically for package installations. +Afterwards they can be configured through Icinga Notifications Web. + +## Available Channels + +- _email_: Email submission via SMTP +- _rocketchat_: Rocket.Chat +- _webhook_: Configurable HTTP/HTTPS queries for your backend + +## Technical Channel Description Channel plugins are processes that run continuously and independently of each other. They receive many requests over their lifetime. They receive JSON-formatted requests on stdin and reply with JSON-formatted responses on stdout. The @@ -73,4 +90,4 @@ Currently, the channel plugin include following three methods: [(example)](../internal/channel/examples/get-info.json) - `SendNotification`: Send the notifications. The `params` key should contain the information about the contact to be notified, corresponding object, the incident and the triggered event. - [(example)](../internal/channel/examples/send-notification.json) \ No newline at end of file + [(example)](../internal/channel/examples/send-notification.json) diff --git a/doc/07-HTTP-API.md b/doc/07-HTTP-API.md new file mode 100644 index 000000000..dc26156c8 --- /dev/null +++ b/doc/07-HTTP-API.md @@ -0,0 +1,47 @@ +# HTTP API + +Icinga Notifications comes with its own HTTP API, [configurable](03-Configuration.md/#http-api-configuration) +via `listen` and `debug-password`. + +## Process Event + +One possible source next to the Icinga 2 API is event submission to the Icinga Notification HTTP API listener. +After creating a source with type _Other_ in Icinga Notifications Web, +the specified credentials can be used for HTTP Basic Authentication of a JSON-encoded +[Event](https://github.com/Icinga/icinga-notifications/blob/main/internal/event/event.go). + +``` +curl -v \ + -u 'source-1:correct horse battery staple' \ + -d '{"extra_tags":{"hostgroup/app-container":null ,"hostgroup/department-dev":null,"hostgroup/env-qa":null,"hostgroup/location-rome":null,"servicegroup/app-mail":null,"servicegroup/department-nms":null,"servicegroup/env-prod":null,"servicegroup/location-berlin":null},"message":"You will be honored for contributing your time and skill to a worthy cause.","name":"dummy-809: random fortune","severity":"ok","source_id":2,"tags":{"host":"dummy-809","service":"random fortune"},"url":"http://localhost/icingaweb2/icingadb/service?name=random%20fortune&host.name=dummy-809","username":""}' \ + 'http://localhost:5680/process-event' +``` + +## Debugging Endpoints + +There are multiple endpoints for dumping specific configurations. +To use those, the `debug-password` must be set and supplied via HTTP Basic Authentication next to an arbitrary username. + +### Dump Config + +The database-stored configuration from Icinga Notifications current viewpoint can be dumped as JSON. + +``` +curl -v -u ':debug-password' 'http://localhost:5680/dump-config' +``` + +### Dump Incidents + +The current incidents can be dumped as JSON. + +``` +curl -v -u ':debug-password' 'http://localhost:5680/dump-incidents' +``` + +### Dump Schedules + +All schedules with their assignee can be dumped in a human-readable form. + +``` +curl -v -u ':debug-password' 'http://localhost:5680/dump-schedules' +```