Bump and docs: v2.12.0 (#923)

docker-compose.yml

@@ -9,7 +9,7 @@ services:
     depends_on:
       - postgres
       - redis
-    image: certpl/mwdb:v2.11.0
+    image: certpl/mwdb:v2.12.0
     restart: on-failure
     env_file:
       # NOTE: use gen_vars.sh in order to generate this file
@@ -22,7 +22,7 @@ services:
     build:
       context: .
       dockerfile: deploy/docker/Dockerfile-web
-    image: certpl/mwdb-web:v2.11.0
+    image: certpl/mwdb-web:v2.12.0
     ports:
       - "80:80"
     restart: on-failure

docs/conf.py

@@ -21,7 +21,7 @@
 author = 'CERT Polska'
 
 # The full version, including alpha/beta/rc tags
-release = '2.11.0'
+release = '2.12.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/index.rst

@@ -28,6 +28,7 @@ Features
    karton-guide
    oauth-guide
    rich-attributes-guide
+   prometheus-guide
 
 Indices and tables
 ==================

docs/prometheus-guide.rst

@@ -0,0 +1,42 @@
+Prometheus metrics
+==================
+
+.. versionadded:: 2.12.0
+
+MWDB allows to enable Prometheus metrics to grab information about API usage by users.
+
+Available metrics:
+
+- ``mwdb_api_requests (method, endpoint, user, status_code)`` that tracks usage of specific endpoints by users and status codes.
+- ``mwdb_deprecated_usage (feature, method, endpoint, user)`` that tracks usage of deprecated API endpoints
+
+Useful PromQL queries (in most cases we use 5-minute window):
+
+- ``sum(increase(mwdb_api_requests{}[5m])/5) by (user)`` - counts general API usage by MWDB users in requests per minute
+- ``sum(increase(mwdb_api_requests{status_code != "200"}[5m])) by (user, status_code)`` - API error rate grouped by user and status_code
+- ``sum(increase(mwdb_api_requests{endpoint=~"api.filedownloadresource|api.requestsampledownloadresource|api.downloadresource|api.filedownloadzipresource", method="GET"}[5m])/5) by (user)`` - file download rate
+- ``sum(increase(mwdb_api_requests{endpoint=~"api.fileresource|api.fileitemresource", method="POST"}[5m])/5) by (user)`` - file upload rate
+
+Setup guide
+-----------
+
+1. First, you need to configure Redis server where metric counters are stored. Redis instance can be configured in MWDB via
+   ``redis_uri`` option in mwdb.ini file or ``MWDB_REDIS_URI`` environment variable.
+
+2. Then you need to enable metrics by setting ``enable_prometheus_metrics=1`` in configuration or ``MWDB_ENABLE_PROMETHEUS_METRICS=1`` in env vars
+
+3. Log in as admin in MWDB and go to http://<mwdb>/settings/users. Create user account that will be used to access prometheus metrics.
+
+4. Go to ``Check user capabilities`` or http://<mwdb>/settings/user/<username>/capabilities to set ``access_prometheus_metrics`` ACL for created account.
+
+5. Then generate API key on http://<mwdb>>/settings/user/<username>/api-keys that will be used to scrape metrics by Prometheus
+
+After setup, metrics can be scraped using ``http://<mwdb-api>/api/varz`` endpoint with API key provided in ``Authorization: Bearer <api_key>`` HTTP header.
+
+You can test it before setting up Prometheus using curl:
+
+.. code-block:: console
+
+   $ curl http://<mwdb-api>/api/varz -H 'Authorization: Bearer <api_key>'
+
+Detailed instructions about Prometheus configuration can be found in `Prometheus docs <https://prometheus.io/docs/prometheus/latest/configuration/configuration/>`_.

docs/requirements.txt

@@ -1,5 +1,3 @@
-Sphinx==4.5.0
-sphinx-rtd-theme==0.5.2
-jsx-lexer==1.0.0
-# FIX: https://github.com/readthedocs/sphinx_rtd_theme/issues/1115
-docutils==0.16
+Sphinx==7.2.6
+sphinx-rtd-theme==2.0.0
+jsx-lexer==2.0.1

docs/setup-and-configuration.rst

@@ -41,12 +41,15 @@ MWDB was tested on Debian-based systems, but should work as well on other Linux
 
 For production environments, you need to install:
 
-
 * **PostgreSQL database** (minimum supported version: 12, https://www.postgresql.org/download/linux/debian/)
 * libfuzzy2 for ssdeep evaluation
+* other native dependencies listed below
 
-Optionally you can install:
+.. code-block:: console
 
+    $ apt install gcc libfuzzy-dev python3-dev python3-venv postgresql-client postgresql-common
+
+Optionally you can install:
 
 * **Docker engine and Docker-Compose** if you want to use the Docker-based setup (https://docs.docker.com/engine/install/)
 * **Redis database** needed by extra features like rate-limiting (https://redis.io/topics/quickstart)
@@ -311,6 +314,8 @@ Extra features:
 * ``enable_registration`` (0 or 1) - Turns on user registration features. Requires additional configuration. Default is ``0``.
 * ``enable_maintenance`` (0 or 1) - Turns on maintenance mode, making MWDB unavailable for users other than ``admin``. Default is ``0``.
 * ``enable_json_logger`` (0 or 1) - Enables JSON logging which may be more convenient for automated log processing. Default is ``0``.
+* ``enable_prometheus_metrics`` (0 or 1) - Enables Prometheus metrics (\ ``/api/varz`` endpoint\ )
+* ``enable_debug_log`` (0 or 1) - Enables debug logging
 * ``redis_uri`` (string) - Redis database connection string, required by rate limiter.
 * ``remotes`` (comma separated strings) - list of MWDB remotes (experimental)
 * ``enable_hooks`` (0 or 1) - enable plugin hooks

docs/whats-changed.rst

@@ -5,7 +5,78 @@ This page describes the most significant changes in the following versions, that
 integration and plugin developers. We usually don't break API compatibility within major version, but plugins may
 have compatibility problems after minor mwdb-core upgrade.
 
-For upgrade instructions, see :ref:`Upgrade mwdb-core to latest version`.
+For upgrade instructions, see :ref:`Upgrading mwdb-core to latest version`.
+
+v2.12.0
+-------
+
+This release contains major changes in search mechanism and drops usage of Flask-RESTful, which might break plugin
+compatibility.
+
+Complete changelog can be found here: `v2.12.0 changelog <https://github.com/CERT-Polska/mwdb-core/releases/tag/v2.12.0>`_.
+
+[Important change] Refactor of search engine
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Search engine uses ``@>`` and ``@?`` PostgreSQL operators and can utilize GIN index on JSONB and ARRAY columns. It means that queries like:
+
+- ``file.name:"sample.exe"``
+- ``cfg.url:"https://example.com"``
+- ``attribute.url:"https://example.com"``
+
+will work much faster. In addition, new search engine comes with various improvements:
+
+- Both inclusive and exclusive ranges are allowed for detatime-columns
+- Range boundaries can be reversed and are automatically sorted
+- Escape parser is no longer regex-based and should be much more consistent
+
+As search engine uses a bit different approach, some things may work a bit different:
+
+- >=, >, <, <= are regular range operators, so they need to be put outside term
+
+  correct form: ``size:>="5kB"``
+
+  incorrect form: ``size:">=5kB"``
+
+Complete description of changes can be found here: https://github.com/CERT-Polska/mwdb-core/pull/906
+
+[Important change] Replaced Flask-RESTful with own lightweight implementation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Flask-RESTful is nice framework for creating REST API Flask services, but it's a bit abandoned which blocks us
+from further development and causes issues with newer Flask version. We switched to our own implementation of ``Resource`` classes.
+
+For API users this change is completely transparent, but if you develop your own plugins, you should change imports:
+
+.. code-block:: diff
+
+   - from flask_restful import Resource
+   + from mwdb.core.service import Resource
+
+Complete description of changes can be found here: https://github.com/CERT-Polska/mwdb-core/pull/916
+
+[Important change] Changes in logging, introduced Prometheus metrics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We're buried in tons of non-meaningful logs without any option to set the verbosity. Examples of non-meaningful logs:
+
+- information about usage of deprecated endpoints
+- 'before_request'/'request' logs which are useful for debugging hanging or slow endpoints, but only for debugging
+- threadName, moduleName, lineNo which doesn't carry any useful information that can't be read from the log itself
+
+That's why in v2.12.0 debug verbosity level was introduced and is turned off by default. If you want to turn on
+debug logs, you can enable it via ``enable_debug_log=1`` option in configuration.
+
+As a replacement for flooding our disk space with log files, we introduced Prometheus metrics that can be
+tracked using Grafana platform. Read more about setup in :ref:`Prometheus metrics`.
+
+v2.11.0
+-------
+
+Minor release with small QoL improvement: added forms to upload configs and blobs directly from eb UI.
+
+Complete changelog can be found here: `v2.11.0 changelog <https://github.com/CERT-Polska/mwdb-core/releases/tag/v2.11.0>`_.
+
 
 v2.10.1
 -------
@@ -125,7 +196,7 @@ Plugin modules are imported dynamically (using `import() <https://developer.mozi
 Check for any runtime errors in DevTools, especially noting messages like ``Plugin ${pluginName} failed to load``.
 
 [Important change] Replaced uWSGI with Gunicorn
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 ``certpl/mwdb`` Docker image uses `Gunicorn <https://docs.gunicorn.org/en/stable/index.html>`_ instead of `uWSGI <https://uwsgi-docs.readthedocs.io/en/latest/>`_
 for serving Python WSGI application. If you have uWSGI-dependent configuration customized via environment variables, you need to change it

mwdb/version.py

@@ -4,5 +4,5 @@
 except IOError:
     git_revision = ""
 
-app_version = "2.11.0"
+app_version = "2.12.0"
 app_build_version = f"{app_version}+{git_revision}" if git_revision else app_version

mwdb/web/package.json

@@ -1,6 +1,6 @@
 {
     "name": "mwdb-web",
-    "version": "2.11.0",
+    "version": "2.12.0",
     "private": true,
     "scripts": {
         "dev": "vite",

setup.py

@@ -7,7 +7,7 @@
 """
 
 setup(name="mwdb-core",
-      version="2.11.0",
+      version="2.12.0",
       description="MWDB Core malware database",
       long_description=LONG_DESCRIPTION,
       author="CERT Polska",