Malcolm v24.02.0

[mmguero]

Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.

v24.01.0...v24.02.0

• Features and enhancements
  • Hedgehog Linux SD card image for Raspberry Pi (Hedgehog Linux SD card image for Raspberry Pi idaholab/Malcolm#250; special thanks to @aut0exec for his work on this)
  • allow configuration of Arkime's ILM/ISM settings (allow configuration of Arkime's ILM/ISM settings idaholab/Malcolm#300)
  • add option for customizing which log types get NetBox enrichment (add option for customizing which log types get NetBox enrichment idaholab/Malcolm#316)
  • improve the extracted_files download page (improve the extracted_files download page idaholab/Malcolm#329)
  • include missing aggregations in API bucket queries (include missing aggregations in API bucket queries idaholab/Malcolm#386)
  • more intelligent .env file checking on startup (more intelligent .env file checking on startup idaholab/Malcolm#387)
  • Malcolm report to itself on capture statistics (Malcolm report to itself on capture statistics idaholab/Malcolm#395)
  • link to Dashboards/Arkime from NetBox devices view (link to Dashboards/Arkime from NetBox devices view idaholab/Malcolm#410)
  • changed default PCAP storage format to zstd(3) for new installations
  • various documentation updates and improvements
  • changed back to using official Zeek .deb files rather than building from source to reduce build times
• Component version updates
  • Arkime to v5.0.0
  • Capa to v7.0.1
  • YARA to v4.5.0
  • Beats to v8.12.1
  • Logstash to v8.12.1
  • Zeek to v6.1.1
• Bug fixes
  • pivot links from Arkime to Kibana in external elasticsearch are not working (pivot links from Arkime to Kibana in external elasticsearch are not working idaholab/Malcolm#335)
  • redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment idaholab/Malcolm#403)
  • allow netbox-restore and netbox-backup to specify container name (allow netbox-restore and netbox-backup to specify container name idaholab/Malcolm#337)
  • fuzzy matching for manufacturers based on OUI to NetBox list is not very good (fuzzy matching for manufacturers based on OUI to NetBox list is not very good idaholab/Malcolm#393) (and updated documentation)
  • source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP idaholab/Malcolm#401)
  • event.severity_tags is not being assigned correctly based on rule.category (event.severity_tags is not being assigned correctly based on rule.category idaholab/Malcolm#402)
  • basic authentication breaks with special characters (basic authentication breaks with special characters idaholab/Malcolm#404)
  • changed some Logstash Ruby variables from global ($) to instance (@) (see "avoiding concurrency issues")
• Configuration changes (in environment variables in ./config/)
  • these variables in arkime.env to allow configuration of Arkime's ILM/ISM settings (allow configuration of Arkime's ILM/ISM settings idaholab/Malcolm#300)

  # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
  # Whether or not Arkime should perform index management
  INDEX_MANAGEMENT_ENABLED=false
  # Time in hours/days before moving to warm and force merge (number followed by h or d)
  INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
  # Time in hours/days before deleting index (number followed by h or d)
  INDEX_MANAGEMENT_RETENTION_TIME=90d
  # Number of replicas for older sessions indices
  INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
  # Number of weeks of history to retain
  INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
  # Number of segments to optimize sessions for
  INDEX_MANAGEMENT_SEGMENTS=1
  # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
  INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
  

  • these variables in dashboards.env to override the values automatically configured for pivot links (pivot links from Arkime to Kibana in external elasticsearch are not working idaholab/Malcolm#335) and /dashboard/ redirect (redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment idaholab/Malcolm#403) for Elasticsearch backend

  # These values are used to handle the Arkime value actions to pivot from Arkime
  #   to Dashboards. The nginx-proxy container's entrypoint will try to formulate
  #   them automatically, but they may be specified explicitly here.
  NGINX_DASHBOARDS_PREFIX=
  NGINX_DASHBOARDS_PROXY_PASS=
  

  • these variables in logstash.env for customizing which log types get NetBox enrichment (add option for customizing which log types get NetBox enrichment idaholab/Malcolm#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash

  # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
  LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
  

  # Zeek log types that will be ignored (dropped) by LogStash
  LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
  

  • these variables in netbox-common.env for adjusting matching device manufacturers to OUIs in NetBox autopopulation

  # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
  NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
  NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
  

  • these variables in suricata-live.env and zeek-live.env that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (Malcolm report to itself on capture statistics idaholab/Malcolm#395)

  # Whether or not enable capture statistics and include them in eve.json
  SURICATA_STATS_ENABLED=false
  SURICATA_STATS_EVE_ENABLED=false
  SURICATA_STATS_INTERVAL=30
  SURICATA_STATS_DECODER_EVENTS=false
  

  # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
  ZEEK_DISABLE_STATS=true
  

  • this variable in zeek.env related to the improvements to the extracted_files download page (improve the extracted_files download page idaholab/Malcolm#329)

  # Whether or not to use libmagic to show MIME types for Zeek-extracted files served
  EXTRACTED_FILE_HTTP_SERVER_MAGIC=false