Add EnableLogging and LogLabel supports for Node NetworkPolicy

This commit introduces limited support for traffic logging in Node NetworkPolicy. The limitations are: - Traffic logs are written only to the system log (not managed by Antrea). Users can filter logs using syslog filters. - The `LogLabel` for Node NetworkPolicy is restricted to a maximum of 12 characters. Node NetworkPolicy's data path is implemented via iptables. An iptables "non-terminating target" `LOG` is added before the final matching rule to log packets to the system kernel log. The logs provide packet match details, such as: ``` Sep 2 10:31:07 k8s-node-control-plane kernel: [6657320.789675] Antrea:I:Allow:allow-http:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=10.10.0.10 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52813 DF PROTO=TCP SPT=57658 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 Sep 2 10:31:11 k8s-node-control-plane kernel: [6657324.899219] Antrea:I:Drop:default-drop:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=192.168.240.201 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27486 DF PROTO=TCP SPT=33152 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 ``` The log prefix (e.g., `Antrea:I:Allow:allow-http:`) is up to 29 characters long and includes a user-provided log label (up to 12 characters). The log prefix format: ``` |---1--| |2| |---3--| |----------4--------| |Antrea|:|I|:|Reject|:|user-provided label|:| |6 |1|1|1|4-6 |1|1-12 |1| ``` - Part 1: Fixed, "Antrea" - Part 2: Direction, "I" (In) or "O" (Out) - Part 3: Action, "Allow", "Drop", or "Reject" - Part 4: User-provided log label, up to 12 characters Signed-off-by: Hongliang Liu <lhongliang@vmware.com>
antrea-io · Sep 11, 2024 · 446c1da · 446c1da
1 parent a996421
commit 446c1da
Show file tree

Hide file tree

Showing 9 changed files with 400 additions and 148 deletions.
diff --git a/docs/antrea-node-network-policy.md b/docs/antrea-node-network-policy.md
@@ -6,6 +6,7 @@
 - [Introduction](#introduction)
 - [Prerequisites](#prerequisites)
 - [Usage](#usage)
+- [Logs](#logs)
 - [Limitations](#limitations)
 <!-- /toc -->
 
@@ -66,11 +67,15 @@ spec:
       ports:
         - protocol: TCP
           port: 80
+      enableLogging: true
+      logLabel: allow-http
     - name: drop-other
       action: Drop
       ports:
         - protocol: TCP
           port: 80
+      enableLogging: true
+      logLabel: default-drop-others
 ```
 
 An example Node NetworkPolicy that blocks egress traffic from Nodes with label
@@ -105,6 +110,52 @@ spec:
           port: 22
 ```
 
+## Logs
+
+The `enableLogging` and `logLabel` options provide limited support for Node NetworkPolicies. Since Node NetworkPolicies
+are implemented using iptables, enabling `enableLogging` causes the Linux kernel to log information about all matching
+packets via the kernel log. However, Antrea cannot process these logs directly. Instead, these logs can be accessed
+through syslog, allowing you to filter and direct them to specific files using syslog syntax.
+
+For example, consider the Node NetworkPolicy `restrict-http-to-node` above. It could generate the following logs:
+
+```text
+Sep  2 10:31:07 k8s-node-control-plane kernel: [6657320.789675] Antrea:I:Allow:allow-http:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=10.10.0.10 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52813 DF PROTO=TCP SPT=57658 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
+Sep  2 10:31:11 k8s-node-control-plane kernel: [6657324.899219] Antrea:I:Drop:default-drop:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=192.168.240.201 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27486 DF PROTO=TCP SPT=33152 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
+```
+
+In these logs, prefixes like `Antrea:I:Allow:allow-http:` and `Antrea:I:Drop:default-drop:` are added by iptables using
+the `--log-prefix` parameter. The iptables log prefix is limited to 29 characters, as described in the
+[iptables-extensions manual](https://ipset.netfilter.org/iptables-extensions.man.html).
+
+The log prefix format includes essential information of a Node NetworkPolicy rule, and consists of four parts,
+formatted as follows:
+
+```text
+|---1--| |2| |---3--| |----------4--------|
+|Antrea|:|I|:|Reject|:|user-provided label|:|
+|6     |1|1|1|4-6   |1|1-12               |1|
+```
+
+- Part 1: Fixed, "Antrea"
+- Part 2: Direction, "I" (In) or "O" (Out)
+- Part 3: Action, "Allow", "Drop", or "Reject"
+- Part 4: User-provided `logLabel`, up to 12 characters
+
+Due to iptables' 29-character prefix limitation, the user-provided  `logLabel` is restricted to 12 characters.
+To manage these logs effectively, you can configure rsyslog on each Node as follows:
+
+```text
+# Example rsyslog configuration to filter Antrea logs
+:msg, contains, "Antrea:I:Allow:allow-http" /var/log/antrea-node-netpol-allow.log
+:msg, contains, "Antrea:I:Drop:default-drop" /var/log/antrea-node-netpol-drop.log
+& stop
+```
+
+This configuration directs logs with the prefix `Antrea:I:Allow:allow-http` to `/var/log/antrea-node-netpol-allow.log`
+and logs with the prefix `Antrea:I:Drop:default-drop` to `/var/log/antrea-node-netpol-drop.log`. The `& stop` command
+ensures that these logs are not processed further.
+
 ## Limitations
 
 - This feature is currently only supported for Linux Nodes.
@@ -118,4 +169,4 @@ spec:
 - FQDN is not supported for ACNPs applied to Nodes.
 - Layer 7 NetworkPolicy is not supported yet.
 - For UDP or SCTP, when the `Reject` action is specified in an egress rule, it behaves identical to the `Drop` action.
-- Traffic logging is not supported yet for ACNPs applied to Nodes.
+- Limited support for traffic logging for ACNPs applied to Nodes.
diff --git a/pkg/agent/controller/networkpolicy/node_reconciler_linux.go b/pkg/agent/controller/networkpolicy/node_reconciler_linux.go
@@ -43,6 +43,10 @@ const (
 	ipv6Any = "::/0"
 )
 
+// The logging of Node NetworkPolicy is implemented by iptables target LOG, which turns on kernel logging of matching
+// packets. The label is useful for distinguishing Node NetworkPolicy logs from other kernel logs.
+const logLabelPrefix = "Antrea"
+
 var ipsetTypeHashIP = ipset.HashIP
 
 /*
@@ -124,7 +128,7 @@ directly.
 type coreIPTRule struct {
 	ruleID   string
 	priority *types.Priority
-	ruleStr  string
+	ruleStrs []string
 }
 
 type chainKey struct {
@@ -256,7 +260,7 @@ func (r *nodeReconciler) batchAdd(rules []*CompletedRule) error {
 			}
 
 			// Collect all core iptables rules.
-			coreIPTRule := &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}
+			coreIPTRule := &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}
 			if rule.Direction == v1beta2.DirectionIn {
 				ingressCoreIPTRules[ipProtocol] = append(ingressCoreIPTRules[ipProtocol], coreIPTRule)
 			} else {
@@ -322,6 +326,11 @@ func (r *nodeReconciler) GetRuleByFlowID(ruleFlowID uint32) (*types.PolicyRule,
 
 func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Protocol]*types.NodePolicyRule, *nodePolicyLastRealized) {
 	ruleID := rule.ID
+	enableLogging := rule.EnableLogging
+	var logLabel string
+	if enableLogging {
+		logLabel = generateLogLabel(rule)
+	}
 	lastRealized := newNodePolicyLastRealized()
 	priority := &types.Priority{
 		TierPriority:   *rule.TierPriority,
@@ -362,7 +371,12 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot
 
 		var serviceIPTRules []string
 		if serviceIPTChain != "" {
-			serviceIPTRules = buildServiceIPTRules(ipProtocol, rule.Services, serviceIPTChain, serviceIPTRuleTarget)
+			serviceIPTRules = buildServiceIPTRules(ipProtocol,
+				rule.Services,
+				serviceIPTChain,
+				serviceIPTRuleTarget,
+				enableLogging,
+				logLabel)
 		}
 
 		ipnets := getIPNetsFromRule(rule, isIPv6)
@@ -383,14 +397,19 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot
 			lastRealized.ipnets[ipProtocol] = ipnet
 		}
 
-		coreIPTRule := buildCoreIPTRule(ipProtocol,
+		coreIPTRules := buildCoreIPTRules(ipProtocol,
 			coreIPTChain,
 			ipset,
 			ipnet,
 			coreIPTRuleTarget,
 			coreIPTRuleComment,
 			service,
-			rule.Direction == v1beta2.DirectionIn)
+			rule.Direction == v1beta2.DirectionIn,
+			// If the target of a core iptables rule is not a service chain, the iptables rule for logging should be
+			// generated along with the core iptables rule. Otherwise, the iptables rules for logging should be generated
+			// along with the service iptables rules.
+			enableLogging && serviceIPTChain == "",
+			logLabel)
 
 		nodePolicyRules[ipProtocol] = &types.NodePolicyRule{
 			IPSet:           ipset,
@@ -399,7 +418,7 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot
 			ServiceIPTChain: serviceIPTChain,
 			ServiceIPTRules: serviceIPTRules,
 			CoreIPTChain:    coreIPTChain,
-			CoreIPTRule:     coreIPTRule,
+			CoreIPTRules:    coreIPTRules,
 			IsIPv6:          isIPv6,
 		}
 	}
@@ -422,7 +441,7 @@ func (r *nodeReconciler) add(rule *CompletedRule) error {
 				return err
 			}
 		}
-		if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, false, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
+		if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, false, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
 			return err
 		}
 	}
@@ -453,7 +472,7 @@ func (r *nodeReconciler) update(lastRealized *nodePolicyLastRealized, newRule *C
 			}
 		}
 		if prevIPSet != ipset || prevIPNet != ipnet {
-			if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
+			if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
 				return err
 			}
 		}
@@ -496,9 +515,7 @@ func (r *nodeReconciler) addOrUpdateCoreIPTRules(chain string, isIPv6 bool, isUp
 	// Get all iptables rules and synchronize them.
 	var ruleStrs []string
 	for _, rule := range rules {
-		if rule.ruleStr != "" {
-			ruleStrs = append(ruleStrs, rule.ruleStr)
-		}
+		ruleStrs = append(ruleStrs, rule.ruleStrs...)
 	}
 	if err := r.routeClient.AddOrUpdateNodeNetworkPolicyIPTables([]string{chain}, [][]string{ruleStrs}, isIPv6); err != nil {
 		return err
@@ -533,7 +550,7 @@ func (r *nodeReconciler) deleteCoreIPTRule(ruleID string, iptChain string, isIPv
 	// Get all the iptables rules and synchronize them.
 	var ruleStrs []string
 	for _, r := range rules {
-		ruleStrs = append(ruleStrs, r.ruleStr)
+		ruleStrs = append(ruleStrs, r.ruleStrs...)
 	}
 	if err := r.routeClient.AddOrUpdateNodeNetworkPolicyIPTables([]string{iptChain}, [][]string{ruleStrs}, isIPv6); err != nil {
 		return err
@@ -614,32 +631,35 @@ func getIPNetsFromRule(rule *CompletedRule, isIPv6 bool) sets.Set[string] {
 	return set
 }
 
-func buildCoreIPTRule(ipProtocol iptables.Protocol,
+func buildCoreIPTRules(ipProtocol iptables.Protocol,
 	iptChain string,
 	ipset string,
 	ipnet string,
 	iptRuleTarget string,
 	iptRuleComment string,
 	service *v1beta2.Service,
-	isIngress bool) string {
+	isIngress bool,
+	enableLogging bool,
+	logLabel string) []string {
 	builder := iptables.NewRuleBuilder(iptChain)
+	var rules []string
 	if isIngress {
 		if ipset != "" {
 			builder = builder.MatchIPSetSrc(ipset, ipsetTypeHashIP)
 		} else if ipnet != "" {
 			builder = builder.MatchCIDRSrc(ipnet)
 		} else {
-			// If no source IP address is matched, return an empty string since the core iptables will never be matched.
-			return ""
+			// If no source IP address is matched, return an empty slice since the core iptables will never be matched.
+			return rules
 		}
 	} else {
 		if ipset != "" {
 			builder = builder.MatchIPSetDst(ipset, ipsetTypeHashIP)
 		} else if ipnet != "" {
 			builder = builder.MatchCIDRDst(ipnet)
 		} else {
-			// If no destination IP address is matched, return an empty string since the core iptables will never be matched.
-			return ""
+			// If no destination IP address is matched, return an empty slice since the core iptables will never be matched.
+			return rules
 		}
 	}
 	if service != nil {
@@ -657,13 +677,26 @@ func buildCoreIPTRule(ipProtocol iptables.Protocol,
 			builder = builder.MatchICMP(service.ICMPType, service.ICMPCode, ipProtocol)
 		}
 	}
-	return builder.SetTarget(iptRuleTarget).
+	if enableLogging {
+		rules = append(rules, builder.CopyBuilder().
+			SetTarget(iptables.LOGTarget).
+			SetLogPrefix(logLabel).
+			Done().
+			GetRule())
+	}
+	rules = append(rules, builder.SetTarget(iptRuleTarget).
 		SetComment(iptRuleComment).
 		Done().
-		GetRule()
+		GetRule())
+	return rules
 }
 
-func buildServiceIPTRules(ipProtocol iptables.Protocol, services []v1beta2.Service, chain string, ruleTarget string) []string {
+func buildServiceIPTRules(ipProtocol iptables.Protocol,
+	services []v1beta2.Service,
+	chain string,
+	ruleTarget string,
+	enableLogging bool,
+	logLabel string) []string {
 	var rules []string
 	builder := iptables.NewRuleBuilder(chain)
 	for _, svc := range services {
@@ -681,6 +714,13 @@ func buildServiceIPTRules(ipProtocol iptables.Protocol, services []v1beta2.Servi
 		case "icmp":
 			copiedBuilder = copiedBuilder.MatchICMP(svc.ICMPType, svc.ICMPCode, ipProtocol)
 		}
+		if enableLogging {
+			rules = append(rules, copiedBuilder.CopyBuilder().
+				SetTarget(iptables.LOGTarget).
+				SetLogPrefix(logLabel).
+				Done().
+				GetRule())
+		}
 		rules = append(rules, copiedBuilder.SetTarget(ruleTarget).
 			Done().
 			GetRule())
@@ -707,3 +747,24 @@ func getServiceTransProtocol(protocol *v1beta2.Protocol) string {
 	}
 	return strings.ToLower(string(*protocol))
 }
+
+func generateLogLabel(rule *CompletedRule) string {
+	// Construct the log label used as iptables log prefix. According to https://ipset.netfilter.org/iptables-extensions.man.html,
+	// the log prefix is up to 29 letters long. The log label should include essential information to help filter the
+	// generated iptables kernel log. As a result, the user-provided log label is limited to 12 characters.
+	// The log label format:
+	// |Antrea|:|I|:|Reject|:|user-provided label|:|
+	// |6     |1|1|1|4-6   |1|1-12               |1|
+	logLabel := fmt.Sprintf("%s:%s:%s", logLabelPrefix, rule.Direction[:1], *rule.Action)
+	if rule.LogLabel != "" {
+		ruleLogLabel := rule.LogLabel
+		// Truncate the user-provided log label if it exceeds 12 characters.
+		if len(ruleLogLabel) > 12 {
+			klog.InfoS("The rule log label that exceeds 12 characters will be truncated", "rule.LogLabel", rule.LogLabel)
+			ruleLogLabel = ruleLogLabel[:12]
+		}
+		logLabel = fmt.Sprintf("%s:%s", logLabel, ruleLogLabel)
+	}
+	logLabel += ":"
+	return logLabel
+}