Merge branch 'main' into patch-5

hashicorp · Jun 12, 2023 · 6651b2d · 6651b2d
2 parents 9ec2b38 + 446a640
commit 6651b2d
Show file tree

Hide file tree

Showing 31 changed files with 2,933 additions and 194 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,70 @@
+## 1.16.0-rc1 (June 12, 2023)
+
+BREAKING CHANGES:
+
+* api: The `/v1/health/connect/` and `/v1/health/ingress/` endpoints now immediately return 403 "Permission Denied" errors whenever a token with insufficient `service:read` permissions is provided. Prior to this change, the endpoints returned a success code with an empty result list when a token with insufficient permissions was provided. [[GH-17424](https://github.com/hashicorp/consul/issues/17424)]
+* peering: Removed deprecated backward-compatibility behavior.
+Upstream overrides in service-defaults will now only apply to peer upstreams when the `peer` field is provided.
+Visit the 1.16.x [upgrade instructions](https://developer.hashicorp.com/consul/docs/upgrading/upgrade-specific) for more information. [[GH-16957](https://github.com/hashicorp/consul/issues/16957)]
+
+SECURITY:
+
+* audit-logging: **(Enterprise only)** limit `v1/operator/audit-hash` endpoint to ACL token with `operator:read` privileges.
+
+FEATURES:
+
+* api: (Enterprise only) Add `POST /v1/operator/audit-hash` endpoint to calculate the hash of the data used by the audit log hash function and salt.
+* cli: (Enterprise only) Add a new `consul operator audit hash` command to retrieve and compare the hash of the data used by the audit log hash function and salt.
+* cli: Adds new command - `consul services export` - for exporting a service to a peer or partition [[GH-15654](https://github.com/hashicorp/consul/issues/15654)]
+* connect: **(Consul Enterprise only)** Implement order-by-locality failover.
+* mesh: Add new permissive mTLS mode that allows sidecar proxies to forward incoming traffic unmodified to the application. This adds `AllowEnablingPermissiveMutualTLS` setting to the mesh config entry and the `MutualTLSMode` setting to proxy-defaults and service-defaults. [[GH-17035](https://github.com/hashicorp/consul/issues/17035)]
+* mesh: Support configuring JWT authentication in Envoy. [[GH-17452](https://github.com/hashicorp/consul/issues/17452)]
+* server: **(Enterprise Only)** added server side RPC requests IP based read/write rate-limiter. [[GH-4633](https://github.com/hashicorp/consul/issues/4633)]
+* server: **(Enterprise Only)** allow automatic license utilization reporting. [[GH-5102](https://github.com/hashicorp/consul/issues/5102)]
+* server: added server side RPC requests global read/write rate-limiter. [[GH-16292](https://github.com/hashicorp/consul/issues/16292)]
+* xds: Add `property-override` built-in Envoy extension that directly patches Envoy resources. [[GH-17487](https://github.com/hashicorp/consul/issues/17487)]
+* xds: Add a built-in Envoy extension that inserts External Authorization (ext_authz) network and HTTP filters. [[GH-17495](https://github.com/hashicorp/consul/issues/17495)]
+* xds: Add a built-in Envoy extension that inserts Wasm HTTP filters. [[GH-16877](https://github.com/hashicorp/consul/issues/16877)]
+* xds: Add a built-in Envoy extension that inserts Wasm network filters. [[GH-17505](https://github.com/hashicorp/consul/issues/17505)]
+
+IMPROVEMENTS:
+
+* * api: Support filtering for config entries. [[GH-17183](https://github.com/hashicorp/consul/issues/17183)]
+* * cli: Add `-filter` option to `consul config list` for filtering config entries. [[GH-17183](https://github.com/hashicorp/consul/issues/17183)]
+* api: Enable setting query options on agent force-leave endpoint. [[GH-15987](https://github.com/hashicorp/consul/issues/15987)]
+* audit-logging: (Enterprise only) enable error response and request body logging [[GH-5669](https://github.com/hashicorp/consul/issues/5669)]
+* audit-logging: **(Enterprise only)** enable error response and request body logging
+* ca: automatically set up Vault's auto-tidy setting for tidy_expired_issuers when using Vault as a CA provider. [[GH-17138](https://github.com/hashicorp/consul/issues/17138)]
+* ca: support Vault agent auto-auth config for Vault CA provider using AliCloud authentication. [[GH-16224](https://github.com/hashicorp/consul/issues/16224)]
+* ca: support Vault agent auto-auth config for Vault CA provider using AppRole authentication. [[GH-16259](https://github.com/hashicorp/consul/issues/16259)]
+* ca: support Vault agent auto-auth config for Vault CA provider using Azure MSI authentication. [[GH-16298](https://github.com/hashicorp/consul/issues/16298)]
+* ca: support Vault agent auto-auth config for Vault CA provider using JWT authentication. [[GH-16266](https://github.com/hashicorp/consul/issues/16266)]
+* ca: support Vault agent auto-auth config for Vault CA provider using Kubernetes authentication. [[GH-16262](https://github.com/hashicorp/consul/issues/16262)]
+* command: Adds ACL enabled to status output on agent startup. [[GH-17086](https://github.com/hashicorp/consul/issues/17086)]
+* command: Allow creating ACL Token TTL with greater than 24 hours with the -expires-ttl flag. [[GH-17066](https://github.com/hashicorp/consul/issues/17066)]
+* connect: **(Enterprise Only)** Add support for specifying "Partition" and "Namespace" in Prepared Queries failover rules.
+* connect: update supported envoy versions to 1.23.10, 1.24.8, 1.25.7, 1.26.2 [[GH-17546](https://github.com/hashicorp/consul/issues/17546)]
+* connect: update supported envoy versions to 1.23.8, 1.24.6, 1.25.4, 1.26.0 [[GH-5200](https://github.com/hashicorp/consul/issues/5200)]
+* fix metric names in /docs/agent/telemetry [[GH-17577](https://github.com/hashicorp/consul/issues/17577)]
+* gateway: Change status condition reason for invalid certificate on a listener from "Accepted" to "ResolvedRefs". [[GH-17115](https://github.com/hashicorp/consul/issues/17115)]
+* http: accept query parameters `datacenter`, `ap` (enterprise-only), and `namespace` (enterprise-only). Both short-hand and long-hand forms of these query params are now supported via the HTTP API (dc/datacenter, ap/partition, ns/namespace). [[GH-17525](https://github.com/hashicorp/consul/issues/17525)]
+* systemd: set service type to notify. [[GH-16845](https://github.com/hashicorp/consul/issues/16845)]
+* ui: Update alerts to Hds::Alert component [[GH-16412](https://github.com/hashicorp/consul/issues/16412)]
+* ui: Update to use Hds::Toast component to show notifications [[GH-16519](https://github.com/hashicorp/consul/issues/16519)]
+* ui: update from <button> and <a> to design-system-components button <Hds::Button> [[GH-16251](https://github.com/hashicorp/consul/issues/16251)]
+* ui: update typography to styles from hds [[GH-16577](https://github.com/hashicorp/consul/issues/16577)]
+
+BUG FIXES:
+
+* Fix a race condition where an event is published before the data associated is commited to memdb. [[GH-16871](https://github.com/hashicorp/consul/issues/16871)]
+* gateways: **(Enterprise only)** Fixed a bug in API gateways where gateway configuration objects in non-default partitions did not reconcile properly. [[GH-17581](https://github.com/hashicorp/consul/issues/17581)]
+* gateways: Fixed a bug in API gateways where binding a route that only targets a service imported from a peer results
+in the programmed gateway having no routes. [[GH-17609](https://github.com/hashicorp/consul/issues/17609)]
+* gateways: Fixed a bug where API gateways were not being taken into account in determining xDS rate limits. [[GH-17631](https://github.com/hashicorp/consul/issues/17631)]
+* peering: Fixes a bug where the importing partition was not added to peered failover targets, which causes issues when the importing partition is a non-default partition. [[GH-16673](https://github.com/hashicorp/consul/issues/16673)]
+* ui: fixes ui tests run on CI [[GH-16428](https://github.com/hashicorp/consul/issues/16428)]
+* xds: Fixed a bug where modifying ACLs on a token being actively used for an xDS connection caused all xDS updates to fail. [[GH-17566](https://github.com/hashicorp/consul/issues/17566)]
+
 ## 1.15.3 (June 1, 2023)
 
 BREAKING CHANGES:

diff --git a/agent/envoyextensions/builtin/http/localratelimit/ratelimit.go b/agent/envoyextensions/builtin/http/localratelimit/ratelimit.go
@@ -60,6 +60,9 @@ func (r *ratelimit) fromArguments(args map[string]interface{}) error {
 	if err := mapstructure.Decode(args, r); err != nil {
 		return fmt.Errorf("error decoding extension arguments: %v", err)
 	}
+	if r.ProxyType == "" {
+		r.ProxyType = string(api.ServiceKindConnectProxy)
+	}
 	return r.validate()
 }
 
@@ -188,7 +191,7 @@ func (r ratelimit) PatchFilter(p extensioncommon.FilterPayload) (*envoy_listener
 }
 
 func validateProxyType(t string) error {
-	if t != "connect-proxy" {
+	if t != string(api.ServiceKindConnectProxy) {
 		return fmt.Errorf("unexpected ProxyType %q", t)
 	}
 

diff --git a/agent/envoyextensions/builtin/http/localratelimit/ratelimit_test.go b/agent/envoyextensions/builtin/http/localratelimit/ratelimit_test.go
@@ -111,6 +111,30 @@ func TestConstructor(t *testing.T) {
 			expectedErrMsg: "cannot parse 'FilterEnforced', -1 overflows uint",
 			ok:             false,
 		},
+		"invalid proxy type": {
+			arguments: makeArguments(map[string]interface{}{
+				"ProxyType":     "invalid",
+				"FillInterval":  30,
+				"MaxTokens":     20,
+				"TokensPerFill": 5,
+			}),
+			expectedErrMsg: `unexpected ProxyType "invalid"`,
+			ok:             false,
+		},
+		"default proxy type": {
+			arguments: makeArguments(map[string]interface{}{
+				"FillInterval":  30,
+				"MaxTokens":     20,
+				"TokensPerFill": 5,
+			}),
+			expected: ratelimit{
+				ProxyType:     "connect-proxy",
+				MaxTokens:     intPointer(20),
+				FillInterval:  intPointer(30),
+				TokensPerFill: intPointer(5),
+			},
+			ok: true,
+		},
 		"valid everything": {
 			arguments: makeArguments(map[string]interface{}{
 				"ProxyType":     "connect-proxy",

diff --git a/agent/envoyextensions/builtin/lua/lua.go b/agent/envoyextensions/builtin/lua/lua.go
@@ -45,6 +45,9 @@ func (l *lua) fromArguments(args map[string]interface{}) error {
 	if err := mapstructure.Decode(args, l); err != nil {
 		return fmt.Errorf("error decoding extension arguments: %v", err)
 	}
+	if l.ProxyType == "" {
+		l.ProxyType = string(api.ServiceKindConnectProxy)
+	}
 	return l.validate()
 }
 
@@ -53,7 +56,7 @@ func (l *lua) validate() error {
 	if l.Script == "" {
 		resultErr = multierror.Append(resultErr, fmt.Errorf("missing Script value"))
 	}
-	if l.ProxyType != "connect-proxy" {
+	if l.ProxyType != string(api.ServiceKindConnectProxy) {
 		resultErr = multierror.Append(resultErr, fmt.Errorf("unexpected ProxyType %q", l.ProxyType))
 	}
 	if l.Listener != "inbound" && l.Listener != "outbound" {

diff --git a/agent/envoyextensions/builtin/lua/lua_test.go b/agent/envoyextensions/builtin/lua/lua_test.go
@@ -54,6 +54,15 @@ func TestConstructor(t *testing.T) {
 			arguments: makeArguments(map[string]interface{}{"Listener": "invalid"}),
 			ok:        false,
 		},
+		"default proxy type": {
+			arguments: makeArguments(map[string]interface{}{"ProxyType": ""}),
+			expected: lua{
+				ProxyType: "connect-proxy",
+				Listener:  "inbound",
+				Script:    "lua-script",
+			},
+			ok: true,
+		},
 		"valid everything": {
 			arguments: makeArguments(map[string]interface{}{}),
 			expected: lua{

diff --git a/agent/envoyextensions/builtin/property-override/property_override.go b/agent/envoyextensions/builtin/property-override/property_override.go
@@ -261,6 +261,9 @@ func (p *propertyOverride) validate() error {
 		}
 	}
 
+	if p.ProxyType == "" {
+		p.ProxyType = api.ServiceKindConnectProxy
+	}
 	if err := validProxyTypes.CheckRequired(string(p.ProxyType), "ProxyType"); err != nil {
 		resultErr = multierror.Append(resultErr, err)
 	}

diff --git a/agent/envoyextensions/builtin/property-override/property_override_test.go b/agent/envoyextensions/builtin/property-override/property_override_test.go
@@ -236,7 +236,7 @@ func TestConstructor(t *testing.T) {
 		// enforces expected behavior until we do. Multi-member slices should be unaffected
 		// by WeakDecode as it is a more-permissive version of the default behavior.
 		"single value Patches decoded as map construction succeeds": {
-			arguments: makeArguments(map[string]any{"Patches": makePatch(map[string]any{})}),
+			arguments: makeArguments(map[string]any{"Patches": makePatch(map[string]any{}), "ProxyType": nil}),
 			expected:  validTestCase(OpAdd, extensioncommon.TrafficDirectionOutbound, ResourceTypeRoute).expected,
 			ok:        true,
 		},

diff --git a/agent/xds/resources_test.go b/agent/xds/resources_test.go
@@ -415,6 +415,19 @@ func getAPIGatewayGoldenTestCases(t *testing.T) []goldenTestCase {
 						Kind: structs.HTTPRoute,
 						Name: "route",
 						Rules: []structs.HTTPRouteRule{{
+							Filters: structs.HTTPFilters{
+								Headers: []structs.HTTPHeaderFilter{
+									{
+										Add: map[string]string{
+											"X-Header-Add": "added",
+										},
+										Set: map[string]string{
+											"X-Header-Set": "set",
+										},
+										Remove: []string{"X-Header-Remove"},
+									},
+								},
+							},
 							Services: []structs.HTTPService{{
 								Name: "service",
 							}},

diff --git a/agent/xds/routes.go b/agent/xds/routes.go
@@ -477,8 +477,6 @@ func (s *ResourceGenerator) routesForAPIGateway(cfgSnap *proxycfg.ConfigSnapshot
 				return nil, err
 			}
 
-			addHeaderFiltersToVirtualHost(&reformatedRoute, virtualHost)
-
 			defaultRoute.VirtualHosts = append(defaultRoute.VirtualHosts, virtualHost)
 		}
 
@@ -1097,16 +1095,6 @@ func injectHeaderManipToRoute(dest *structs.ServiceRouteDestination, r *envoy_ro
 	return nil
 }
 
-func addHeaderFiltersToVirtualHost(dest *structs.HTTPRouteConfigEntry, vh *envoy_route_v3.VirtualHost) {
-	for _, rule := range dest.Rules {
-		for _, header := range rule.Filters.Headers {
-			vh.RequestHeadersToAdd = append(vh.RequestHeadersToAdd, makeHeadersValueOptions(header.Add, true)...)
-			vh.RequestHeadersToAdd = append(vh.RequestHeadersToAdd, makeHeadersValueOptions(header.Set, false)...)
-			vh.RequestHeadersToRemove = append(vh.RequestHeadersToRemove, header.Remove...)
-		}
-	}
-}
-
 func injectHeaderManipToVirtualHost(dest *structs.IngressService, vh *envoy_route_v3.VirtualHost) error {
 	if !dest.RequestHeaders.IsZero() {
 		vh.RequestHeadersToAdd = append(

diff --git a/agent/xds/testdata/routes/api-gateway-with-http-route-and-inline-certificate.latest.golden b/agent/xds/testdata/routes/api-gateway-with-http-route-and-inline-certificate.latest.golden
@@ -1,31 +1,50 @@
 {
-  "versionInfo":  "00000001",
-  "resources":  [
+  "versionInfo": "00000001",
+  "resources": [
     {
-      "@type":  "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
-      "name":  "8080",
-      "virtualHosts":  [
+      "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
+      "name": "8080",
+      "virtualHosts": [
         {
-          "name":  "api-gateway-listener-9b9265b",
-          "domains":  [
+          "name": "api-gateway-listener-9b9265b",
+          "domains": [
             "*",
             "*:8080"
           ],
-          "routes":  [
+          "routes": [
             {
-              "match":  {
-                "prefix":  "/"
+              "match": {
+                "prefix": "/"
               },
-              "route":  {
-                "cluster":  "service.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul"
-              }
+              "route": {
+                "cluster": "service.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul"
+              },
+              "requestHeadersToAdd": [
+                {
+                  "header": {
+                    "key": "X-Header-Add",
+                    "value": "added"
+                  },
+                  "append": true
+                },
+                {
+                  "header": {
+                    "key": "X-Header-Set",
+                    "value": "set"
+                  },
+                  "append": false
+                }
+              ],
+              "requestHeadersToRemove": [
+                "X-Header-Remove"
+              ]
             }
           ]
         }
       ],
-      "validateClusters":  true
+      "validateClusters": true
     }
   ],
-  "typeUrl":  "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
-  "nonce":  "00000001"
+  "typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
+  "nonce": "00000001"
 }
diff --git a/command/agent/agent.go b/command/agent/agent.go
@@ -231,6 +231,9 @@ func (c *cmd) run(args []string) int {
 	ui.Info(fmt.Sprintf(" Gossip Encryption: %t", config.EncryptKey != ""))
 	ui.Info(fmt.Sprintf("  Auto-Encrypt-TLS: %t", config.AutoEncryptTLS || config.AutoEncryptAllowTLS))
 	ui.Info(fmt.Sprintf("       ACL Enabled: %t", config.ACLsEnabled))
+	if config.ServerMode {
+		ui.Info(fmt.Sprintf(" Reporting Enabled: %t", config.Reporting.License.Enabled))
+	}
 	ui.Info(fmt.Sprintf("ACL Default Policy: %s", config.ACLResolverSettings.ACLDefaultPolicy))
 	ui.Info(fmt.Sprintf("         HTTPS TLS: Verify Incoming: %t, Verify Outgoing: %t, Min Version: %s",
 		config.TLS.HTTPS.VerifyIncoming, config.TLS.HTTPS.VerifyOutgoing, config.TLS.HTTPS.TLSMinVersion))

diff --git a/go.mod b/go.mod
@@ -38,11 +38,11 @@ require (
 	github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
 	github.com/hashicorp/consul-awsauth v0.0.0-20220713182709-05ac1c5c2706
 	github.com/hashicorp/consul-net-rpc v0.0.0-20221205195236-156cfab66a69
-	github.com/hashicorp/consul/api v1.20.0
-	github.com/hashicorp/consul/envoyextensions v0.1.2
-	github.com/hashicorp/consul/proto-public v0.2.1
-	github.com/hashicorp/consul/sdk v0.13.1
-	github.com/hashicorp/consul/troubleshoot v0.1.2
+	github.com/hashicorp/consul/api v1.22.0-rc1
+	github.com/hashicorp/consul/envoyextensions v0.3.0-rc1
+	github.com/hashicorp/consul/proto-public v0.4.0-rc1
+	github.com/hashicorp/consul/sdk v0.14.0-rc1
+	github.com/hashicorp/consul/troubleshoot v0.3.0-rc1
 	github.com/hashicorp/go-bexpr v0.1.2
 	github.com/hashicorp/go-checkpoint v0.5.0
 	github.com/hashicorp/go-cleanhttp v0.5.2

diff --git a/website/content/docs/agent/config/config-files.mdx b/website/content/docs/agent/config/config-files.mdx
@@ -711,6 +711,10 @@ Refer to the [formatting specification](https://golang.org/pkg/time/#ParseDurati
     servers in all federated datacenters must have this enabled before any client can use
     [`use_streaming_backend`](#use_streaming_backend).
 
+- `reporting`<EnterpriseAlert inline /> - This option allows options for HashiCorp reporting.
+    - `license` - The license object allows users to control automatic reporting of license utilization metrics to HashiCorp.
+      - `enabled`: (Defaults to `true`) Enables automatic license utilization reporting.
+
 - `segment` <EnterpriseAlert inline /> - Equivalent to the [`-segment` command-line flag](/consul/docs/agent/config/cli-flags#_segment).
 
   ~> **Warning:** The `segment` option cannot be used with the [`partition`](#partition-1) option.

diff --git a/website/content/docs/agent/config/index.mdx b/website/content/docs/agent/config/index.mdx
@@ -73,6 +73,7 @@ The following agent configuration options are reloadable at runtime:
     them without a restart provides a recovery path that doesn't involve
     downtime. They generally shouldn't be changed otherwise.
 - [RPC rate limits](/consul/docs/agent/config/config-files#limits)
+- [Reporting](/consul/docs/agent/config/config-files#reporting)
 - [HTTP Maximum Connections per Client](/consul/docs/agent/config/config-files#http_max_conns_per_client)
 - Services
 - TLS Configuration