Merge branch 'main' into docs/jwt-auth-for-intentions

CHANGELOG.md

@@ -1,3 +1,70 @@
+## 1.16.0-rc1 (June 12, 2023)
+
+BREAKING CHANGES:
+
+* api: The `/v1/health/connect/` and `/v1/health/ingress/` endpoints now immediately return 403 "Permission Denied" errors whenever a token with insufficient `service:read` permissions is provided. Prior to this change, the endpoints returned a success code with an empty result list when a token with insufficient permissions was provided. [[GH-17424](https://github.com/hashicorp/consul/issues/17424)]
+* peering: Removed deprecated backward-compatibility behavior.
+Upstream overrides in service-defaults will now only apply to peer upstreams when the `peer` field is provided.
+Visit the 1.16.x [upgrade instructions](https://developer.hashicorp.com/consul/docs/upgrading/upgrade-specific) for more information. [[GH-16957](https://github.com/hashicorp/consul/issues/16957)]
+
+SECURITY:
+
+* audit-logging: **(Enterprise only)** limit `v1/operator/audit-hash` endpoint to ACL token with `operator:read` privileges.
+
+FEATURES:
+
+* api: (Enterprise only) Add `POST /v1/operator/audit-hash` endpoint to calculate the hash of the data used by the audit log hash function and salt.
+* cli: (Enterprise only) Add a new `consul operator audit hash` command to retrieve and compare the hash of the data used by the audit log hash function and salt.
+* cli: Adds new command - `consul services export` - for exporting a service to a peer or partition [[GH-15654](https://github.com/hashicorp/consul/issues/15654)]
+* connect: **(Consul Enterprise only)** Implement order-by-locality failover.
+* mesh: Add new permissive mTLS mode that allows sidecar proxies to forward incoming traffic unmodified to the application. This adds `AllowEnablingPermissiveMutualTLS` setting to the mesh config entry and the `MutualTLSMode` setting to proxy-defaults and service-defaults. [[GH-17035](https://github.com/hashicorp/consul/issues/17035)]
+* mesh: Support configuring JWT authentication in Envoy. [[GH-17452](https://github.com/hashicorp/consul/issues/17452)]
+* server: **(Enterprise Only)** added server side RPC requests IP based read/write rate-limiter. [[GH-4633](https://github.com/hashicorp/consul/issues/4633)]
+* server: **(Enterprise Only)** allow automatic license utilization reporting. [[GH-5102](https://github.com/hashicorp/consul/issues/5102)]
+* server: added server side RPC requests global read/write rate-limiter. [[GH-16292](https://github.com/hashicorp/consul/issues/16292)]
+* xds: Add `property-override` built-in Envoy extension that directly patches Envoy resources. [[GH-17487](https://github.com/hashicorp/consul/issues/17487)]
+* xds: Add a built-in Envoy extension that inserts External Authorization (ext_authz) network and HTTP filters. [[GH-17495](https://github.com/hashicorp/consul/issues/17495)]
+* xds: Add a built-in Envoy extension that inserts Wasm HTTP filters. [[GH-16877](https://github.com/hashicorp/consul/issues/16877)]
+* xds: Add a built-in Envoy extension that inserts Wasm network filters. [[GH-17505](https://github.com/hashicorp/consul/issues/17505)]
+
+IMPROVEMENTS:
+
+* * api: Support filtering for config entries. [[GH-17183](https://github.com/hashicorp/consul/issues/17183)]
+* * cli: Add `-filter` option to `consul config list` for filtering config entries. [[GH-17183](https://github.com/hashicorp/consul/issues/17183)]
+* api: Enable setting query options on agent force-leave endpoint. [[GH-15987](https://github.com/hashicorp/consul/issues/15987)]
+* audit-logging: (Enterprise only) enable error response and request body logging [[GH-5669](https://github.com/hashicorp/consul/issues/5669)]
+* audit-logging: **(Enterprise only)** enable error response and request body logging
+* ca: automatically set up Vault's auto-tidy setting for tidy_expired_issuers when using Vault as a CA provider. [[GH-17138](https://github.com/hashicorp/consul/issues/17138)]
+* ca: support Vault agent auto-auth config for Vault CA provider using AliCloud authentication. [[GH-16224](https://github.com/hashicorp/consul/issues/16224)]
+* ca: support Vault agent auto-auth config for Vault CA provider using AppRole authentication. [[GH-16259](https://github.com/hashicorp/consul/issues/16259)]
+* ca: support Vault agent auto-auth config for Vault CA provider using Azure MSI authentication. [[GH-16298](https://github.com/hashicorp/consul/issues/16298)]
+* ca: support Vault agent auto-auth config for Vault CA provider using JWT authentication. [[GH-16266](https://github.com/hashicorp/consul/issues/16266)]
+* ca: support Vault agent auto-auth config for Vault CA provider using Kubernetes authentication. [[GH-16262](https://github.com/hashicorp/consul/issues/16262)]
+* command: Adds ACL enabled to status output on agent startup. [[GH-17086](https://github.com/hashicorp/consul/issues/17086)]
+* command: Allow creating ACL Token TTL with greater than 24 hours with the -expires-ttl flag. [[GH-17066](https://github.com/hashicorp/consul/issues/17066)]
+* connect: **(Enterprise Only)** Add support for specifying "Partition" and "Namespace" in Prepared Queries failover rules.
+* connect: update supported envoy versions to 1.23.10, 1.24.8, 1.25.7, 1.26.2 [[GH-17546](https://github.com/hashicorp/consul/issues/17546)]
+* connect: update supported envoy versions to 1.23.8, 1.24.6, 1.25.4, 1.26.0 [[GH-5200](https://github.com/hashicorp/consul/issues/5200)]
+* fix metric names in /docs/agent/telemetry [[GH-17577](https://github.com/hashicorp/consul/issues/17577)]
+* gateway: Change status condition reason for invalid certificate on a listener from "Accepted" to "ResolvedRefs". [[GH-17115](https://github.com/hashicorp/consul/issues/17115)]
+* http: accept query parameters `datacenter`, `ap` (enterprise-only), and `namespace` (enterprise-only). Both short-hand and long-hand forms of these query params are now supported via the HTTP API (dc/datacenter, ap/partition, ns/namespace). [[GH-17525](https://github.com/hashicorp/consul/issues/17525)]
+* systemd: set service type to notify. [[GH-16845](https://github.com/hashicorp/consul/issues/16845)]
+* ui: Update alerts to Hds::Alert component [[GH-16412](https://github.com/hashicorp/consul/issues/16412)]
+* ui: Update to use Hds::Toast component to show notifications [[GH-16519](https://github.com/hashicorp/consul/issues/16519)]
+* ui: update from <button> and <a> to design-system-components button <Hds::Button> [[GH-16251](https://github.com/hashicorp/consul/issues/16251)]
+* ui: update typography to styles from hds [[GH-16577](https://github.com/hashicorp/consul/issues/16577)]
+
+BUG FIXES:
+
+* Fix a race condition where an event is published before the data associated is commited to memdb. [[GH-16871](https://github.com/hashicorp/consul/issues/16871)]
+* gateways: **(Enterprise only)** Fixed a bug in API gateways where gateway configuration objects in non-default partitions did not reconcile properly. [[GH-17581](https://github.com/hashicorp/consul/issues/17581)]
+* gateways: Fixed a bug in API gateways where binding a route that only targets a service imported from a peer results
+in the programmed gateway having no routes. [[GH-17609](https://github.com/hashicorp/consul/issues/17609)]
+* gateways: Fixed a bug where API gateways were not being taken into account in determining xDS rate limits. [[GH-17631](https://github.com/hashicorp/consul/issues/17631)]
+* peering: Fixes a bug where the importing partition was not added to peered failover targets, which causes issues when the importing partition is a non-default partition. [[GH-16673](https://github.com/hashicorp/consul/issues/16673)]
+* ui: fixes ui tests run on CI [[GH-16428](https://github.com/hashicorp/consul/issues/16428)]
+* xds: Fixed a bug where modifying ACLs on a token being actively used for an xDS connection caused all xDS updates to fail. [[GH-17566](https://github.com/hashicorp/consul/issues/17566)]
+
 ## 1.15.3 (June 1, 2023)
 
 BREAKING CHANGES:

GNUmakefile

@@ -337,7 +337,7 @@ fmt: $(foreach mod,$(GO_MODULES),fmt/$(mod))
 .PHONY: fmt/%
 fmt/%:
 	@echo "--> Running go fmt ($*)"
-	@cd $* && go fmt ./...
+	@cd $* && gofmt -s -l -w .
 
 .PHONY: lint
 lint: $(foreach mod,$(GO_MODULES),lint/$(mod)) lint-container-test-deps

agent/config/builder.go

@@ -828,6 +828,7 @@ func (b *builder) build() (rt RuntimeConfig, err error) {
 		Version:                    stringVal(c.Version),
 		VersionPrerelease:          stringVal(c.VersionPrerelease),
 		VersionMetadata:            stringVal(c.VersionMetadata),
+		Experiments:                c.Experiments,
 		// What is a sensible default for BuildDate?
 		BuildDate: timeValWithDefault(c.BuildDate, time.Date(1970, 1, 00, 00, 00, 01, 0, time.UTC)),
 

agent/config/config.go

@@ -183,6 +183,7 @@ type Config struct {
 	EncryptKey                       *string             `mapstructure:"encrypt" json:"encrypt,omitempty"`
 	EncryptVerifyIncoming            *bool               `mapstructure:"encrypt_verify_incoming" json:"encrypt_verify_incoming,omitempty"`
 	EncryptVerifyOutgoing            *bool               `mapstructure:"encrypt_verify_outgoing" json:"encrypt_verify_outgoing,omitempty"`
+	Experiments                      []string            `mapstructure:"experiments" json:"experiments,omitempty"`
 	GossipLAN                        GossipLANConfig     `mapstructure:"gossip_lan" json:"-"`
 	GossipWAN                        GossipWANConfig     `mapstructure:"gossip_wan" json:"-"`
 	HTTPConfig                       HTTPConfig          `mapstructure:"http_config" json:"-"`

agent/config/default.go

@@ -209,6 +209,9 @@ func DevSource() Source {
 		ports = {
 			grpc = 8502
 		}
+		experiments = [
+			"resource-apis"
+		]
 	`,
 	}
 }

agent/config/runtime.go

@@ -1498,6 +1498,9 @@ type RuntimeConfig struct {
 
 	Reporting ReportingConfig
 
+	// List of experiments to enable
+	Experiments []string
+
 	EnterpriseRuntimeConfig
 }
 

agent/config/runtime_test.go

@@ -325,6 +325,7 @@ func TestLoad_IntegrationWithFlags(t *testing.T) {
 			rt.DisableAnonymousSignature = true
 			rt.DisableKeyringFile = true
 			rt.EnableDebug = true
+			rt.Experiments = []string{"resource-apis"}
 			rt.UIConfig.Enabled = true
 			rt.LeaveOnTerm = false
 			rt.Logging.LogLevel = "DEBUG"
@@ -6355,6 +6356,7 @@ func TestLoad_FullConfig(t *testing.T) {
 		EnableRemoteScriptChecks:         true,
 		EnableLocalScriptChecks:          true,
 		EncryptKey:                       "A4wELWqH",
+		Experiments:                      []string{"foo"},
 		StaticRuntimeConfig: StaticRuntimeConfig{
 			EncryptVerifyIncoming: true,
 			EncryptVerifyOutgoing: true,

agent/config/testdata/TestRuntimeConfig_Sanitize.golden

@@ -199,6 +199,7 @@
     "EnableRemoteScriptChecks": false,
     "EncryptKey": "hidden",
     "EnterpriseRuntimeConfig": {},
+    "Experiments": [],
     "ExposeMaxPort": 0,
     "ExposeMinPort": 0,
     "GRPCAddrs": [],

agent/config/testdata/full-config.hcl

@@ -285,6 +285,9 @@ enable_syslog = true
 encrypt = "A4wELWqH"
 encrypt_verify_incoming = true
 encrypt_verify_outgoing = true
+experiments = [
+    "foo"
+]
 http_config {
     block_endpoints = [ "RBvAFcGD", "fWOWFznh" ]
     allow_write_http_from = [ "127.0.0.1/8", "22.33.44.55/32", "0.0.0.0/0" ]

agent/config/testdata/full-config.json

@@ -327,6 +327,9 @@
   "encrypt": "A4wELWqH",
   "encrypt_verify_incoming": true,
   "encrypt_verify_outgoing": true,
+  "experiments": [
+    "foo"
+  ],
   "http_config": {
     "block_endpoints": [
       "RBvAFcGD",
@@ -407,17 +410,17 @@
   "raft_snapshot_interval": "30s",
   "raft_trailing_logs": 83749,
   "raft_logstore": {
-    "backend" : "wal",
-    "disable_log_cache":  true,
+    "backend": "wal",
+    "disable_log_cache": true,
     "verification": {
-        "enabled": true,
-        "interval":"12345s"
+      "enabled": true,
+      "interval": "12345s"
     },
     "boltdb": {
-        "no_freelist_sync": true
+      "no_freelist_sync": true
     },
     "wal": {
-       "segment_size_mb": 15
+      "segment_size_mb": 15
     }
   },
   "read_replica": true,
@@ -927,4 +930,4 @@
   "xds": {
     "update_max_per_second": 9526.2
   }
-}
+}

agent/consul/options.go

@@ -39,6 +39,8 @@ type Deps struct {
 	// HCP contains the dependencies required when integrating with the HashiCorp Cloud Platform
 	HCP hcp.Deps
 
+	Experiments []string
+
 	EnterpriseDeps
 }
 

agent/consul/server.go

@@ -79,6 +79,7 @@ import (
 	raftstorage "github.com/hashicorp/consul/internal/storage/raft"
 	"github.com/hashicorp/consul/lib"
 	"github.com/hashicorp/consul/lib/routine"
+	"github.com/hashicorp/consul/lib/stringslice"
 	"github.com/hashicorp/consul/logging"
 	"github.com/hashicorp/consul/proto-public/pbresource"
 	"github.com/hashicorp/consul/proto/private/pbsubscribe"
@@ -131,6 +132,8 @@ const (
 	reconcileChSize = 256
 
 	LeaderTransferMinVersion = "1.6.0"
+
+	catalogResourceExperimentName = "resource-apis"
 )
 
 const (
@@ -807,7 +810,7 @@ func NewServer(config *Config, flat Deps, externalGRPCServer *grpc.Server, incom
 		s.internalResourceServiceClient,
 		logger.Named(logging.ControllerRuntime),
 	)
-	s.registerResources()
+	s.registerResources(flat)
 	go s.controllerManager.Run(&lib.StopChannelContext{StopCh: shutdownCh})
 
 	go s.trackLeaderChanges()
@@ -858,11 +861,14 @@ func NewServer(config *Config, flat Deps, externalGRPCServer *grpc.Server, incom
 	return s, nil
 }
 
-func (s *Server) registerResources() {
-	catalog.RegisterTypes(s.typeRegistry)
-	catalog.RegisterControllers(s.controllerManager, catalog.DefaultControllerDependencies())
+func (s *Server) registerResources(deps Deps) {
+	if stringslice.Contains(deps.Experiments, catalogResourceExperimentName) {
+		catalog.RegisterTypes(s.typeRegistry)
+		catalog.RegisterControllers(s.controllerManager, catalog.DefaultControllerDependencies())
+
+		mesh.RegisterTypes(s.typeRegistry)
+	}
 
-	mesh.RegisterTypes(s.typeRegistry)
 	reaper.RegisterControllers(s.controllerManager)
 
 	if s.config.DevMode {

agent/envoyextensions/builtin/http/localratelimit/ratelimit.go

@@ -60,6 +60,9 @@ func (r *ratelimit) fromArguments(args map[string]interface{}) error {
 	if err := mapstructure.Decode(args, r); err != nil {
 		return fmt.Errorf("error decoding extension arguments: %v", err)
 	}
+	if r.ProxyType == "" {
+		r.ProxyType = string(api.ServiceKindConnectProxy)
+	}
 	return r.validate()
 }
 
@@ -188,7 +191,7 @@ func (r ratelimit) PatchFilter(p extensioncommon.FilterPayload) (*envoy_listener
 }
 
 func validateProxyType(t string) error {
-	if t != "connect-proxy" {
+	if t != string(api.ServiceKindConnectProxy) {
 		return fmt.Errorf("unexpected ProxyType %q", t)
 	}
 

agent/envoyextensions/builtin/http/localratelimit/ratelimit_test.go

@@ -111,6 +111,30 @@ func TestConstructor(t *testing.T) {
 			expectedErrMsg: "cannot parse 'FilterEnforced', -1 overflows uint",
 			ok:             false,
 		},
+		"invalid proxy type": {
+			arguments: makeArguments(map[string]interface{}{
+				"ProxyType":     "invalid",
+				"FillInterval":  30,
+				"MaxTokens":     20,
+				"TokensPerFill": 5,
+			}),
+			expectedErrMsg: `unexpected ProxyType "invalid"`,
+			ok:             false,
+		},
+		"default proxy type": {
+			arguments: makeArguments(map[string]interface{}{
+				"FillInterval":  30,
+				"MaxTokens":     20,
+				"TokensPerFill": 5,
+			}),
+			expected: ratelimit{
+				ProxyType:     "connect-proxy",
+				MaxTokens:     intPointer(20),
+				FillInterval:  intPointer(30),
+				TokensPerFill: intPointer(5),
+			},
+			ok: true,
+		},
 		"valid everything": {
 			arguments: makeArguments(map[string]interface{}{
 				"ProxyType":     "connect-proxy",

agent/envoyextensions/builtin/lua/lua.go

@@ -45,6 +45,9 @@ func (l *lua) fromArguments(args map[string]interface{}) error {
 	if err := mapstructure.Decode(args, l); err != nil {
 		return fmt.Errorf("error decoding extension arguments: %v", err)
 	}
+	if l.ProxyType == "" {
+		l.ProxyType = string(api.ServiceKindConnectProxy)
+	}
 	return l.validate()
 }
 
@@ -53,7 +56,7 @@ func (l *lua) validate() error {
 	if l.Script == "" {
 		resultErr = multierror.Append(resultErr, fmt.Errorf("missing Script value"))
 	}
-	if l.ProxyType != "connect-proxy" {
+	if l.ProxyType != string(api.ServiceKindConnectProxy) {
 		resultErr = multierror.Append(resultErr, fmt.Errorf("unexpected ProxyType %q", l.ProxyType))
 	}
 	if l.Listener != "inbound" && l.Listener != "outbound" {

agent/envoyextensions/builtin/lua/lua_test.go

@@ -54,6 +54,15 @@ func TestConstructor(t *testing.T) {
 			arguments: makeArguments(map[string]interface{}{"Listener": "invalid"}),
 			ok:        false,
 		},
+		"default proxy type": {
+			arguments: makeArguments(map[string]interface{}{"ProxyType": ""}),
+			expected: lua{
+				ProxyType: "connect-proxy",
+				Listener:  "inbound",
+				Script:    "lua-script",
+			},
+			ok: true,
+		},
 		"valid everything": {
 			arguments: makeArguments(map[string]interface{}{}),
 			expected: lua{

agent/envoyextensions/builtin/property-override/property_override.go

@@ -261,6 +261,9 @@ func (p *propertyOverride) validate() error {
 		}
 	}
 
+	if p.ProxyType == "" {
+		p.ProxyType = api.ServiceKindConnectProxy
+	}
 	if err := validProxyTypes.CheckRequired(string(p.ProxyType), "ProxyType"); err != nil {
 		resultErr = multierror.Append(resultErr, err)
 	}

agent/envoyextensions/builtin/property-override/property_override_test.go

@@ -236,7 +236,7 @@ func TestConstructor(t *testing.T) {
 		// enforces expected behavior until we do. Multi-member slices should be unaffected
 		// by WeakDecode as it is a more-permissive version of the default behavior.
 		"single value Patches decoded as map construction succeeds": {
-			arguments: makeArguments(map[string]any{"Patches": makePatch(map[string]any{})}),
+			arguments: makeArguments(map[string]any{"Patches": makePatch(map[string]any{}), "ProxyType": nil}),
 			expected:  validTestCase(OpAdd, extensioncommon.TrafficDirectionOutbound, ResourceTypeRoute).expected,
 			ok:        true,
 		},

agent/setup.go

@@ -73,6 +73,7 @@ func NewBaseDeps(configLoader ConfigLoader, logOut io.Writer, providedLogger hcl
 		return d, err
 	}
 	d.WatchedFiles = result.WatchedFiles
+	d.Experiments = result.RuntimeConfig.Experiments
 	cfg := result.RuntimeConfig
 	logConf := cfg.Logging
 	logConf.Name = logging.Agent

agent/xds/resources_test.go

@@ -415,6 +415,19 @@ func getAPIGatewayGoldenTestCases(t *testing.T) []goldenTestCase {
 						Kind: structs.HTTPRoute,
 						Name: "route",
 						Rules: []structs.HTTPRouteRule{{
+							Filters: structs.HTTPFilters{
+								Headers: []structs.HTTPHeaderFilter{
+									{
+										Add: map[string]string{
+											"X-Header-Add": "added",
+										},
+										Set: map[string]string{
+											"X-Header-Set": "set",
+										},
+										Remove: []string{"X-Header-Remove"},
+									},
+								},
+							},
 							Services: []structs.HTTPService{{
 								Name: "service",
 							}},