Can't sync VirtualGateway and GatewayRoute using Kustomize and Fluxv2

[mikestef9]

Describe the bug
The App Mesh webhook prevents GatewayRoute resource from being created before VirtualGateway is present.

Steps to reproduce

• Install Flux v2 and configure to sync to a Git repository.
• Create a folder structure like below in the Git repository.

ingress-gw

• kustomization.yaml
• gateway-route.yaml
• virtual-gateway.yaml

Contents of kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - virtual-gateway.yaml
  - gateway-route.yaml

When Flux syncs this folder to the cluster (behind the scenes Flux always Kustomize to generate the raw YAML to apply), it generates an order where the GatewayRoute is applied first. Even if you put the VirtualGateway and GatewayRouter in the same file, separated by --- with VirtualGateway resource first, the same behavior applies.

Following error is output

Error from server (failed to find matching virtualGateway for gatewayRoute: gateway-route-paths, expecting 1 but found 0): error when creating "virtual-gateway.yaml": admission webhook "mgatewayroute.appmesh.k8s.aws" denied the request: failed to find matching virtualGateway for gatewayRoute: gateway-route-paths, expecting 1 but found 0

Expected outcome
I expect the order of VirtualGateway and VirtualRouter being applied not to matter, and the resources to be applied successfully to the cluster.

Environment

• App Mesh controller version 1.4.2
• Envoy version v1.20.0.1-prod
• Are you using any integrations? X-ray, Jaeger etc. If so versions? No
• Kubernetes version 1.21
• Using EKS (yes/no), if so version? Yes, EKS 1.21

Additional Context:
You can workaround this using Fluxv2 Kustomize dependsOn feature, making sure VirtualGateway gets applied in a Kustomization before another Kustomization that has the GatewayRoute, but it adds considerable burden on the user, when declarative manifests should just work.

[cgchinmay]

Was able to repro the issue, kustomize has certain ordering rules and that makes VirtualGateway to always appear after gatewayroute which causes this issue. Change on the controller side won't be trivial and will require a design change. For now we will have to use the work around.

[cgchinmay]

Closing this issue as we dont have any immediate plans to address this in the controller. We will have to use the suggested workaround for kustomize

[amcginlay]

This problem also emerges when one tries to deploy meshed resources via Helm.

IMO we have a precedent for how it should work. Compare, for example, what happens when you create a VirtualService which depends upon a currently unknown VirtualNode. In this case the controller admits the VirtualService and keeps it in a “pending” state until the VirtualNode is “running”.

The same behaviour should exist between VirtualGateway and GatewayRoute. The dependency should be resolved, eventually, rather than rejected prematurely.

[ar4096]

Good point @amcginlay. We will look into how the controller is resolving this issue in the VirtualService reconciliation loop.