Kubernetes deployment issues

[Briaker]

Hi all, I've spent the better part of my weekend trying to get a server running in my Kubernetes cluster. But I can't get the server to start.

Currently I can get the logs as far as LogAIModule: Creating AISystem for world DedicatedserverEntry but it always stops there.

If anyone has any idea what might be causing this, please let me know. I've been bashing my head against my keyboard for a while trying to get this going...

Thanks.

Logs

After deleting /config:

...
[2021.10.31-23.51.36:221][  0]LogOnline: OSS: Creating online subsystem instance for: STEAM
[2021.10.31-23.51.36:221][  0]LogOnline: Warning: STEAM: Steam API failed to initialize!
[2021.10.31-23.51.36:221][  0]LogOnline: Display: STEAM: OnlineSubsystemSteam::Shutdown()
[2021.10.31-23.51.36:221][  0]LogOnline: OSS: Creating online subsystem instance for: STEAM
[2021.10.31-23.51.36:221][  0]LogOnline: Warning: STEAM: Steam API failed to initialize!
[2021.10.31-23.51.36:221][  0]LogOnline: Display: STEAM: OnlineSubsystemSteam::Shutdown()
[2021.10.31-23.51.36:221][  0]LogOnline: OSS: Creating online subsystem instance for: STEAM
[2021.10.31-23.51.36:221][  0]LogOnline: Warning: STEAM: Steam API failed to initialize!
[2021.10.31-23.51.36:221][  0]LogOnline: Display: STEAM: OnlineSubsystemSteam::Shutdown()
[2021.10.31-23.51.36:221][  0]LogServerQuery: Binding Server Query Socket to port 15777
[2021.10.31-23.51.36:221][  0]LogStreaming: Warning: Failed to read file '/home/steam/.config/Epic/FactoryGame/Saved/SaveGames/ServerSettings.15777' error.
[2021.10.31-23.51.36:221][  0]LogGame: MaxNumBackupsaves: 25
[2021.10.31-23.51.36:222][  0]GameAnalyticsModule: Display: Loaded GUID 7C536CA9-F4C0-409A-9AB4-021B3A22F2E7
[2021.10.31-23.51.36:222][  0]LogOnline: Warning: OSS: [EOS SDK | Plugin] Begin Player Session...
[2021.10.31-23.51.36:222][  0]LogInit: Display: Game Engine Initialized.
[2021.10.31-23.51.36:222][  0]LogNetVersion: Set ProjectVersion to ++FactoryGame+main-CL-170636. Version Checksum will be recalculated on next use.
[2021.10.31-23.51.36:222][  0]LogConsoleResponse: Display: Can't switch to RHI thread mode when we are not running a multithreaded renderer.
[2021.10.31-23.51.36:222][  0]LogConsoleResponse: Display: RHIThread is disabled.
[2021.10.31-23.51.36:222][  0]LogInit: OnPostEngineInit has been broadcasted successfully.
[2021.10.31-23.51.36:224][  0]LogInit: Display: Starting Game.
[2021.10.31-23.51.36:224][  0]LogNet: Browse: /Game/FactoryGame/Map/DedicatedserverEntry?name=Host?sessionName=SatisfactoryServer?Visibility=SV_FriendsOnly?loadgame=savefile?listen?bUseIpSockets
[2021.10.31-23.51.36:224][  0]LogLoad: LoadMap: /Game/FactoryGame/Map/DedicatedserverEntry?name=Host?sessionName=SatisfactoryServer?Visibility=SV_FriendsOnly?loadgame=savefile?listen?bUseIpSockets

After a reboot it gets past LoadMap but then stops shortly after

After restarting the container:

...
[2021.10.31-23.56.32:790][  0]LogNetVersion: Set ProjectVersion to ++FactoryGame+main-CL-170636. Version Checksum will be recalculated on next use.
[2021.10.31-23.56.32:790][  0]LogConsoleResponse: Display: Can't switch to RHI thread mode when we are not running a multithreaded renderer.
[2021.10.31-23.56.32:790][  0]LogConsoleResponse: Display: RHIThread is disabled.
[2021.10.31-23.56.32:790][  0]LogInit: OnPostEngineInit has been broadcasted successfully.
[2021.10.31-23.56.32:792][  0]LogInit: Display: Starting Game.
[2021.10.31-23.56.32:792][  0]LogNet: Browse: /Game/FactoryGame/Map/DedicatedserverEntry?name=Host?sessionName=SatisfactoryServer?Visibility=SV_FriendsOnly?loadgame=savefile?listen?bUseIpSockets
[2021.10.31-23.56.32:792][  0]LogLoad: LoadMap: /Game/FactoryGame/Map/DedicatedserverEntry?name=Host?sessionName=SatisfactoryServer?Visibility=SV_FriendsOnly?loadgame=savefile?listen?bUseIpSockets
[2021.10.31-23.56.32:792][  0]LogWorld: BeginTearingDown for /Temp/Untitled_0
[2021.10.31-23.56.32:793][  0]LogWorld: UWorld::CleanupWorld for Untitled, bSessionEnded=true, bCleanupResources=true

I've also tried creating a customized docker image [ github | dockerhub ] that changes LocalMapOptions in Engine.ini to LocalMapOptions=?name=Player. I was trying to simplify the config as much as I could to get it started.

This got me a little bit further to LogAIModule...

...
[2021.11.01-00.12.11:415][  0]LogNetVersion: Set ProjectVersion to ++FactoryGame+main-CL-170636. Version Checksum will be recalculated on next use.
[2021.11.01-00.12.11:415][  0]LogConsoleResponse: Display: Can't switch to RHI thread mode when we are not running a multithreaded renderer.
[2021.11.01-00.12.11:415][  0]LogConsoleResponse: Display: RHIThread is disabled.
[2021.11.01-00.12.11:416][  0]LogInit: OnPostEngineInit has been broadcasted successfully.
[2021.11.01-00.12.11:418][  0]LogInit: Display: Starting Game.
[2021.11.01-00.12.11:418][  0]LogNet: Browse: /Game/FactoryGame/Map/DedicatedserverEntry?name=Player
[2021.11.01-00.12.11:418][  0]LogLoad: LoadMap: /Game/FactoryGame/Map/DedicatedserverEntry?name=Player
[2021.11.01-00.12.11:418][  0]LogWorld: BeginTearingDown for /Temp/Untitled_0
[2021.11.01-00.12.11:418][  0]LogWorld: UWorld::CleanupWorld for Untitled, bSessionEnded=true, bCleanupResources=true
[2021.11.01-00.12.11:420][  0]LogUObjectHash: Compacting FUObjectHashTables data took   1.44ms
[2021.11.01-00.12.11:457][  0]LogAIModule: Creating AISystem for world DedicatedserverEntry

Other info:

Allocated resources:
Ram: 12GB
CPU: 10000M (10 cpu cores, 12 available)

Deployment config:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "43"
    meta.helm.sh/release-name: satisfactory-5
    meta.helm.sh/release-namespace: satisfactory
  generation: 99
  labels:
    app: satisfactory-5-satisfactory
    app.kubernetes.io/managed-by: Helm
    chart: satisfactory-0.0.15
    heritage: Helm
    io.cattle.field/appId: satisfactory-5
    release: satisfactory-5
    run: satisfactory-5-satisfactory
  name: satisfactory-5-satisfactory
  namespace: satisfactory
  resourceVersion: "158293324"
  selfLink: /apis/apps/v1/namespaces/satisfactory/deployments/satisfactory-5-satisfactory
  uid: 737041e2-6c23-4cd4-952e-1dbb28f2b021
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: satisfactory-5-satisfactory
      io.cattle.field/appId: satisfactory-5
      run: satisfactory-5-satisfactory
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: satisfactory-5-satisfactory
        io.cattle.field/appId: satisfactory-5
        run: satisfactory-5-satisfactory
    spec:
      containers:
      - env:
        - name: DEBUG
          value: "false"
        - name: MAXPLAYERS
          value: "12"
        - name: PGID
          value: "1001"
        - name: PUID
          value: "1001"
        - name: SKIPUPDATE
          value: "false"
        - name: STEAMBETA
          value: "false"
        image: briaker/satisfactory-server:latest
        imagePullPolicy: Always
        name: satisfactory-5-satisfactory
        ports:
        - containerPort: 7777
          name: game
          protocol: UDP
        - containerPort: 15000
          name: beacon
          protocol: UDP
        - containerPort: 15777
          name: query
          protocol: UDP
        resources:
          requests:
            cpu: "10"
            memory: 12Gi
        securityContext: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config
          name: configdir
      dnsPolicy: ClusterFirst
      nodeName: gameframe-major
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: configdir
        persistentVolumeClaim:
          claimName: satisfactory-5-claim

[wolveix]

Hey! I work with Kubernetes on a daily basis, so hopefully I can help you figure this out.

Firstly, are you using the early access or the experimental version of the game? The main Docker images currently only work with update 5/experimental. If you're on early access, you need to use the v4 branch.

If you're using experimental, then we can proceed to troubleshoot. Are you exposing the deployment through a service/ingress? Or are you connecting to the cluster through the same local network?

What happens when you enter the server IP into the server manager in-game?

[Briaker]

I'm using the experimental version. The deployment is

To a give a bit of context, I've setup a bare metal kubernetes cluster that has a customized nip.io/xip.io style DNS service that redirects wildcard subdomains the cluster. (i.e: hello.mydomain.com -> cluster app exposed on hello.mydomain.com). So far I've used this with other game servers and it's been working fine. As far as I can tell the port 15777 is reachable from outside the network using nmap, but I'm not a network expert so I could be wrong. And to add another layer of complexity the cluster managed using Rancher, with custom Rancher charts and questions to deploy different game servers.

The current deployment is exposed with nodeName: satisfactory.mydomain.com... and as I was writing this I realized completely missed the service deployment config template when I copied it from another project... so let me just add that and Ill get back to this. I also realize that I may need to change the --service-node-port-range to allow for the new ports.

[Briaker]

I added the service config, but I'm running into issues with the port numbers being out of range of the default service port numbers (--service-node-port-range=25000-32767).
I might try to set the port numbers using the cli arguments from the wiki, but the notes there mention that the Beacon port can't be changed...

Right now it's set to use NodePort:

# svc-template.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ template "satisfactory.fullname" . }}
  labels:
    app: {{ template "satisfactory.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
spec:
  type: NodePort
  {{- if .Values.satisfactory.externalTrafficPolicy }}
  externalTrafficPolicy: {{ .Values.satisfactory.externalTrafficPolicy }}
  {{- end }}
  ports:
  - name: game
    port: 7777
    targetPort: game
    nodePort: 7777
    protocol: UDP
  - name: beacon
    port: 15000
    targetPort: beacon
    nodePort: 15000
    protocol: UDP
  - name: query
    port: 15777
    targetPort: query
    nodePort: 15777
    protocol: UDP
  selector:
    app: {{ template "satisfactory.fullname" . }}
    run: {{ template "satisfactory.fullname" . }}
  {{- if .Values.satisfactory.externalIPs }}
  externalIPs:
    {{- with .Values.satisfactory.externalIPs }}
      {{- range . }}
        - {{ . | quote }}
      {{- end }}
    {{- end }}
  {{- end }}

[Briaker]

I changed the service ports in /etc/kubernetes/manifests/kube-apiserver.yaml to --service-node-port-range=7000-32767 and the service was successfully created, but there was no change in the satisfactory logs. Still stopping on Creating AISystem for world DedicatedserverEntry.

[2021.11.02-03.46.26:209][  0]LogInit: Display: Starting Game.
[2021.11.02-03.46.26:209][  0]LogNet: Browse: /Game/FactoryGame/Map/DedicatedserverEntry?name=Player
[2021.11.02-03.46.26:209][  0]LogLoad: LoadMap: /Game/FactoryGame/Map/DedicatedserverEntry?name=Player
[2021.11.02-03.46.26:209][  0]LogWorld: BeginTearingDown for /Temp/Untitled_0
[2021.11.02-03.46.26:209][  0]LogWorld: UWorld::CleanupWorld for Untitled, bSessionEnded=true, bCleanupResources=true
[2021.11.02-03.46.26:211][  0]LogUObjectHash: Compacting FUObjectHashTables data took   1.38ms
[2021.11.02-03.46.26:243][  0]LogAIModule: Creating AISystem for world DedicatedserverEntry

I'm gonna assume that the part my server is stuck on is not reliant on having the network setup correct?

[wolveix]

Sounds like a fun setup :) The log stopping there shouldn't really be an issue to be honest. In general, as long as the container isn't stopping/restarting, then it's working fine. Are you still unable to reach it from the in-game server browser?