Flatout 3 has incorrect rendering

[XaeroVincent]

When testing Direct3D 9 game, Flatout 3: Chaos & Destruction under DXVK, the graphics are rendered very dark and with incorrect colors. The game is rendered correctly with WineD3D.

The game doesn't run out of the box under Proton. At least with my system, it's necessary to disable Fsync and Esync (to prevent a game freeze at the loading screen) and install a native DLL override of d3dcompiler_47 (to actually see the graphics)

Screenshots:

DXVK:

WineD3D:

Apitrace file:
https://1drv.ms/u/s!Ah8jJX7TFptmijSxVdpRGgknvvU9?e=sCwrHc

[K0bin]

The game enables alpha to coverage once at startup and never disables it. It uses the ATI ATOC hack in the apitrace.
Unlike the Nvidia one which is only active if alpha test is active as well, our implementation of the ATI one has no additional constraints.

The documentation doesn't mention any either:

So now we'll have to figure out how the ATI ATOC hack works exactly.
There's a few conditions I could imagine:

• it's only enabled when alpha test is enabled. (like the NV one, Josh says he tested that ages ago though and it doesn't behave like that.)
• it's only enabled when rendering to a multisampled RT.
• it's only enabled when blending is not enabled. ¯\_(ツ)_/¯

Time to write some tests...