Hull shader input: VtxOutputGeo0 not array when it should be.

[vk2gpu]

When working on getting triangle tessellation to work, I noticed a problem when doing a simple pass through for the control points that doesn't appear to be a problem with quads on my driver (though the code generation bug still exists).

Unless you are modifying the control points, VtxOutputGeo0 is not declared as an array (3 for tri, 4 for quad). This causes a link error for me. To work around this whilst testing things I modified the shader I was using to simply multiply the control points by a value to ensure their modification, VtxOutputGeo0 in this case was declared as an array of size 3. The code that works is as follows:

struct IA_OUTPUT
{
    float4 cpoint : CPOINT;
    float4 colour : COLOR;
};

struct VS_OUTPUT
{
    float4 cpoint : SV_Position;
    float4 colour : COLOR;
};

matrix World;
matrix View;
matrix Projection;

VS_OUTPUT VS(IA_OUTPUT input)
{
    VS_OUTPUT output;

    output.cpoint = mul( input.cpoint, World );
    output.cpoint = mul( output.cpoint, View );
    output.cpoint = mul( output.cpoint, Projection );

    output.colour = input.colour;
    return output;
}

struct HS_CONSTANT_OUTPUT
{
    float edges[3] : SV_TessFactor;
    float innerEdges: SV_InsideTessFactor;
};

struct HS_OUTPUT
{
    float4 cpoint : SV_Position;
    float4 colour : COLOR;
};

float InnerFactor = 1.0f;
float OuterFactor = 1.0f;

HS_CONSTANT_OUTPUT HSConst()
{
    HS_CONSTANT_OUTPUT output;

    output.innerEdges = InnerFactor;
    output.edges[0] = OuterFactor;
    output.edges[1] = OuterFactor;
    output.edges[2] = OuterFactor;

    return output;
}

[domain("tri")]
[partitioning("integer")]
[outputtopology("triangle_ccw")]
[outputcontrolpoints(3)]
[patchconstantfunc("HSConst")]
HS_OUTPUT HS(InputPatch<VS_OUTPUT, 3> ip, uint id : SV_OutputControlPointID)
{
    HS_OUTPUT output;
    output.cpoint = ip[id].cpoint * 1.01f;
    output.colour = ip[id].colour * 1.01f;
    return output;
}

struct DS_OUTPUT
{
    float4 position : SV_Position;
    float4 colour : COLOR;
};


float4 interpolate(float3 TessCoord, float4 v0, float4 v1, float4 v2 )
{
    float4 a = TessCoord.x * v0;
    float4 b = TessCoord.y * v1;
    float4 c = TessCoord.z * v2;
    return a + b + c;
}

[domain("tri")]
DS_OUTPUT DS(HS_CONSTANT_OUTPUT input, OutputPatch<HS_OUTPUT, 3> op, float3 uv : SV_DomainLocation)
{
    DS_OUTPUT output;

    output.position = interpolate(uv, op[0].cpoint, op[1].cpoint, op[2].cpoint);

    output.colour = interpolate(uv, op[0].colour, op[1].colour, op[2].colour);

    return output;
}


float4 PS(DS_OUTPUT input) : SV_Target0
{
    return input.colour;
}

The code that shows the bug is as follows:

struct IA_OUTPUT
{
    float4 cpoint : CPOINT;
    float4 colour : COLOR;
};

struct VS_OUTPUT
{
    float4 cpoint : SV_Position;
    float4 colour : COLOR;
};

matrix World;
matrix View;
matrix Projection;

VS_OUTPUT VS(IA_OUTPUT input)
{
    VS_OUTPUT output;

    output.cpoint = mul( input.cpoint, World );
    output.cpoint = mul( output.cpoint, View );
    output.cpoint = mul( output.cpoint, Projection );

    output.colour = input.colour;
    return output;
}

struct HS_CONSTANT_OUTPUT
{
    float edges[3] : SV_TessFactor;
    float innerEdges: SV_InsideTessFactor;
};

struct HS_OUTPUT
{
    float4 cpoint : SV_Position;
    float4 colour : COLOR;
};

float InnerFactor = 1.0f;
float OuterFactor = 1.0f;

HS_CONSTANT_OUTPUT HSConst()
{
    HS_CONSTANT_OUTPUT output;

    output.innerEdges = InnerFactor;
    output.edges[0] = OuterFactor;
    output.edges[1] = OuterFactor;
    output.edges[2] = OuterFactor;

    return output;
}

[domain("tri")]
[partitioning("integer")]
[outputtopology("triangle_ccw")]
[outputcontrolpoints(3)]
[patchconstantfunc("HSConst")]
HS_OUTPUT HS(InputPatch<VS_OUTPUT, 3> ip, uint id : SV_OutputControlPointID)
{
    HS_OUTPUT output;
    output.cpoint = ip[id].cpoint;
    output.colour = ip[id].colour;
    return output;
}

struct DS_OUTPUT
{
    float4 position : SV_Position;
    float4 colour : COLOR;
};


float4 interpolate(float3 TessCoord, float4 v0, float4 v1, float4 v2 )
{
    float4 a = TessCoord.x * v0;
    float4 b = TessCoord.y * v1;
    float4 c = TessCoord.z * v2;
    return a + b + c;
}

[domain("tri")]
DS_OUTPUT DS(HS_CONSTANT_OUTPUT input, OutputPatch<HS_OUTPUT, 3> op, float3 uv : SV_DomainLocation)
{
    DS_OUTPUT output;

    output.position = interpolate(uv, op[0].cpoint, op[1].cpoint, op[2].cpoint);

    output.colour = interpolate(uv, op[0].colour, op[1].colour, op[2].colour);

    return output;
}


float4 PS(DS_OUTPUT input) : SV_Target0
{
    return input.colour;
}

[James-Jones]

If you do nothing to the control points then fxc will not generate a control point phase unless you compile with /Od. So your HLSL has an assignment instructions but fxc ignores them and will be missing from the bytecode. This array size to use for VtxOutputGeo0 is simply not known. The driver has to generate the missing assignments, but does not know how many generate until draw time because it needs to know what the domain shader inputs are.

A similar thing exists in opengl whereby you can have a evaluation shader without a control shader and the driver will generate one internally for you - although that is not allowed in the ES extension. So you might want to use this awful feature and not use a hull shader, just set GL_PATCH_DEFAULT_INNER_LEVEL and GL_PATCH_DEFAULT_OUTER_LEVEL.

It should be possible for HLSLcc to be changed to handle this scenario if you guarantee that the domain shader will be compiled first and then information from that shader can be saved into the GLSLCrossDependencyData struct and used when compiling the hull shader.

[vk2gpu]

I thought as much. In the example given VtxOutputGeo0 is not actually used, a thought I had was to just not write it out into the GLSL? Used cross dependency data sounds good too, though in this particular case it seems like it would have no gain, correct me if I'm wrong of course.

[James-Jones]

I believe if you have a control shader then it must write gl_Position otherwise you are feeding uninitialized positions to subsequent stages. I have pushed a change to remove the unused input variable.