specify / implement skinning

[fabrobinet]

No description provided.

[fabrobinet]

I suggest important changes for the skinning proposal.

1. flatten the datas - get rid of weight indices coming from COLLADA (but of course keep bone ids).
   -> having varying number of influences per vertex do not map well at all in hardware accelerated rendering.
2. have a fix number of influences per vertex. (force "0" weight when necessary to fill gaps).

Doing so would allow us to use vertex attributes for both weights and bone indices. They could be anything between FLOAT and FLOAT_VEC4 .And SHORT & SHORT_VEC4 for bone indices.
That is a pragmatic approach because it is what people end up doing in code and it would integrate nicely with the existing SPEC.

Typically, for hardware rendering (in every examples I have seen and in my experience) the number of influences limit is 4. (to fit a FLOAT_VEC4).
But we could actually handle buckets of 4 influences in FLOAT_VEC4 attributes (and less) and not be limited in the number of influences.
The converter would have an option to restrict to 4 the number of influence for optimal performances.

It could be thought that one drawback of this approach would be to waste space because attributes of fixed size.
It's not so obvious, because we would get rid of indices, and thus this "waste" due to fixed size would be compensated (compared with the input data from COLLADA).

Last but not least, matrixes for bones must be stored in uniform memory, but this one is limited.
We should split meshes according to a maximum number of bones per primitive for destination API.
The minimum uniform memory is specified in GL SPEC's but can be extended at will by vendors.
So I suggest that by default the converter would split according to the known minimum uniform memory,
but another value could be forced to adapt the splitting for destination hardware. Maybe… this should be set by an extension.

[tparisi]

I am totally OK with #2 - fixed number of influence. 4 is typical. Let's do that.

I have no opinion on #1... Remi seems to think it's ok

[fabrobinet]

@tparisi @RemiArnaud @pjcozzi this is what we have for skinning.
I initially started with Tony's proposal but handling skinning for GL based API needed to re-organize datas.
This is classic setup for hardware skinning and does not prevent to do software skinning for the ones who wants to.

Beginning with skin property in skins…
So we don't have a controllers and controller with skin or morph inside, it's just simpler this way because the skin property would have to be there anyway.

    "skins": {
        "Figure_2-skin": {
            "bindShapeMatrix": [
                1,
                0,
                0,
                0,
                0,
                1,
                0,
                0,
                0,
                0,
                1,
                0,
                0,
                0,
                0,
                1
            ],
            "inverseBindMatrices": {
                "bufferView": "bufferView_1372",
                "byteOffset": 0,
                "count": 18,
                "type": "FLOAT_MAT4"
            },
            "joints": [
                "Figure_2hip_2_node",
                "Figure_2chest_2_node",
                "Figure_2head_2_node",
                "Figure_2rShldr_2_node",
                "Figure_2rForeArm_2_node",
                "Figure_2lShldr_2_node",
                "Figure_2lForeArm_2_node",
                "Figure_2lMid1_2_node",
                "Figure_2lPinky1_2_node",
                "Figure_2lThumb1_2_node",
                "Figure_2rThigh_2_node",
                "Figure_2rShin_2_node",
                "Figure_2rFoot_2_node",
                "Figure_2rToe_2_node",
                "Figure_2lThigh_2_node",
                "Figure_2lShin_2_node",
                "Figure_2lFoot_2_node",
                "Figure_2lToe_2_node"
            ]
        }
    },

Then how do we use this ?

In node we have an instanceSkin

            "instanceSkin": {
                "skeletons": [
                    "Figure_2BODY_2_node"
                ],
                "skin": "Figure_2-skin",
                "sources": [
                    "Figure_2_geometry"
                ]
            },

As discussed earlier then, we adopted an hardware rendering friendly approach with the weights and joints indices passed as attributes to the mesh. This simpler approach fits nicely in the existing design. (so we got rid of weights indices).
Note that in this example we are using a maximum of 4 influences per vertex but by adding another attribute we could reach more.

Then a mesh is defined this way (note WEIGHT and JOINT):

    "meshes": {
        "Figure_2_geometry": {
            "name": "Figure_2_geometry",
            "primitives": [
                {
                    "indices": "indices_1006",
                    "material": "material.0",
                    "primitive": "TRIANGLES",
                    "semantics": {
                        "JOINT": "attribute_1366",
                        "NORMAL": "attribute_1184",
                        "POSITION": "attribute_1182",
                        "TEXCOORD_0": "attribute_1186",
                        "WEIGHT": "attribute_1363"
                    }
                },

and finally, in the technique we a special semantic "JOINT_MATRIX" that is skinning matrix that is the concatenation of all the matrices needed for skinning, note that the way this matrix is implemented is up the adopter and thus is mostly of use for the adopter who want to use the provided shaders.

    "techniques": {
        "technique1": {
            "parameters": {
                "jointMat": {
                    "semantic": "JOINT_MATRIX",
                    "type": "FLOAT_MAT4"
                },

Last but not least, all this is implemented in the converter and in montage viewer as well.

Next thing to be discussed is how to handle a maximum number of bones per mesh, this is planned and I will complete the proposal once this is implemented and tested.

But what we have should be enough to start updating the specification.

[fabrobinet]

@kmammou FYI: this is my proposal for skinning, as you see weights and bone indices are attributes (say stream) and should fit well in your compression.

[tparisi]

@fabrobinet this looks great. Simpler is always better!

So what do weights and joints indices look like? Are you going with FLOAT and FLOAT_VEC4, SHORT and SHORT_VEC4? Do you have an example we can look at?

[fabrobinet]

I will send an example over email.

weights and joints are both FLOAT_VEC4. i.e 4 weights * 4 influences (index to retrieve in the matrix array).
due to attribute constraints on OpenGL/ES 2.0. But they could be SHORT_VEC4 on desktop. These parameters will change depending on profile.

[pjcozzi]

@fabrobinet with the exception of what is to come with #164, have there been any changes to skinning since your above example?

[fabrobinet]

I think it is up to date, but there are still some cases I need to implement and provide input for, and check implication on the spec if any when:

• a mesh should be split because the number of uniforms (due matrices array) is too big to be processed
• a skinned mesh is split due to number of vertices > 65536
• should be possible to handle a number of influences > 4, it is just a matter of adding one attribute

[pjcozzi]

should be possible to handle a number of influences > 4, it is just a matter of adding one attribute

I know you are @RemiArnaud discussed this, but if this adds any complexity at all, I would skip it; I'm not sure that anyone cares in practice.

[RemiArnaud]

I noticed that inverseBindMatrices in skins is an accessor.
All the accessors so far have been grouped in the accessors object, so for uniformity, the inverseBindMatrices should be an accessorID instead?

"inverseBindMatrices": {
                "bufferView": "bufferView_61",
                "byteOffset": 0,
                "count": 7,
                "type": 35676
            },

[pjcozzi]

@RemiArnaud yes. See #238. Please search the issues.

[RemiArnaud]

thanks !

Another question: how do I know the size of JOINT_MATRIX ?
I don't find that information in the glTF json

Especially in the case there are multiple meshes.
I assume that we have multiple meshes if a skin has to split the JOINT_MATRIX uniform?

[pjcozzi]

how do I know the size of JOINT_MATRIX ?

It's length is (at least) the number of inverse-bind matrices. The generated shader can be optimized, see #183.

I assume that we have multiple meshes if a skin has to split the JOINT_MATRIX uniform?

Yes. @fabrobinet will know if this is already implemented in the converter.

[RemiArnaud]

I talked to Fabrice about this, the size is in a completely different part of the json - can't allocate the uniform by reading the technique. We should add a size to uniforms, when different than 1. Opening a new issue to track this.

[pjcozzi]

@fabrobinet is this resolved for 1.0?

[fabrobinet]

we can close this, later points in this issue are all tracked by individual issues already

[meshula]

Some years have passed, and the four weight limitation is an issue. Six weights is common, and our use case has eight. Unreal currently supports a maximum of eight. Is there an open issue, or is this the right place to discuss it? I'm not asking for a variable number of influences per vertex, only that the number of influences be declared in the gltf file.

[lexaknyazev]

@meshula See #878.

[meshula]

Thanks