Prevent overflow in lua_newuserdatadtor

[XmiliaH]

In case a large userdata size is passed to lua_newuserdatadtor it might overflow the size resulting in luaU_newudata actually allocating the object without a memory error. This will then result in overwriting part of the metatable pointer of the userdata.

This PR fixes this issue by checking for the overflow and in such cases pass a size value which will cause a memory error in luaU_newudata.

[zeux]

size should probably have been int given that we don't support userdata above that anyway... ah well. Thanks!