Fix mach hardware exception forwarding (#105003)

* Fix mach hardware exception forwarding

A recent change to enable AVX512 register state processing in the mach
exception handling on macOS x64 has broken a PAL test and also a case
when a hardware exception occurs in 3rd party native code.
In both cases runtime hangs, another exception occurs while forwarding
the exception message and that ends up happenning infinitely.

The problem is caused by the fact that the MachMessageInfo has grown,
since we've changed a field containing float state to contain avx512
state instead. But the buffer that stores the message is of fixed size
of 1500 bytes and it is not sufficient.

I have grown the buffer size to the necessary size, but there was another
issue lurking behind the first one. The MachExceptionInfo::RestoreState
was trying to restore the float state always as x86_FLOAT_STATE instead
of choosing x86_FLOAT_STATE, x86_AVX_STATE or x86_AVX512_STATE depending
of which of them was stored. So the thread_set_state was failing.

This change fixes both of the problems.

* Simplify the RestoreState

* Set the message buffer to the size of the mach_msg_t plus trailer size

src/coreclr/pal/src/exception/machexception.cpp

@@ -1325,7 +1325,7 @@ void MachExceptionInfo::RestoreState(mach_port_t thread)
     kern_return_t machret = thread_set_state(thread, x86_THREAD_STATE, (thread_state_t)&ThreadState, x86_THREAD_STATE_COUNT);
     CHECK_MACH("thread_set_state(thread)", machret);
 
-    machret = thread_set_state(thread, x86_FLOAT_STATE, (thread_state_t)&FloatState, x86_FLOAT_STATE_COUNT);
+    machret = thread_set_state(thread, FloatState.ash.flavor, (thread_state_t)&FloatState, FloatState.ash.count);
     CHECK_MACH("thread_set_state(float)", machret);
 
     machret = thread_set_state(thread, x86_DEBUG_STATE, (thread_state_t)&DebugState, x86_DEBUG_STATE_COUNT);

src/coreclr/pal/src/exception/machmessage.h

@@ -186,11 +186,6 @@ class MachMessage
     void ReplyToNotification(MachMessage& message, kern_return_t eResult);
 
 private:
-    // The maximum size in bytes of any Mach message we can send or receive. Calculating an exact size for
-    // this is non trivial (basically because of the security trailers that Mach appends) but the current
-    // value has proven to be more than enough so far.
-    static const size_t kcbMaxMessageSize = 1500;
-
     // The following are structures describing the formats of the Mach messages we understand.
 
     // Request to set the register context on a particular thread.
@@ -298,6 +293,9 @@ class MachMessage
         } data;
     } __attribute__((packed));;
 
+    // The maximum size in bytes of any Mach message we can send or receive including possible trailers
+    static const size_t kcbMaxMessageSize = sizeof(mach_message_t) + MAX_TRAILER_SIZE;
+
     // Re-initializes this data structure (to the same state as default construction, containing no message).
     void ResetMessage();