Problems with assigning accelerometer values to Gameobjects in unity

[TheHarmacist97]

Required Info
Camera Model	D435I
Firmware Version	05.13.00.50
Operating System & Version	Windows 10
Platform	PC
SDK Version	2.50.0
Language	C#/Unity
Segment	VR/AR

My objective is to use the IMU in D435I to move the ingame camera. This includes translation and rotation of said camera in unity

I've tried the following:

The function OnNewSample(Frame frame) is subscribed to the m_pipeline.OnNewSample event and the RSDevice is running on the Unity Thread. This function is used to capture the IMU values sent by the Realsense device.

private void OnNewSample(Frame frame)
   {
       try
       {
           if (frame.IsComposite)
           {
               using var fs = frame.As<FrameSet>();
               using Frame fg = fs.FirstOrDefault(gyroMatcher);
               if (fg != null)
               {
                   IMUGyro = Marshal.PtrToStructure<Vector3>(fg.Data);

               }
               else
               {
                   Debug.Log("null Gyro Frame");
               }
               using var fr = fs.FirstOrDefault(acccelMatcher);
               if (fr != null)
               {
                   IMUAccel = Marshal.PtrToStructure<Vector3>(fr.Data);
                   IMUAccel += gravityMag * Mathf.Rad2Deg * IMUGyro;

               }
               else
               {
                   Debug.Log("null Accel frame");
               }
               return;
           }

           if (!acccelMatcher(frame))
           {
               Debug.Log("Accel Stream did not match");
               return;
           }

           using (frame)
           {
               IMUAccel = Marshal.PtrToStructure<Vector3>(frame.Data);
           }
       }
       catch (Exception e)
       {
           Debug.Log(e.ToString());
       }
   }

To use the captured values, I've added this snippet inside the Update() function that are assigning the data to the cameras

            q = new Vector3(-IMUGyro.x, IMUGyro.y, IMUGyro.z);
            transform.Rotate(q);
            
            transform.Translate(IMUAccel);

We're facing these problems:

1. Repeated use of the gyro causes it to have angle offsets(i.e. , after using it for a while, resetting it to the starting position does not rotate the camera in a similar way, rather it has a rotation that seems random)
2. We cannot figure out how to accurately offset the values given by the accelerometer and to apply it appropriately to the Unity Camera
3. Even after calibration, the accelerometer normal oscillates between 9.83 - 9.86, with an extra drift in the x and z axes

@MartyG-RealSense @dorodnic

[TheHarmacist97]

Also, comments regarding code smell and unsafe code are appreciated :)
Thanks

[MartyG-RealSense]

Hi @TheHarmacist97 I would recommend first trying to put your snippet inside a FixedUpdate() function instead of Update(), as FixedUpdate() should update about 4x faster than Update and is also especially suited for physics calculations and transforms that involve physics.

I do not have any advice to offer about code smell and unsafe code though. The official Unity documentation provides the following short article:

https://docs.unity3d.com/ScriptReference/PlayerSettings-allowUnsafeCode.html

[TheHarmacist97]

Hey @MartyG-RealSense, thanks for the prompt response

I still think Update() should be better for the transform control of the camera, as it is not a physics object(no rigidbody) and FixedUpdate has a default time-step of 0.02s (50 times a second) while normal Update has a smaller time-step. Since we're getting 150+ frames per second at the moment, I feel it's best to use change transforms in the Update() function.

Thanks for the advice about unsafe code, although I was asking for the capture of the pipeline and disposal of said resources

[MartyG-RealSense]

A common practice in Unity is to put camera control code in a LateUpdate() function so that it updates last after all of the Update() code has completed. This results in smoother camera movement.

There is some RealSense Unity wrapper advice about pipeline disposal and garbage collection at #1477 and #3896

[MartyG-RealSense]

Hi @TheHarmacist97 Do you require further assistance with this case, please? Thanks!

[TheHarmacist97]

We're still facing the problem with the accelerometer values as mentioned above.
We'd like some guidance as to how to interpret those values correctly in code and assign them to the virtual Camera
@MartyG-RealSense

[MartyG-RealSense]

In a past Unity project that converted RealSense camera movements into movements of a Unity object such as a camera, I applied the following procedure.

1. Create an empty object to act as the parent of the camera, and create three separate child objects to control the X, Y and Z axes. Then child the Unity camera to Z so that the camera object rotated in the appropriate directions as the X, Y and Z childs were rotated.

MASTER CAMERA JOINT
CAMERA JOINT X
CAMERA JOINT Y
CAMERA JOINT Z
MAIN CAMERA

In each of the X, Y and Z joints I created a individual animation clip that would rotate the object backwards and forwards along the full 360 degree axis. Unity animation clips have a playback timeline value between 0 (start) and 1 (end). So by setting a playback value between 0 and 1, you can cause the animation clip to rotate the object between 0 and 360 degrees on that particular axis that the clip is rotating on (X, Y or Z).

The tricky part was converting the values provided by the camera into a numeric scale between 0 and 1 to control the forward and backward playback of the animation timeline. In this case, the best way to do that may be to divide the camera gyro angle by 360, so 360 / 360 = 1, 180 / 360 = 0.5, etc.

---

If your scripting can a gyro angle value in degrees then a simpler method may be to simply apply that value directly to the object's X, Y or Z rotation transform. I personally prefer using animation clips though because they are much more stable, as an object may suddenly wildly change orientation if using raw values if there is a glitch in those values.

[Vizonary]

In a past Unity project that converted RealSense camera movements into movements of a Unity object such as a camera, I applied the following procedure.

1. Create an empty object to act as the parent of the camera, and create three separate child objects to control the X, Y and Z axes. Then child the Unity camera to Z so that the camera object rotated in the appropriate directions as the X, Y and Z childs were rotated.

MASTER CAMERA JOINT CAMERA JOINT X CAMERA JOINT Y CAMERA JOINT Z MAIN CAMERA

In each of the X, Y and Z joints I created a individual animation clip that would rotate the object backwards and forwards along the full 360 degree axis. Unity animation clips have a playback timeline value between 0 (start) and 1 (end). So by setting a playback value between 0 and 1, you can cause the animation clip to rotate the object between 0 and 360 degrees on that particular axis that the clip is rotating on (X, Y or Z).

The tricky part was converting the values provided by the camera into a numeric scale between 0 and 1 to control the forward and backward playback of the animation timeline. In this case, the best way to do that may be to divide the camera gyro angle by 360, so 360 / 360 = 1, 180 / 360 = 0.5, etc.

If your scripting can a gyro angle value in degrees then a simpler method may be to simply apply that value directly to the object's X, Y or Z rotation transform. I personally prefer using animation clips though because they are much more stable, as an object may suddenly wildly change orientation if using raw values if there is a glitch in those values.

Thanks for the advice @MartyG-RealSense , can we use the accelerometer values to translate the camera's position instead of just rotating it. Also, should we lerp the remapped values of the gyro to reduce the unwanted jitter?

[MartyG-RealSense]

Perhaps the accel value could be used as a multiplier to make the camera travel from position A to position B faster. For example:

X, Y or Z position = X, Y or Z position x 9.80.

So as the accel value increases above zero, the object starts moving along the axis as the accel value is added to the position, with greater accel values resulting in faster movement.

Using lerp to change smoothly between the previous and current states would be a good idea, yes.

[TheHarmacist97]

Hey, I really did not understand your previous comment. We wanted to use the D435i's IMU to emulate the values inside unity as if we were using the D435i as a head-mounted unit.

So if the person moves 3 meters toward the left, the camera should also move 3 meters toward the left.

[MartyG-RealSense]

The 400 Series models such as the D435i do not have the inbuilt abilty to track the pose (position + rotation) of the camera device using its IMU. The RealSense T265 Tracking Camera, which has the same IMU component as the D435i, has that feature and has an official T265-only RealSense Unity wrapper example program for SLAM navigation. There is a YouTube video of it at https://www.youtube.com/watch?v=Hhfw3Z4DQ1A

[MartyG-RealSense]

Hi @TheHarmacist97 Do you require further assistance with this case, please? Thanks!

[TheHarmacist97]

Without buying the T265 camera separately, is there any possible solution to achieve the abovementioned objective?
Can't we directly utilize the accelerometer values we're getting in Unity

[MartyG-RealSense]

My understanding is that it is possible to interface with the functions of the RealSense C# wrapper from within Unity, as described by an expert on the subject of C# and Unity at #1570 (comment) and #8347 (comment)

There is ony a limited amount of references though about accessing IMU values via the C# wrapper, like those at #8746 (comment) and #2996 (comment)

[MartyG-RealSense]

Hi @TheHarmacist97 Do you have an update about this case that you can provide, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.