Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix loading 200 Hz gaze data in Pupil Invisible recordings #2204

Merged
merged 3 commits into from
Nov 11, 2021

Conversation

papr
Copy link
Contributor

@papr papr commented Nov 5, 2021

When Pupil Player opens a Pupil Invisible recording for the first time, it performs an in-place upgrade procedure due to compatibility reasons. In this process, it loads PI gaze data and generates Pupil Player compatible gaze data based on it.

Due to a flaw in the previous gaze conversion procedure, Pupil Cloud post-processed 200 Hz gaze data could be loaded multiple times or not at all.

This PR moves the prior implementation from video_capture.utils to pupil_recording.update.invisible and restructures the code flow such that 200 Hz gaze data is processed separately from the case where gaze data is loaded from the real-time recorded files.

Below a detailed description of what should be happening:

Pupil Invisible Companion records gaze data into three different sets of files. Their
names can be matched by the following regex patterns:

  • ^gaze ps[0-9]+.raw$
  • ^gaze ps[0-9]+.time$
  • ^worn ps[0-9]+.raw$

The worn data is a stream of values of either 0 or 255, indicating that the glasses
were (not) worn. Pupil Player maps these to gaze confidence values of 0.0 and 1.0
respectively.

Since all *.time files are converted to Pupil Player before this function is being
called, we match the ^gaze ps[0-9]+_timestamps.npy$ pattern on the recording files
instead. When looking for the location and worn data, the function just replaces the
necessary parts of the timestamp file names instead of performing separate regex
matches.

If the recording was successfully post-processed and downloaded from Pupil Cloud, it
will contain 200Hz-densified gaze data. This data replaces the real-time recorded
data by Pupil Invisible Companion and is stored in three files:

  • gaze_200hz.raw
  • gaze_200hz.time (or gaze_200hz_timestamps.npy if upgraded)
  • worn_200hz.raw

The worn data is a special case as it was introduced at different points in time to
Pupil Invisible Companion and Pupil Cloud. In other words, it is possible that there
is no worn data, only real-time recorded worn data, or 200 Hz worn data. The latter
is preferred. If 200 Hz gaze data is only available with real-time recorded worn
data, the latter is interpolated to 200 Hz using a k-nearest-neighbour (k=1)
approach. If no worn data is available, or the numbers of worn samples and gaze
timestamps are not consistent, Pupil Player assumes a confidence value of 1.0 for
every gaze point.

When Pupil Player opens a Pupil Invisible recording for the first time, it performs an in-place upgrade procedure due to compatibility reasons. In this process, it loads PI gaze data and generates Pupil Player compatible gaze data based on it. 

Due to a flaw in the previous gaze conversion procedure, Pupil Cloud post-processed 200 Hz gaze data could be loaded multiple times or not at all.

This PR moves the prior implementation from `video_capture.utils` to `pupil_recording.update.invisible` and restructures the code flow such that 200 Hz gaze data is processed separately from the case where gaze data is loaded from the real-time recorded files.

Below a detailed description of what should be happening:

Pupil Invisible Companion records gaze data into three different sets of files. Their 
names can be matched by the following regex patterns:
    - `^gaze ps[0-9]+.raw$`
    - `^gaze ps[0-9]+.time$`
    - `^worn ps[0-9]+.raw$`

The worn data is a stream of values of either 0 or 255, indicating that the glasses
were (not) worn. Pupil Player maps these to gaze confidence values of 0.0 and 1.0
respectively.

Since all `*.time` files are converted to Pupil Player before this function is being
called, we match the `^gaze ps[0-9]+_timestamps.npy$` pattern on the recording files
instead. When looking for the location and worn data, the function just replaces the
necessary parts of the timestamp file names instead of performing separate regex
matches.

If the recording was successfully post-processed and downloaded from Pupil Cloud, it
will contain 200Hz-densified gaze data. This data replaces the real-time recorded
data by Pupil Invisible Companion and is stored in three files:
    - `gaze_200hz.raw`
    - `gaze_200hz.time` (or `gaze_200hz_timestamps.npy` if upgraded)
    - `worn_200hz.raw`

The worn data is a special case as it was introduced at different points in time to
Pupil Invisible Companion and Pupil Cloud. In other words, it is possible that there
is no worn data, only real-time recorded worn data, or 200 Hz worn data. The latter
is preferred. If 200 Hz gaze data is only available with real-time recorded worn
data, the latter is interpolated to 200 Hz using a k-nearest-neighbour (k=1)
approach. If no worn data is available, or the numbers of worn samples and gaze
timestamps are not consistent, Pupil Player assumes a confidence value of 1.0 for
every gaze point.
@papr papr added type: bug product: Pupil Invisible Issues related to Pupil Invisible labels Nov 5, 2021
@papr papr requested review from romanroibu and N-M-T November 5, 2021 16:09
@papr papr requested a review from N-M-T November 8, 2021 14:32
@papr papr marked this pull request as ready for review November 8, 2021 14:51
@papr papr merged commit 4db40b9 into develop Nov 11, 2021
@papr papr deleted the avoid_loading_cloud_200hz_gaze_multiple_times branch November 11, 2021 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
product: Pupil Invisible Issues related to Pupil Invisible type: bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants