diff --git a/src/alpha-lab/README.md b/src/alpha-lab/README.md index da4b9ca38..939c20ed2 100644 --- a/src/alpha-lab/README.md +++ b/src/alpha-lab/README.md @@ -98,7 +98,7 @@ export default { title: "RIM Room", text: "We pushed the limits of markerless mapping with Pupil Cloud’s Reference Image Mapper - scanning an entire apartment.", to: "/alpha-lab/multiple-rim/", - img: "desk-overlay.png", + img: "desk-heatmap.jpeg", }, { title: "Look at my hand!", diff --git a/src/alpha-lab/multiple-rim.md b/src/alpha-lab/multiple-rim.md index dd7daba45..928df0264 100644 --- a/src/alpha-lab/multiple-rim.md +++ b/src/alpha-lab/multiple-rim.md @@ -4,93 +4,157 @@ permalink: /alpha-lab/multiple-rim/ tags: [Pupil Invisible, Neon, Cloud] --- -# Map and visualize gaze onto multiple reference images taken from the same environment +# Map and visualize gaze on multiple reference images taken from the same environment
- +

-In the [Reference Image Mapper](/enrichments/reference-image-mapper/) guide, we learnt how to properly set up a Reference Image Mapper enrichment, with a single reference image. However, there are some cases in which it would be useful to map gaze onto multiple reference images taken from the same environment - for example, moving in a room while interacting with certain parts of it. - ::: tip -Before continuing, ensure you are familiar with the [Reference Image Mapper](/enrichments/reference-image-mapper) enrichment. Check out [this explainer video](https://www.youtube.com/watch?v=ygqzQEzUIS4&t=56s) for reference. +Level-up your Reference Image Mapper workflow to extract insights from participants freely exploring their environment! ::: -## Reference Images - -First, we will take pictures of the areas and/or furniture of the room we are interested in. - -| | | -| ------------------------------------------------------ | -------------------------------------------- | -| | | -| | | - -## Scanning recordings +## Exploring gaze patterns in multiple regions of an environment -In this guide, we want to map gaze onto different parts of a living room, for this reason, we recorded **two** scanning videos. We chose to use more than one scanning recording because the environment is a bit too big to be effectively scanned just by a single one. +Understanding where people focus their gaze while exploring their environment is a topic of interest for researchers in +diverse fields, ranging from Art and Architecture to Zoology. The [Reference Image Mapper](/enrichments/reference-image-mapper/) +enrichment in Pupil Cloud makes it possible to map gaze onto 3D real-world environments and generate heatmaps. These provide +an informative overview of visual exploration patterns and also pave the way for further analysis, such as region of interest analysis. -Based on the environment dimension/complexity, you might need to do the same and record separate scanning videos. - -Please follow our [best practices](/enrichments/reference-image-mapper/#scanning-best-practices) for optimal scanning. +In this guide, we will demonstrate how to use the [Reference Image Mapper](/enrichments/reference-image-mapper/) to map a +participant's gaze onto various regions of a living environment as they freely navigate through it. ::: tip -To ensure good scanning of big plain surfaces - like tables and kitchen countertops - enrich them with features. Use a printed tablecloth and/or place items to produce a successful mapping! +Before continuing, ensure you are familiar with the [Reference Image Mapper](/enrichments/reference-image-mapper) enrichment. +Check out [this explainer video](https://www.youtube.com/watch?v=ygqzQEzUIS4&t=56s) for reference. ::: -
- +## The tools at hand +The [Reference Image Mapper](/enrichments/reference-image-mapper/) enables mapping of gaze onto a +_single_ reference image of an environment. However, there is often a need to analyze _multiple_ regions for a more in-depth +understanding of visual exploration. This guide demonstrates how to accomplish this by applying the enrichment multiple +times during the same recording to generate mappings and heatmaps for different regions. + +## Steps +For the analysis, we will need the following: +- Multiple reference images of the environment +- Single or multiple scanning recordings. The choice of whether to use single or multiple scanning recordings depends on +the dimensions of the space to be explored (see below for examples) +- An eye tracking recording taken as the participant(s) move freely within the environment +- User-inputted [events](/neon/basic-concepts/events) to segment the recording(s) into [sections](/enrichments/#enrichment-sections) based on +the areas the person was looking at + +1. **Capture Reference Images:** Take pictures of the areas or objects within the environment you wish to investigate. Here are some example pictures of different areas and pieces of furniture in our environment (a living room, dining area, and kitchen): + +
+
+ Desk +
+
+ TV1 +
+
+ TV2 +
-
- +
+
+ Table +
+
+ Kitchen +
+
+ Cupboard +
-::: danger -**Scanning Recording Duration** -
-
-Please record a scanning video that is less than 3 minutes long! -
-
-The Reference Image Mapper enrichment does **not** accept longer recordings. -::: +
-## Run the enrichments +2. **Record Scanning Videos:** For this guide, we used *five* separate scanning recordings to cover the environment. If you have a bigger or more complex environment, it might be necessary to use more. On the other hand, it might be possible to use fewer if you have a smaller environment or if you can capture sufficient data. Remember, each scanning recording must be **under 3 minutes in duration**. + +Check out these videos which show how we made the scans (also be sure to follow our [best practices](/enrichments/reference-image-mapper/#scanning-best-practices) for optimal scanning): -Here we recorded just one video where the wearer was asked to walk and freely explore the living room. Now it is time to map the subject's gaze from this video into the five pictures above. +::: tip +The Reference Image Mapper prefers feature-rich environments. If you have large plain surfaces, like empty tables or countertops, +consider placing some strategic items within the environment to increase the chances of successful mapping. +::: + +
+
+
+ +
+
+
+
+ +
+
+
-
- +
+
+
+ +
+
+
+
+ +
+
-During the recording, the user looked at the same furniture and parts of the room multiple times. We suggest you focus on -specific [sections](/enrichments/#enrichment-sections) of the recording based on which part of the -room the user is exploring. +
+ +
-For this recording, we used the following [event annotations](/invisible/basic-concepts/events) to run five Reference Image Mapper enrichments: +3. **Eye Tracking Recordings:** Make an eye tracking recording while the participant(s) freely explore and visually interact with various elements within the environment. (You can of course make these prior to the reference images and scanning recordings). -- Cupboard: `cupboard.begin` and `cupboard.end` -- Desk: `desk.begin` and `desk.end` -- Kitchen: `kitchen.begin` and `kitchen.end` -- TV: `tv.begin` and `tv.end` -- Table: `table.begin` and `table.end` +
-## Final results +4. **Add Custom Events:** During the eye tracking recording, users may focus on a specific region once or multiple times. I.e. they may revisit that region. By adding custom [event](/neon/basic-concepts/events) annotations corresponding to these periods, you can create [sections](/enrichments/#enrichment-sections) for the enrichments to be computed. This enables you to run each enrichment only on the section(s) of recording where a certain region is being gazed at. For this guide, we used the following event annotations to run five Reference Image Mapper enrichments: + - Desk: `desk.begin` and `desk.end` + - TV area 1: `tv1.begin` and `tv1.end` + - TV area 2: `tv2.begin` and `tv2.end` + - Table: `table.begin` and `table.end` + - Kitchen: `kitchen.begin` and `kitchen.end` + - Cupboard: `cupboard.begin` and `cupboard.end` -It may take several minutes to run these enrichments depending on how long your recordings are. Once everything is finished, you can visualize how gaze is simultaneously mapped both on the recording and the reference images from the Project Editor view (as shown in the video at the very beginning of this guide). +5. **Create and run the enrichments:** You will need to create a separate enrichment for each reference image. A reasonable naming scheme *could* correspond to each area of the environment, like ‘cupboard’, ‘desk’ etc. In the temporal selection of each enrichment, be sure to use the appropriate events labels. E.g. for ‘cupboard’, you would use `cupboard.begin` and `cupboard.end`. Now, run the enrichments to map the subject's gaze from the recording onto the multiple reference images you captured. -From the Enrichment view, you can visualize heatmaps of each reference image: +## Final results -| | | -| ------------------------------------------------------ | --------------------------------------------------------- | -| | | -| | | -| | | -| | | +Once the enrichments are completed, you can view the heatmaps which illustrate areas which attracted more gaze. Additionally, you'll have the option to download gaze and fixation data mapped within the bounds of the pictures, enabling you to conduct further in-depth analyses. + +
+
+ Desk +
+
+ TV1 +
+
+ TV2 +
+
+
+
+ Table +
+
+ Kitchen +
+
+ Cupboard +
+
-That's it. We look forward to seeing your own mapped environments! +
diff --git a/src/media/alpha-lab/cupboard-heatmap.jpeg b/src/media/alpha-lab/cupboard-heatmap.jpeg new file mode 100644 index 000000000..1c396296a Binary files /dev/null and b/src/media/alpha-lab/cupboard-heatmap.jpeg differ diff --git a/src/media/alpha-lab/cupboard-img.png b/src/media/alpha-lab/cupboard-img.png deleted file mode 100644 index 6b14afcbe..000000000 Binary files a/src/media/alpha-lab/cupboard-img.png and /dev/null differ diff --git a/src/media/alpha-lab/cupboard-overlay.png b/src/media/alpha-lab/cupboard-overlay.png deleted file mode 100644 index 620518f5f..000000000 Binary files a/src/media/alpha-lab/cupboard-overlay.png and /dev/null differ diff --git a/src/media/alpha-lab/cupboard.jpeg b/src/media/alpha-lab/cupboard.jpeg new file mode 100644 index 000000000..bbfc9a84d Binary files /dev/null and b/src/media/alpha-lab/cupboard.jpeg differ diff --git a/src/media/alpha-lab/desk-heatmap.jpeg b/src/media/alpha-lab/desk-heatmap.jpeg new file mode 100644 index 000000000..b8491a9f9 Binary files /dev/null and b/src/media/alpha-lab/desk-heatmap.jpeg differ diff --git a/src/media/alpha-lab/desk-img.png b/src/media/alpha-lab/desk-img.png deleted file mode 100644 index 67a8c3183..000000000 Binary files a/src/media/alpha-lab/desk-img.png and /dev/null differ diff --git a/src/media/alpha-lab/desk-overlay.png b/src/media/alpha-lab/desk-overlay.png deleted file mode 100644 index cccfd0a64..000000000 Binary files a/src/media/alpha-lab/desk-overlay.png and /dev/null differ diff --git a/src/media/alpha-lab/desk.jpeg b/src/media/alpha-lab/desk.jpeg new file mode 100644 index 000000000..264c568e9 Binary files /dev/null and b/src/media/alpha-lab/desk.jpeg differ diff --git a/src/media/alpha-lab/kitchen+table-img.jpeg b/src/media/alpha-lab/kitchen+table-img.jpeg deleted file mode 100644 index 000eb6c34..000000000 Binary files a/src/media/alpha-lab/kitchen+table-img.jpeg and /dev/null differ diff --git a/src/media/alpha-lab/kitchen+table-overlay.png b/src/media/alpha-lab/kitchen+table-overlay.png deleted file mode 100644 index 5527740b9..000000000 Binary files a/src/media/alpha-lab/kitchen+table-overlay.png and /dev/null differ diff --git a/src/media/alpha-lab/kitchen-heatmap.jpeg b/src/media/alpha-lab/kitchen-heatmap.jpeg new file mode 100644 index 000000000..011dde700 Binary files /dev/null and b/src/media/alpha-lab/kitchen-heatmap.jpeg differ diff --git a/src/media/alpha-lab/kitchen-imgs.png b/src/media/alpha-lab/kitchen-imgs.png deleted file mode 100644 index e6d01ec22..000000000 Binary files a/src/media/alpha-lab/kitchen-imgs.png and /dev/null differ diff --git a/src/media/alpha-lab/kitchen-overlay.png b/src/media/alpha-lab/kitchen-overlay.png deleted file mode 100644 index 38935e109..000000000 Binary files a/src/media/alpha-lab/kitchen-overlay.png and /dev/null differ diff --git a/src/media/alpha-lab/kitchen.jpeg b/src/media/alpha-lab/kitchen.jpeg new file mode 100644 index 000000000..4441c33a8 Binary files /dev/null and b/src/media/alpha-lab/kitchen.jpeg differ diff --git a/src/media/alpha-lab/table-heatmap.jpeg b/src/media/alpha-lab/table-heatmap.jpeg new file mode 100644 index 000000000..916d1043a Binary files /dev/null and b/src/media/alpha-lab/table-heatmap.jpeg differ diff --git a/src/media/alpha-lab/table.jpeg b/src/media/alpha-lab/table.jpeg new file mode 100644 index 000000000..60d40e54e Binary files /dev/null and b/src/media/alpha-lab/table.jpeg differ diff --git a/src/media/alpha-lab/tv-img.png b/src/media/alpha-lab/tv-img.png deleted file mode 100644 index c04f1a02b..000000000 Binary files a/src/media/alpha-lab/tv-img.png and /dev/null differ diff --git a/src/media/alpha-lab/tv-overlay.png b/src/media/alpha-lab/tv-overlay.png deleted file mode 100644 index a39b30583..000000000 Binary files a/src/media/alpha-lab/tv-overlay.png and /dev/null differ diff --git a/src/media/alpha-lab/tv1-heatmap.jpeg b/src/media/alpha-lab/tv1-heatmap.jpeg new file mode 100644 index 000000000..0607cc30d Binary files /dev/null and b/src/media/alpha-lab/tv1-heatmap.jpeg differ diff --git a/src/media/alpha-lab/tv1.jpeg b/src/media/alpha-lab/tv1.jpeg new file mode 100644 index 000000000..7a84a8a2a Binary files /dev/null and b/src/media/alpha-lab/tv1.jpeg differ diff --git a/src/media/alpha-lab/tv2-heatmap.jpeg b/src/media/alpha-lab/tv2-heatmap.jpeg new file mode 100644 index 000000000..cd5c0a48c Binary files /dev/null and b/src/media/alpha-lab/tv2-heatmap.jpeg differ diff --git a/src/media/alpha-lab/tv2.jpeg b/src/media/alpha-lab/tv2.jpeg new file mode 100644 index 000000000..75d023772 Binary files /dev/null and b/src/media/alpha-lab/tv2.jpeg differ