Thumbnail images not displayed with Windows-App and Browser

[RannerDesign]

Steps to reproduce

The Client-Server-API describes in chapter 10.2.1.1 how to post a message with image
https://spec.matrix.org/unstable/client-server-api/#mimage

A single image can successfully be posted as described and is displayed correctly with Element via Browser, Windows-App and IOS-App.

However, if thumbnail images are included as described, the result depends on the platform. On IOS in the list of events in the room shows the thumbnail image and on touch the original fullsize image.

Accessing the same room with same user via Browser or Windows-App, in the list of events the original image is shown rendered to thumbnail dimensions. On click the original full-size image is displayed.

So via Browser and Windows-App a thumbnail image seams to be not supported.

Test Environment

The tests were performed in the following environments:

Element Desktop on Windows 10 pro 22H2 (Build 19045.4894)
Version von Element: 1.11.78
Krypto-Version: Rust SDK 0.7.2 (7a21514), Vodozemac 0.7.0

Element Web
Version von Element: 1.11.78
Krypto-Version: Rust SDK 0.7.2 (7a21514), Vodozemac 0.7.0
accessed with Chrome Version 129.0.6668.70 (Offizieller Build) (64-Bit)

Element IOS client
Element 1.11.18 (20240917151940)
Matrix SDK 0.27.13
Rust Crypto SDK 0.7.2 (Vodozemac 0.7.0)

The following server and user environment was used:

user_name:	Testuser1040
user_id:	@m1040a:matrix.org
room_name:	TH3
room_id:	!mNtkvlOYMJmlSPpMZt:matrix.org

The generation of events was performed with Python scripts using matrix-nio

Test to reproduce this issue

There is the original full-size image with the following data:

'name': 	'TestImg5',
'filename': 	'WP2024-12_Was denn nun_[Ranner]_IMG_4159.jpg',
'filesize': 	565820,
'mime': 	'image/jpeg',
'width': 	2000,
'height': 	1333,
'author': 	'Ranner',
'title': 	'Text under the image',
'key': 		'nckoKcJcbkjnRAgeYGJxrMwg',
'url': 		'mxc://matrix.org/nckoKcJcbkjnRAgeYGJxrMwg',
'urlweb': 	'https://matrix-client.matrix.org/_matrix/media/v3/download/matrix.org/nckoKcJcbkjnRAgeYGJxrMwg'

Based on this image, a thumbnail image has been generated with the following data:

'thumbinfo': {
	'w': 		324, 
	'h': 		296, 
	'size': 	14834, 
	'mimetype': 'image/jpeg'
},
'thumburl': 	'mxc://matrix.org/XPmtzngECSbRojBtXKcnSyHG',
'thumburlweb': 	'https://matrix-client.matrix.org/_matrix/media/v3/download/matrix.org/XPmtzngECSbRojBtXKcnSyHG'

Both uploads have been successful and both images are accessible via urlweb or thumburlweb links.

With these two images an event has been successfully posted in a room:

room:	!mNtkvlOYMJmlSPpMZt:matrix.org
event:	$hQ1RE9_Q26aldmHOYVd5U9137OKwcwVUyfOj5lvqzyM

Screenshots

Under IOS the generated thumbnail image is shown, and touching on this the fullsize original image is displayed.
Under Browser and Windows-App the original image (2000x1333px) is shown rendered to 323x215px. Clicking on this, the original image is displayed. The thumbnail image is not shown at all.

Browser (result not expected)

Windows App (result not expected)

IOS App (result as expected)

Event Raw Data

Looking on this event on the 3 platforms shows nearly identical raw data.

• Browser, Windows-App and IOS-App differ in unsigned.age

• Browser/Windows-App have unsigned.membership: "join" which is missing under IOS

• The sequence of the json entries is same in Browser and Windows-App but differs in IOS-App, but this should never be a problem

Browser

{
	"content": {
		"body": "WP2024-12_Was denn nun_[Ranner]_IMG_4159.jpg",
		"filename": "WP2024-12_Was denn nun_[Ranner]_IMG_4159.jpg",
		"info": {
			"h": 1333,
			"mimetype": "image/jpeg",
			"size": 565820,
			"thumbnail_info": {
				"h": 296,
				"mimetype": "image/jpeg",
				"size": 14834,
				"w": 324
			},
			"thumbnail_url": "mxc://matrix.org/XPmtzngECSbRojBtXKcnSyHG",
			"w": 2000
		},
		"msgtype": "m.image",
		"url": "mxc://matrix.org/nckoKcJcbkjnRAgeYGJxrMwg"
	},
	"origin_server_ts": 1727447933228,
	"sender": "@m1040a:matrix.org",
	"type": "m.room.message",
	"unsigned": {
		"membership": "join",
		"age": 637
	},
	"event_id": "$hQ1RE9_Q26aldmHOYVd5U9137OKwcwVUyfOj5lvqzyM",
	"room_id": "!mNtkvlOYMJmlSPpMZt:matrix.org"
}

Windows App

{
	"content": {
		"body": "WP2024-12_Was denn nun_[Ranner]_IMG_4159.jpg",
		"filename": "WP2024-12_Was denn nun_[Ranner]_IMG_4159.jpg",
		"info": {
			"h": 1333,
			"mimetype": "image/jpeg",
			"size": 565820,
			"thumbnail_info": {
				"h": 296,
				"mimetype": "image/jpeg",
				"size": 14834,
				"w": 324
			},
			"thumbnail_url": "mxc://matrix.org/XPmtzngECSbRojBtXKcnSyHG",
			"w": 2000
		},
		"msgtype": "m.image",
		"url": "mxc://matrix.org/nckoKcJcbkjnRAgeYGJxrMwg"
	},
	"origin_server_ts": 1727447933228,
	"sender": "@m1040a:matrix.org",
	"type": "m.room.message",
	"unsigned": {
		"membership": "join",
		"age": 1699
	},
	"event_id": "$hQ1RE9_Q26aldmHOYVd5U9137OKwcwVUyfOj5lvqzyM",
	"room_id": "!mNtkvlOYMJmlSPpMZt:matrix.org"
}

IOS App

{
	"sender": "@m1040a:matrix.org",
	"content": {
		"body": "WP2024-12_Was denn nun_[Ranner]_IMG_4159.jpg",
		"filename": "WP2024-12_Was denn nun_[Ranner]_IMG_4159.jpg",
		"info": {
			"size": 565820,
			"mimetype": "image/jpeg",
			"thumbnail_url": "mxc://matrix.org/XPmtzngECSbRojBtXKcnSyHG",
			"w": 2000,
			"h": 1333,
			"thumbnail_info": {
				"size": 14834,
				"mimetype": "image/jpeg",
				"w": 324,
				"h": 296
			}
		},
		"url": "mxc://matrix.org/nckoKcJcbkjnRAgeYGJxrMwg",
		"msgtype": "m.image"
	},
	"origin_server_ts": 1727447933228,
	"room_id": "!mNtkvlOYMJmlSPpMZt:matrix.org",
	"event_id": "$hQ1RE9_Q26aldmHOYVd5U9137OKwcwVUyfOj5lvqzyM",
	"unsigned": {
		"age": 1585138
	},
	"type": "m.room.message"
}

Outcome

What did you expect?

Thumbnail of image shown in list of messages, full-size image shown on click
same as IOS app

What happened instead?

Windows app and Browser show full-size image rendered to thumbnail dimension, thumbnail noch shown

Operating system

Windows 10 pro 22H2 (Build 19045.4894)

Application version

Element: 1.11.78

How did you install the app?

element.io

Homeserver

matrix.org

Will you send logs?

No

[RannerDesign]

An additional test on Android device showed the same as for Element Windows and Element Web: thumbnail are ignored. Only Element IOS works correctly.

[MidhunSureshR]

Hey 👋 ,
This is intentional, you can see the code in https://github.com/element-hq/matrix-react-sdk/blob/develop/src/components/views/messages/MImageBody.tsx

[RannerDesign]

Thanks @MidhunSureshR for handling this issue.
When it's intentional, that thumbnails are not displayed, then it's an unintended miracle, that the thumbnail is displayed with Element IOS.
Could it be, that with Element Windows, Elements Android and Element Web the onImageError event is triggered removing this.state.thumbUrl and IOS doesn't?
So closing the issue doesn't look convincing to me

[t3chguy]

The iOS codebase is completely different and shares only crypto code. The design, requirements and choices are very different between the codebases, as well as the technical constraints in the platforms themselves. Web will only use the thumbnail in certain cases, in many cases it chooses to avoid the thumbnail due to the restrictions they impose. In your case your thumbnail is absolutely tiny and would not conform to HiDPI requirements, in other cases the thumbnail may lack colour space and other metadata. There is zero guarantee that a thumbnail will be used over the original media, and as you exemplified it is also an abuse vector as the thumbnail may contain one image and the real media a different image.

The spec says nothing close to saying clients must use thumbnails: https://spec.matrix.org/v1.11/client-server-api/#thumbnails