Tracking: Client-side media processing #61447
Comments
swissspidy
added
[Feature] Media
|
Firstly, thanks so much for the detail and all the background work!
This came up in a recent discussion I was having. What's your take on the level of effort to integrate this feature? I was browsing the media experiments repo code and was wondering if there were any further considerations, beyond what you've explored already. It would be a fantastic feature to have - happy to have a shot at it, or try to organize someone to have a shot at it 😄 if you think it's worth it at this stage. I'll take your guidance. Thank you! |
|
I heard of similar discussions recently :) Always great to hear interest in this kind of topic. And I am glad you already browsed the media experiments code a little bit. If you are interested solely in HEIC conversion, it by itself could be implemented relatively easily in theory. See #61861 as a demonstration. However, as with everything else mentioned in this ticket, there are some important aspects to consider. For instance:
And perhaps most importantly: (Edit: dynamically linking via loading libheif-js from an externally hosted script might be OK, but again, not a lawyer) ➡️ Aside: Thanks to this POC PR I discovered that the current license check in this repo is broken 😬 I am fixing it in #61868 Bottom line: I've deliberately split this project into multiple phases as it's more complex than what it might seem on the surface. I strongly suggest tackling this step by step. Happy to discuss HEIC itself separately elsewhere. |
|
Thanks for going into detail Pascal! I like your idea of a "Would you like to convert this to a web safe format?" modal that teaches the user what's going on. I've asked Automattic's legal team for some non-IANAL guidance about I think supporting HEIC is the most important outcome here. You encounter these files more and more nowadays and it's very frustrating to users when they don't "just work" in WordPress. I'd push for starting with HEIC support and then expanding on that into other areas (compression, resizing, converting GIFs, etc.) |
That's what I'm reading from https://www.gnu.org/licenses/gpl-faq.html#gpl-compat-matrix — if WordPress uses code under LGPLv3, then the "combined" work (WordPress + lib) would then fall under the terms of GPLv3 Bit of a wild suggestion, but is there a chance we could fallback to backend processing via Imagick: https://imagemagick.org/script/formats.php#supported I say "wild" as I assume HEIC files can be pretty massive, and batching would be pricey in terms of performance.
https://www.npmjs.com/package/heic2any or https://github.com/MaestroError/php-heic-to-jpg (both MIT) might also be worth a looksy 🤷🏻
Best answer. 😄 |
Cool. I'm going to implement it in my plugin soon so we can see how well it works UX-wise. For WCEU I plan on working on a POC for bringing parts of that plugin to Gutenberg.
Not sure I agree with this one. Performance issues due to too large & heavy images are quite widespread. HEIC is mostly an iPhone thing and not everyone uses those. As long as there are legal question marks anyway, I'd recommend focusing on server-side HEIC conversion in CORE-53645.
heic2any uses the same libheif library under the hood and the usage would be exactly the same, so the MIT license there seems incorrect to me. php-heic-to-jpg only works server-side and requires an executable file on the server, which is not suitable for WP. For server-side support I recommend focusing on ImageMagick support in CORE-53645. |
I agree. Thanks @swissspidy, sorry for derailing your issue here with talk of HEIC 😀 |
|
At WordCamp Europe I had a great conversation with @youknowriad and @gziolo where I was able to demo Media Experiments and elaborate on its inner workings and some design decisions. Now, I made a ton of (documentation) improvements so people can better understand how it works. The technical overview doc is a great entry point. I would appreciate if y'all could check it out. All code is 100% TypeScript and well-documented (check out the readmes!), which should help with reviews as well. cc @ntsekouras who previously showed interest in reviewing the codebase As the plugin name implies, there are some experimental features in there right now, but the core part, the client-side media processing & compression is what matters most and is what I am proposing to gradually merge into Gutenberg. That said, if we think this functionality is better tested at scale in its own plugin for now, we could also consider further testing Media Experiments separately as part of the Performance Lab family of plugins before bringing it into Gutenberg. |
|
@swissspidy So I've been thinking about this more (specially the store related stuff). So I've come up to the realization that we might be able to actually bundle most of the package/behavior into the generic In other words, all block-editors using the block-editor package can gain support for:
But, to be able to do so, we need to split the "WP specific logic" from the "Media handling logic". For the "upload API" we already have that it's the For the rest, I'm not sure I understand if there's any WP specific logic in the rest of the logic (the client side optimizations, the compression...). At first glance, there's none right? I guess what I'm proposing is the following: Splitting the MediaExperiment into two parts:
How feasible would that be? |
|
I'd love to start testing some of these enhancements on the plugin, we can gate them with a feature flag if anything feels too experimental yet. |
|
Riad's suggestion for inclusion in block editor seems like a good plan. I can definitely help reviewing and with whatever needed. |
Interesting 👀 Let me analyze the code base a bit more to provide a good answer. Just so I understand correctly: Right now, everything is built as sort of a drop-in replacement for the
What you are suggesting:
Is that correct?
As mentioned, everything was built as a drop-in replacement for
|
Yes, that's correct.
Yes, the problem with that is that media-utils has always been a stateless package and making it stateful means access to its store from other packages which would break existing assumptions: things like multiple editors for instance is one example of that.
Yeah, I think my answer here would be to try to make these use mediaUpload setting (like adding things to the arguments to handle these cases) or provide alternative settings to the block editor (which also means potentially disable these behaviors if these settings are not present)
💯 Yes, I discovered that too late, there's an ESlint rule that is ignored there because of this (see top of the file) |
|
OK, understood! In that case I will go through my plugin once more and make the necessary changes so it's easier to reason about & eventually port over. |
@noisysocks Curious, have you ever heard back on this? |
|
I did. Your understanding was correct, we'd need to relicense WordPress or use a different library 😞 |
|
Good news, I found some possible libheif-js alternative in the form of ffmpeg. Will share details at swissspidy/media-experiments#483 |
|
I have been thinking about poster generation for a while. Poster generation can be done without the need to FFMPeg. This can be done using canvas tag. @swissspidy and myself worked on something similar in web stories. See this example. We could add some middleware to the api-fetch. This can defect if a video file is being uploaded and generate the post image, uploaded and set it as the feature image. There is already logic to detect if it is a media upload request, isMediaUploadRequest. Poster image generate would be a good POC about handling media in the browser. |
|
Poster generation like this is already implemented in my media-experiments plugin and can be ported over in a later stage, according to the roadmap. |
Where is the roadmap? |
|
Literally in the issue description. |
Thought there would a blog post on make with more detail. The above doesn't really give a tonne of detail about poster images, other than you mentioning that may generate poster.
Can you provide more detail how this might be archived, like using a canvas tag or ffmpeg? Will this be done as part of the upload process transparently to users or requires uses to press a button? Regarding the roadmap, as I am trying to help, I am looking at other part of it, to see what others, like myself could work on. I also trying to understand why things are ordered how they are. |
You can find a working implementation using https://github.com/swissspidy/media-experiments. Right now it uses canvas but there's an ffmpeg version as well. It's all automatically behind the scenes during the upload, but there is also a button for generating a poster for existing older videos.
The first step is setting the foundation for client-side media processing, with the focus on images, providing a way to compress images and generate all sub-sizes in the browser. After that, other media formats like GIFs and videos can follow. If you're interested in any of that, I recommend checking out https://github.com/swissspidy/media-experiments. |
|
I have checked out the repo. I have been watching the repo since you created it. Not to mentioned much of work in the repo is based on work that you and I worked on in web stories. But what is in that repo are called experiments. It is unclear what is and isn't being ported over to Gutenberg. My point is this, even through I have followed every step of this, I still think the above "roadmap" is unclear and needs more detail. This detail is required for a number of reasons
Again, just want to understand your reasoning and support here. This work is great and will be great benefit to everyone. Just want to make that benefit clear to everyone who read this ticket. 👍 |
This is something that was previously mentioned in #55106 and has been heavily explored in Media Experiments already (see for example this blog post for an overview of the latter). I believe the core logic at the heart of Media Experiments is a great foundation to implement such a feature, and Gutenberg is the right place to further develop it and bring it to more people.
Overview
Current image processing in WordPress relies on server-side resources and older image libraries, leading to potential performance issues and limited support for modern image formats such as AVIF. This results in a subpar user experience, particularly with resource-intensive tasks like resizing and compressing images. Additionally, the lack of modern compression tools like MozJPEG further hinders optimization efforts.
Client-side media processing offers a solution by leveraging the browser's capabilities to handle tasks like image resizing and compression. This approach not only alleviates the strain on server resources but also enables the use of more advanced image formats and compression techniques, ultimately improving website performance and user experience. By tapping into technologies like WebAssembly, WordPress can provide a more efficient and seamless media handling process for both new and existing content.
Resources
Roadmap
Given the many possibilities client-side media processing unlocks, this project can be split into multiple phases, each with their dedicated goals and success criteria. For example:
Phase 1
Goals
Nice to have
Non-goals
Phase 2
Goals
Nice to have
Non-goals
Phase 3
Goals
<picture>, e.g. for using AVIF with a JPEG fallbackNice to have
Out of scope
Non-goals at the moment, or something for far out in the future:
Caveats and risks
Technical complexity
WASM-based image optimization requires
SharedArrayBuffersupport, which in turn requires cross-origin isolation. Implementing that in a robust way without breaking other parts of the editor is challenging. There are currently some known issues in Firefox and Safari due to these browsers not supportingcredentiallessiframe embeds. Embed previews in the editor currently do not work because of this, until those browsers add support for<iframe credentialless>.Image compression by itself is rather complex due to the vast amount of different combinations of codecs and encoding options available. While the existing server-side image handling in WordPress has been proven over many years, doing everything client-side is new territory. There will be many edge cases that need to be handled.
As for overall architecture design, the existing Media Experiments project contains a very solid foundation that already solves many challenging problems and could be built upon.
Confidence level
Reception in the community so far has been extremely positive, including from WordPress leadership. There is clearly a desire for this kind of solution.
The biggest risk is about compatibility with other parts of the editor (plugins, embed previews) and certain environments because of the cross-origin isolation required for
SharedArrayBuffer. Not breaking existing sites is very important. The risk can be mitigated through extensive testing, additional safeguards, and documentation. Lack of<iframe credentialless>support is not ideal as it degrades editor UX in some cases, but it does not break the authoring experience as a fallback exists.While image optimization could theoretically be done through other means such as the canvas API, said API is extremely limited across browsers. For example, there is a lack of support for compression options (e.g. lossless or lossy) and mime types (no AVIF everywhere, no WebP in Safari).
Implementation status
mediaUploadMiddleware#64843vipspackage #64845The text was updated successfully, but these errors were encountered: