Use appropriate platform-native toolkits for decoding video

[DCNick3]

We currently support piping media to ffmpeg and gstreamer backends.

ffmpeg will always be a pain to ship, while gstreamer is only reasonably easy to use on linux. Hence, for better user experience, we need to support more video decoding APIs

• Linux: gstreamer is fine
• Windows: DirectShow (wider support) or Windows Media Foundation (possibly better tooling/API?)
• macOS: Video Toolkit or AVFoundation (I think those are just different levels of the same API)
• Android: MediaCodec
• Web: WebCodecs

[xmoezzz]

Hi,
I made a PoC that only uses openh264 to decode a video with mp4 container and h.264 stream.
I guess which aligns with your purpose, and you can check this: https://github.com/xmoezzz/mp4_openh264

[DCNick3]

Oh, nice. I did briefly try using OpenH264 via the same openh264 crate, but ran into issues while trying to play umineko videos. As I remember, the frames seemed reordered when playing back. With hindsight, this might have been a similar issue to #36, which you fixed, but I ultimately didn't get down to it.

I tried your demo, and the results seem similar to my previous attempts of using openh264. Some examples (door.mp4, no06_gold.mp4) have incorrectly ordered frames, like I had before, while others just outright crash (laser_blue.mp4, ship1.mp4). Here are some of the videos I had issues with, if you want to try debugging this: movie.zip.

P.S. after disappointment in openh264 I also learned a bit about the patent situation around H.264. TL;DR: you technically violate some U.S. patents if you distribute an h.264 decoder with you app, which definitely happens when you statically link to openh264.

This gave me the idea of using operating systems' APIs, as described in the initial post. OS vendors are already paying licenses for those H.264 decoders (except on Linux :P), so might as well try to use them. I have a working demo of a cross-platform-ish library for doing this, supporting Windows Media Foundation and gstreamer backends. I haven't touched this demo since July, but I can send it your way if this is the direction you would want to pursue.

[xmoezzz]

Thank you for your reply!
I guess you're right. Video seems like the least important part of a game engine, but it’s actually the most annoying because of patent and license issues.
For cross-platform purposes, I think it should be fine to ask linux users to install gstreamer.

[DCNick3]

Oh, btw, if we are looking at other permissively-licensed h264 decoders and ignore the patent issue, there are 3 other related decoders one might consider:

• https://github.com/ittiam-systems/libavc, used in AOSP, includes an encoder, Apache 2.0 license
• https://github.com/ittiam-systems/lib264, related codebase by the same company, doesn't include an encoder, BSD 3-clause license
• https://github.com/oneam/h264bsd, an (old?) fork of the above, states to improve compatibility, Apache 2.0 license

[xmoezzz]

Oh, btw, if we are looking at other permissively-licensed h264 decoders and ignore the patent issue, there are 3 other related decoders one might consider:

• https://github.com/ittiam-systems/libavc, used in AOSP, includes an encoder, Apache 2.0 license
• https://github.com/ittiam-systems/lib264, related codebase by the same company, doesn't include an encoder, BSD 3-clause license
• https://github.com/oneam/h264bsd, an (old?) fork of the above, states to improve compatibility, Apache 2.0 license

Thanks! I'll check that.