Fetch API is not fetching inside-ZIM resources

[benoit74]

#1276 has revealed that on both Windows and Linux, all tests linked to inline JS are not working properly.

These are tests 01* of "Javascript" page of the ZIM available at https://dev.library.kiwix.org/#lang=eng&q=zimit+test+website

Do not mind about tests 08* and 09*, they are known to not be reliable yet due to other issues not yet solved in zimit/warc2zim.

These tests are known to work ok on all readers (kiwix-desktop is probably the last one not yet tested), at least I'm 100% sure they do work on kiwix-android, kiwix-apple, kiwix-serve, kiwix-js

These tests are very specific to zimit/warc2zim. They rely on proper operation of wombat, the JS library used by zimit/warc2zim ZIMs to replace dynamically / on-the-fly URLs loaded by JS code. It is hence "normal" that the URL inside JS code is "online", it will be rewritten by wombat to the proper in-ZIM URL.

[kelson42]

OK, if this is a blocker to get Zimit video working, I guess this is the new TOP priority issue.

[benoit74]

OK, if this is a blocker to get Zimit video working, I guess this is the new TOP priority issue.

It looks like it is a very distinct issue

[kelson42]

@benoit74 OK, so far we have done our best to remove all inline JS because this is not supported in Kiwix JS (and there is - looks like - no way to fix this). @Jaifroid could confirm. We had an effort specifically about that 2 years ago. This is actually still a constraint applying on all our custom scrapers if we want to continue to have things working on Kiwix JS. This is very important.

This is the reason why this limitation of Kiwix Desktop has not been considered properly. But if this impacts ZIMit, where we have less control about the page content, then we will have to find a fix here with high priority.

[benoit74]

The specific inline JS of the test website is working fine inside Kiwix PWA ... no idea why.

[Jaifroid]

Morning all. The restriction on inline JS (and eval, and a few other things) only affects the Chrome and Firefox Browser Extensions when they are running in Restricted mode (or in ServiceWorkerLocal mode in the case of Chrome). There is no workaround to run JS in the ZIM in those contexts. Chrome (and Chromium derivatives) happen to be the most popular platform on which Kiwix JS runs.

There is a workaround for browsers which fully support ServiceWorker mode from a secure origin. This is effectively to "abandon" the extension, and redirect users to a secure offline-first PWA (https://browser-extension.kiwix.org). I introduced this workaround a couple of years ago so that we could support dynamic ZIMs such as Zimit-based ZIMs. PWAs are not restricted (except by a custom CSP controlled by the developer), and can run inline JS and even eval, because the origin is secure. But of course, redirecting to a PWA requires at least limited access to the Internet the first time the user does this. We just have to accept that in Browser Extension context.

While the userbase who would use the Browser Extension entirely in local modes is diminishing, we have historically made a big effort, as @kelson42 says, to remove unnecessary inline JS from standard ZIMs that can be read in Restricted mode. It needlessly places an obstacle in front of users with old PCs and old browsers, who just want offline Wikipedia (the most common use case). Hence the reason for my hyper-vigilance about finding any inline JS creeping in to baseline ZIMs such as Wikimedia projects.

[Jaifroid]

#1276 has revealed that on both Windows and Linux, all tests linked to inline JS are not working properly.

These tests are known to work ok on all readers (kiwix-desktop is probably the last one not yet tested), at least I'm 100% sure they do work on kiwix-android, kiwix-apple, kiwix-serve, kiwix-js

These tests are very specific to zimit/warc2zim. They rely on proper operation of wombat, the JS library used by zimit/warc2zim ZIMs to replace dynamically / on-the-fly URLs loaded by JS code. It is hence "normal" that the URL inside JS code is "online", it will be rewritten by wombat to the proper in-ZIM URL.

@benoit74 It's perfectly fine to use the PWA (pwa.kiwix.org or browser-extension.kiwis.org, either is fine) to run those tests, as they are tests of Zimit, which by definition requires a lot of inline JS which is fully supported by PWAs. So no problem there on my side.

[benoit74]

Thanks for all the details @Jaifroid !

[veloman-yunkan]

While opening in kiwix-desktop the Javascript page of the "Zimit test website" ZIM file the following output is generated in the terminal:

js: You are using the fallback version (version 2.0.0-dev9) of wombatSetup.js, this is not expected outside dev
js: 'window.webkitStorageInfo' is deprecated. Please use 'navigator.webkitTemporaryStorage' or 'navigator.webkitPersistentStorage' instead.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/content.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/w%C3%A9bsite.test.openzim.org/javascript/content.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/cont%C3%A9nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/w%C3%A9bsite.test.openzim.org/javascript/cont%C3%A9nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/cont%F0%9F%8E%81nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/w%C3%A9bsite.test.openzim.org/javascript/cont%F0%9F%8E%81nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/cont%C3%A9nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/w%C3%A9bsite.test.openzim.org/javascript/cont%C3%A9nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/w%C3%A9bsite.test.openzim.org/javascript/content.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/content.txt%3Fquery%3Dvalue. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/content.txt%3Fquery%3Dvalu%3De. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/content.txt%3Fquery%3Dvalu%25e. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/cont%3Fnt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/cont%3Fnt.txt%3Fquery%3Dvalue. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/cont%F0%9F%8E%81nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/w%C3%A9bsite.test.openzim.org/javascript/cont%F0%9F%8E%81nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/cont%F0%9F%8E%81nt.txt. URL scheme "zim" is not supported.
js: Fetch API cannot load zim://7aaf6cc6-e68e-fbba-af18-da96f0004aa3.zim/website.test.openzim.org/javascript/cont!nt.txt. URL scheme "zim" is not supported.
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught (in promise) TypeError: Failed to fetch
js: Uncaught TypeError: Failed to resolve module specifier "resources". Relative references must start with either "/", "./", or "../".
js: Uncaught TypeError: Failed to resolve module specifier "javascript/resources". Relative references must start with either "/", "./", or "../".

Now going to debug why the URL scheme "zim://" isn't available to the Fetch API.

[kelson42]

Now going to debug why the URL scheme "zim://" isn't available to the Fetch API.

See also #917

[veloman-yunkan]

Looks like the only fix to this issue is to use Qt 6.6 or newer - see https://stackoverflow.com/questions/64892161/qtwebengine-fetch-api-fails-with-custom-scheme

[kelson42]

Looks like the only fix to this issue is to use Qt 6.6 or newer - see https://stackoverflow.com/questions/64892161/qtwebengine-fetch-api-fails-with-custom-scheme

Would you be able to implement the fix based on qt6+?

FYI, once 2.4 release, the first thing I would like to do would be to move to QT6 as default (only?) version and library for our binaries.