WebXR Integration 🥽 (again)

[abcamiletto]

This PR adds out-of-the-box support for VR devices.

The pain point is that VR devices require HTTPS for WebXR to function, as noted in the react-three/xr documentation. This appears to be a strict requirement.

While this is not a problem for someone using a viser export for a webpage, it is for anyone using viser as a 3D visualizer as the default server is in HTTP.
To address this, the current draft implementation dynamically generates self-signed certificates and uses them to launch an HTTPS server, enabling VR mode. It's ugly but it works.

If you're open to supporting HTTPS and merging this PR, I'm happy to refine the implementation further.

There are a couple of open points:

• Would you be okay with adding cryptography as an optional dependency? For users that just want to have an https visualizer without having to provide their own SSL certificate. Alternatively, we could keep the https functionality only by passing explicitly ssl_context
• Do you have any preferences regarding how the API should be structured?

Currently, there is only an (ugly) https=False flag to ViserServer
From my perspective as a viser user myself, developing a downstream library, I would like to have something along the lines

server = ViserServer(enable_webxr=True) # without certs
server = ViserServer(ssl_context=ssl_context) # for explicit https

I am also open to splitting this PR into two, one for HTTPS and one for WebXR support.

Let me know what you think!

To Do:

• Make Enter VR / Enter AR buttons optional, appearing only when a VR environment is detected.
• Allow users to provide their own certificates.
• Make cryptography an optional dependency.
• Leverage controller inputs to move around in the scene

[brentyi]

Thanks @abcamiletto for the PR! I'm also interested in thoughts from @cvachha.

My initial feelings:

• For API, both server.ViserServer(enable_webxr=True) and server.scene.enable_webxr(True) seem okay to me. I guess this should raise a warning or error if HTTPS is not also enabled.
• For configuring HTTPS, I'm wondering what you think of either:
  • (1) Shipping a pair of keys directly in viser instead of generating them online. Document appropriately. This could eliminate the cryptography dependency.
  • (2) Just exposing ssl_context as an argument, and have the user figure out the details. This might be verbose, but it's maximally flexible for the user. And there'd be less to maintain on our end.
• It would be nice to avoid extra dependencies, but adding cryptography wouldn't be the end of the world. For simplicity I'd prefer to either add it as a core dependency or leave it out entirely.

[abcamiletto]

Hi!
Thanks for the quick response.

I think it is reasonable to ship a pair of keys instead of having cryptography as a dependency. I did not think of that, and it seems a better solution to me.

A more detailed proposal could be

• ViserProject(enable_webxr=True, ssl_context=None) --> launches an https server with the shipped self-signed certificates. Raises a warning that it's a bad idea to deploy it as it is.
• ViserProject(..., ssl_context=ssl_context) --> launches an https server with the provided ssl_context, no matter what enable_webxr is. The front-end takes care of showing VR related buttons (if it detects a VR browser)

This seems feasible to me and I don't foresee major blockers.

[cvachha]

I think the solution to use HTTPS to get webxr working looks great! It probably would be useful to have a method to toggle/activate the webxr buttons in the python script if the develop prefers instead of having them on by default since certain applications like Nerfstudio won't be able to directly use webxr for most functionality (panels, nerf/splat rendering).

There's also a list of features I'm planning on working on including features to make this more than just a vr viewer like controller support, hand tracking, and panels (though adding 2D floating panels might be difficult since I believe HTML can't render in webxr).

[richardrevesz]

Hey @abcamiletto , thanks for the great work!

I wanted to raise your attention that when I try to build it, I get this error

~\viser\src\viser\client>yarn build
yarn run v1.22.22
$ tsc && vite build
src/App.tsx:11:10 - error TS2300: Duplicate identifier 'Canvas'.

11 import { Canvas, useThree, useFrame } from "@react-three/fiber";
            ~~~~~~

src/App.tsx:52:10 - error TS2300: Duplicate identifier 'Canvas'.

52 import { Canvas } from "@react-three/fiber";

Commenting out line 52 resolves it. :)

I still need to test it in VR, on mobile in AR mode it is not working properly (just renders the view on a plane), but VR does, I am happy :)

[richardrevesz]

Hey @abcamiletto , thanks for the great work!

I wanted to raise your attention that when I try to build it, I get this error

~\viser\src\viser\client>yarn build
yarn run v1.22.22
$ tsc && vite build
src/App.tsx:11:10 - error TS2300: Duplicate identifier 'Canvas'.

11 import { Canvas, useThree, useFrame } from "@react-three/fiber";
            ~~~~~~

src/App.tsx:52:10 - error TS2300: Duplicate identifier 'Canvas'.

52 import { Canvas } from "@react-three/fiber";

Commenting out line 52 resolves it. :)

I still need to test it in VR, on mobile in AR mode it is not working properly (just renders the view on a plane), but VR does, I am happy :)

Ok, tested in VR, it says "This page isn't responding"

My setup is Meta Quest 3, connected with Meta Quest Link (AirLink) to my PC, running Chrome (134.0.6998.178).
When I click Enter VR, Chrome opens, but throws the error above.
(Chrome is running supersplat viewer for example happily)

Please let me know if you need any further input on this, I am happy to provide logs or whatever you might need :)

[abcamiletto]

Hi! I will work again on this soon. The setup I was testing with was Meta Quest 3, wireless, using Quest native browser.
That seemed to be working. But there are, of course, still quite some edges to smooth out for this branch to be ready.

[richardrevesz]

Oh, I see. It didn't occur to me, because my 20 thousand iteration is 1.8Gb, so I thought my best chance to visualize it is in tethered mode, but I will try that too.
Thank you, and looking forward to see this coming together :)

[richardrevesz]

Oh, I see. It didn't occur to me, because my 20 thousand iteration is 1.8Gb, so I thought my best chance to visualize it is in tethered mode, but I will try that too. Thank you, and looking forward to see this coming together :)

Okay, so I tested with the same setup. For me the view in VR is simply on a 2D plane, so basically the same as before hitting "Enter VR", and everything else is just black. Similar to the behavior on mobile when hitting "Enter AR".

Also, once I exit this view with the menu button and close the immersive experience, I can't re-open it until I restart the browser, so something is not exiting gracefully.