BlockifyVR (Virtual Reality)

[Brackets-Coder]

BlockifyVR

This is a very-much-still-work-in-progress Virtual Reality extension I've been working on since January 27th of 2023.

The end goal is to have:

• Little to no performance overhead with the extension
• Cross-platform compatibility for all major 6DOF headsets A-frame supports
• An easy-to-use system that's coherent and efficient
• Works well with any 3D engine
• Bug-free enough for optimal user experience without frequent disruptions

Here are some things to note:

• It includes a local copy of the A-frame JS library, version 1.6.0, MIT License. This is for VR Management, rendering, and ease of development as I'd have to write the entire thing from scratch using WebXR, WebGL, and the Gamepad API otherwise. Setting up the VR boilerplate and stereoscopy and other things would be a pain without it.
• Gallery thumbnail, documentation, and sample projects are not currently available and will be worked on as I get nearer towards release
• The rendering uses a plane fixed to the camera which directly mirrors the scratch stage with an unshaded texture each frame. This is relatively performant and works well, but there could be better solutions to look into. The plane is placed 1 meter away from the camera and A-frame uses stereoscopic rendering to reduce eye strain. Note that the plane automatically scales to the size of the camera (display) while attempting to maintain the ratio of the stage. I use this instead of drawing directly to the canvas because it won't let me and I would have to re-write the renderer that's already tailored for VR'.
• The above note means that ONLY THE STAGE LAYERS will be rendered. Any 2-dimensional overlays (reporter monitors, "ask" block, etc.) WILL NOT work in a VR session.
• Sorry the code's a bit messy and outdated and bad

Upcoming release plan:

v1.0-pre-alpha: Earliest version. Minimal functionality, proof-of-concept version. Not available here. Not open source.

v1.0-alpha: work-in-progress. Expect significant bugs and poor performance. Should not be used.

v1.0-beta: Cross-platform compatibility, major bug fixes, and optimizations have been added.

v1.0-release-candidate: Current version. Feature complete. Meets standards of full release with minimal issues. During this release period, preparations for full release include gallery thumbnail, sample project, documentation, etc.

v1.0: First release. Feature complete and meets all standards of great performance, very few issues, cross-platform compatibility, and intuitive design.

v1.5: Bugfixes and optimizations. Possibly a small feature or change, such as controller vibrations.

v2.0: Big update. Most likely something like hand tracking support for some platforms. Also will include some minor bugfixes and optimizations.

v2.2: Bugfixes and optimizations.

Active Development

v1.0-release-candidate

• Bug fixes and optimizations
• Prep for release
• Sample project
• Website for extra resources and Documentation

[hammouda101010]

that looks cool

[hammouda101010]

that looks cool

It looks cool on the flat screen right now, but it's not so great when you try it. You'd probably get motion sickness. I'm still working on it.

sadly i dont think that's possible to stop this.

[Xeltalliv]

Does this show the same scratch 3D perspective to both eyes?

[Brackets-Coder]

Does this show the same scratch 3D perspective to both eyes?

A-frame has stereoscopic rendering built in and I've enabled it in the rendering settings, so no. This fixes some issues with visualizing depth, so the eye offset allows your brain to process the image with more clarity. Even if the matrices aren't used in the renderer, it should properly use the correct eye offset. Why do you ask, just out of curiosity? If you'd like I could add a block that disables or enables this feature, in case the Scratcher wants to build that framework themselves, which I'd doubt.

Keep in mind this is still a work in progress and most of the code is many months old so it's not the most efficient way to do VR.

[Xeltalliv]

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

[Brackets-Coder]

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

I believe that there can be two XRViewerPose matrices for each eye if available, according to Mozilla Docs. You might want to look into that again. It'd be great if Augmented Reality could support AR headsets such as the Oculus Quest 3 in addition to just mobile touchscreen devices. There should be a views array that can do this. An example can be seen here. Just a suggestion though. I simply think having AR headsets would be a nice update.

[Xeltalliv]

I'll take a look when I have time. Thought it will probably be hard because the only kind of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

[Brackets-Coder]

I'll take a look when I have time. Thought it will probably be hard because the only kinf of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

As for VR testing, you wouldn't really need a VR headset. I've used Meta Quest's Immersive Web Emulator extension for testing when I didn't care to load up the entire headset.

[Drago-Cuven]

ah yes, I remember seeing a trailer for this on scratch. I wanted to help, but now I can't

[Drago-Cuven]

nvm I think I can
i'm Martinelplayz on scratch btw

[Thebloxers998]

could i try the extension?

Hi siskka7 it's me Thebloxers998

[siskka7]

Thebloxers998

hmm https://github.com/TurboWarp/extensions/blob/2837a7adfbfc2e3aba49108003686affa1cb7beb/extensions/MasterMath/BlockifyVR.js

[Thebloxers998]

Thebloxers998

hmm https://github.com/TurboWarp/extensions/blob/2837a7adfbfc2e3aba49108003686affa1cb7beb/extensions/MasterMath/BlockifyVR.js

I'm not sure what you're "hmm"ing. It's just the source code.

Hi, nice extension

[Xeltalliv]

I'll take a look when I have time.

I took a look and have no idea. It's all code specific to AFRAME. I don't know AFRAME. AFRAME is too big and complicated for me to try to debug, especially if the only tool in my disposal is WebXR API Emulator browser extension.

[Thebloxers998]

Nic3

[siskka7]

4. R.js" to download JUST THE EXTENSION. Do NOT download the entire .zip source code of the whole repositor

thank for your very fast response

[Brackets-Coder]

4. R.js" to download JUST THE EXTENSION. Do NOT download the entire .zip source code of the whole repositor

thank for your very fast response

You're welcome. I try to stay on top of things.

Unfortunately I don't have a fully functional Sample Project published for you to use but if you'd like I can put the work-in-progress one on the releases page. Just let me know

[Thebloxers998]

"Ready for review" Yay

[Brackets-Coder]

Matrix blocks don't work. Attempting to resolve. Not ready for review

[siskka7]

it works now .i found on quest 3 that there is down black stripe like it looks like it was flat screen folowing me (no rotation lag) but screen ends down

[Brackets-Coder]

it works now .i found on quest 3 that there is down black stripe like it looks like it was flat screen folowing me (no rotation lag) but screen ends down

yes, I'm aware that in certain contexts there is a small "black strip" at the bottom of the screen. This is a result of the Display Plane scaling logic attempting to account for the variation in aspect ratio between the Scratch stage and physical display of the Quest 3. I'm still attempting to resolve this, so the temporary and best solution is to switch the stage size to a wider aspect ratio, such as 640x360, which is by far the best for quality and performance.

[Brackets-Coder]

For better viewing experience, use fullscreen mode with a stage size like 800x450 for a higher resolution. This may come with a small negative performance impact.

[Brackets-Coder]

Progress update

So I've fixed the really big display scaling bug but in the process of testing for release I discovered that the get matrix block doesn't work at all because of entirely incorrect implementation.
Soo.... I've decided to do something crazy but necessary.
I've decided to remove the entire pose-matrices component and attempt to retrieve the viewer pose, projection matrices, combined matrices, etc. directly from the camera and rewrite the entire get matrix block from scratch to do everything in just a few lines.
Is it stupid? Maybe. do I think it will work? Maybe.

Probably just a few more weeks at most before I open this up again but it's hard to say

[siskka7]

i found that when blocks dont work on quest 3 (not big problem for me beceause i can live withow it + we have if blocks)

[Brackets-Coder]

i found that when blocks dont work on quest 3 (not big problem for me beceause i can live withow it + we have if blocks)

Could you please specify which blocks weren't working on Quest 3?
I'm assuming you're referring to the "When" hat blocks, such as "when key pressed" or "when controller connected".

Please note that most blocks will return nothing at all unless you're in a VR session (e.g., pressing the block to reveal its value in the editor doesn't work). They must be used properly and will only return their proper values after the enter vr mode block has finished executing. The same rules go for the "when" hat blocks.

This is why I needed to provide a sample project, as I haven't made the blocks return any in-editor fallbacks yet.

By my testing, all blocks currently work on quest 2 except the get matrix block.

[siskka7]

i found that when blocks dont work on quest 3 (not big problem for me beceause i can live withow it + we have if blocks)

Could you please specify which blocks weren't working on Quest 3? I'm assuming you're referring to the "When" hat blocks, such as "when key pressed" or "when controller connected".

Please note that most blocks will return nothing at all unless you're in a VR session (e.g., pressing the block to reveal its value in the editor doesn't work). They must be used properly and will only return their proper values after the enter vr mode block has finished executing. The same rules go for the "when" hat blocks.

This is why I needed to provide a sample project, as I haven't made the blocks return any in-editor fallbacks yet.

By my testing, all blocks currently work on quest 2 except the get matrix block.

Oh i inspected only when (key/button) pressed and i was in vr mode not in editor

[siskka7]

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code)
Edit: probably not useful

[siskka7]

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

[Brackets-Coder]

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.

Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

[siskka7]

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.

Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

[Brackets-Coder]

I just published a new release version here which should fix the issue with the "when button" blocks. Note that I haven't yet tested this to confirm it's working but it was working last time before I changed the inputs to accept reporters. I've reverted this change.

Note that this release also brings experimental WORK IN PROGRESS changes to the "get matrix" block which SHOULD NOT be used.

For now, the best alternative is to just use the default perspective projection Simple3D offers and use the "camera FOV" block with it. This will only work if you confirm your headset rotation transformations are computed in the right order (ZXY, with headset Y rotation as X and headset X rotation as Y in Simple3D, varies between renderers)

[siskka7]

here is my test project that logs vr input https://github.com/siskka7/siskka7/blob/main/vr%20blockify%20input%20logger.sb3
(tutorial in project) note that this is before was "when block" was fixed i hope this will help developing

[Brackets-Coder]

here is my test project that logs vr input https://github.com/siskka7/siskka7/blob/main/vr%20blockify%20input%20logger.sb3 (tutorial in project) note that this is before was "when block" was fixed i hope this will help developing

Let's continue this discussion here and not take up so much of the extension PR discussion on testing.

this discussion already has the most comments of all the PRs

[Drago-Cuven]

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

https://github.com/immersive-web/body-tracking

[Brackets-Coder]

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

https://github.com/immersive-web/body-tracking

Good to know, but it would still be too impractical to implement, not many would use body tracking. Hand tracking is a different story, and may come in 2.0, but even then I doubt it'd be used often as controllers are just easier and more cross-platform

I'll consider it but I've already been on this extension two years so idk

also "This API primarily exposes the joints of a person's body. It can be used to render an avatar in VR scenarios. It does not provide access to a full hbodynd mesh."

also "Each joint space is an XRSpace, with its -Y direction pointing perpendicular to the skin, outwards from the palm, and -Z direction pointing along their associated bone, away from the wrist. This space will return null poses when the joint loses tracking."

Probably doesn't support leg tracking, only torso + arms + hands

[siskka7]

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)
probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.
Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

I believe that the "When button pressed" block issue you're describing is due to a recent change where I made the block inputs allow reporters, but I don't think hat blocks like inputs that allow reporters. I'll get another release out soon that should fix this issue. I'm not sure what you mean by the other blocks though, everything else seems to work fine except the get matrix block

i think there is big misunderstanding i dont thought hat block but (picture1)

and the menu looks like (picture2)

[Brackets-Coder]

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)
probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.
Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

I believe that the "When button pressed" block issue you're describing is due to a recent change where I made the block inputs allow reporters, but I don't think hat blocks like inputs that allow reporters. I'll get another release out soon that should fix this issue. I'm not sure what you mean by the other blocks though, everything else seems to work fine except the get matrix block

i think there is big misunderstanding i dont thought hat block but (picture1) and the menu looks like (picture2)

intriguing. Turbowarp menus aren't supposed to look like that, but I can guarantee it's probably not an extension issue.

[siskka7]

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)
probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.
Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

I believe that the "When button pressed" block issue you're describing is due to a recent change where I made the block inputs allow reporters, but I don't think hat blocks like inputs that allow reporters. I'll get another release out soon that should fix this issue. I'm not sure what you mean by the other blocks though, everything else seems to work fine except the get matrix block

i think there is big misunderstanding i dont thought hat block but (picture1) and the menu looks like (picture2)

intriguing. Turbowarp menus aren't supposed to look like that, but I can guarantee it's probably not an extension issue.

so there is no fix idea :(

[Brackets-Coder]

Don't know why the "when button" blocks still aren't working... sigh I guess I have another bug to fix cuz it was working earlier

[siskka7]

Don't know why the "when button" blocks still aren't working... sigh I guess I have another bug to fix cuz it was working earlier
Oh did not mean that

[Thebloxers998]

Don't know why the "when button" blocks still aren't working... sigh I guess I have another bug to fix cuz it was working earlier

Dang, but how did it end up bugging. Did you check the logic of the block

[Brackets-Coder]

Don't know why the "when button" blocks still aren't working... sigh I guess I have another bug to fix cuz it was working earlier

Dang, but how did it end up bugging. Did you check the logic of the block

Yes, I checked the block but it was working earlier so it was probably a dumb oversight change that broke it.

I'll just go back in the commit history and see when it was working.