Right now screen casting is not supported for Wayland. Currently the screen is captured either via autopilot, which only supports X11, or using XShmCreateImage here: https://github.com/H-M-H/Weylus/blob/435373be5141a2b03aeab6549225a5a5790ed1b7/lib/linux/capture.c .

To get Wayland to work there are some things I still need to find out:

• check if we a running Wayland or X (this is probably easy)
• find an API to capture the screen and selected windows, that is fast enough
  • preferably similar to the X11 API that allows me to get a pointer to the RGB buffer of some window
• API to request which windows should show up in the window selection menu

If someone wants to help me here a minimal working example of taking a screenshot of a selected window in Wayland would be great, provided it is written in Rust, C or C++. Alternatively point me to a tool that already does that.
I will probably write some Trait to abstract different window systems, so I would rather not want a pull request now.

About the input issues: I am not sure what's causing that, I do not do anything specific to X11.
Can you tell me if libinput debug-events or evtest outputs anything strange?

Also this should probably be a separate issue.

Looks like the tool you used (https://github.com/ammen99/wf-recorder) for recording and https://github.com/emersion/grim can be used as a reference.

Hey @H-M-H, thank you for your willingness to work on Wayland support and I'm sorry I can't do more to help right now. Realistically, I don't think I'm going to be able to help you debug libinput or evtest right now. I love this project and think it's absolutely amazing. Wish I had more time to help out.

Alright, I totally understand that, thanks for your report anyways! Maybe someone else can help out.

I did some research and the situation regarding capturing the screen or even windows on wayland is pretty dire. There is no equivalent to X11's XShmGetImage or XGetImage, Wayland itself just does not support capturing the screen. Instead it is the responsibility of the compositor to expose an API to take screenshots. But unfortunately this is not standardized, there is:

• wlroots' wlr-screencopy-unstable-v1 which does not work for gnome or kde
• there is Wayland's reference compositor Weston with another way
• there is gnome's screenshot and screen cast interface via dbus, both probably too slow for proper streaming
• there is kde's dbus interface for screenshots

What about PipeWire?

Capture and playback of audio and video with minimal latency.

Interesting, I did not know about PipeWire, the docs are a bit lacking but PipeWire is used by webrtc, so a look at how they do it might help. @pointhi do you know what latency one can expect using PipeWire? For reference: capturing a 1920x1080 raw bgr0 frame on my system takes about 0.8 milliseconds if the screen contents do not change too much.

I never used PipeWire, but it was designed for remote desktop applications on Wayland to my knowledge.

strangely enough, unfocused windows do get captured, until the very first event sent to them which focuses them and then the output becomes black

I think I have a good idea of what's going on: If a window is not focused, Weylus captures it directly by its window handle but if it is active Weylus captures the part of the root window containing the window to be captured. The reason for this is to have things like context menus that are their own windows in themselves still show up. But I guess XWayland doesn't like capturing the root window and just decides to have it black.

Please check if things work better with the latest build: https://github.com/H-M-H/Weylus/actions/runs/585393503#artifacts

I started experimenting with pipewire as this seems to be the most promising solution here: https://github.com/H-M-H/Weylus/tree/wayland

For now only requesting a screen cast via org.freedesktop.portal.ScreenCast is implemented. That means there now is a file descriptor that can be used to connect to pipewire. Unfortunately the docs on how to use pipewire directly are still very lacking. For example pw_remote_connect_fd mentioned in the docs for org.freedesktop.portal.ScreenCast has been removed in pipewire 0.3 and seems to be pw_context_connect_fd now. All I could find on video processing is: https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/doc/tutorial5.md

It looks like pipewire implements a gstreamer plugin, I suspect gstreamer's API is not as badly documented, so this is perhaps worth a look.

A problem that remains is that it looks like there is no way to get the position of a captured window, which means Weylus won't be able to properly map the input to the window. As it stands only capturing the full desktop will be usable under Wayland.

For anyone interested, d632a7d implements screen casting via gstreamer. Binaries can be found here. There still are quite some issues, refer to the commit message for details.

Looks like once this is resolved, working with cursor metadata via gstreamer will make dynamically showing and hiding the cursor possible.

Regarding window titles I opened a feature request here: flatpak/xdg-desktop-portal#569

Thanks a lot for working on this :)

Thanks! So far walking this path has been rather rocky and I have hit quite some bugs on the way:

Bugs

• kwin_wayland dies and tears down the entire plasma desktop
• PipeWire deadlock No.1
• PipeWire deadlock No.2
• PipeWire returns empty buffers when it shouldn't
• Missing context for position property of pipewire streams
• Inconsistency (fortunately not really a bug) between xdg-desktop-portal-gtk and xdg-desktop-portal-kde

Missing features

• PipeWire streams have no name
• Can not get position of PipeWire streams (this breaks touch input mapping)
• Can not get cursor data from PipeWire (breaks switching displaying cursor without rerequesting permission to capture the screen)

Fortunately I could workaround all PipeWire deadlocks and at least on GNOME Weylus can now capture the desktop and windows and so far it seems not too unstable.

Feel free to try the latest builds.

Considering the current situation I will definitely mark Wayland screen capture experimental and make it opt in.

Hi,

I've tried installing weylus on a Pi 4 running manjaro sway. Everything works other than desktop capture. I can capture chromium tabs and weylus itself, but the desktop capture returns the following warning:

WARN weylus::capturable: Failed to get list of capturables via dbus/pipewire: Failed to obtain screen capture.

Chromium itself is able to capture the desktop, as I am able to share the screen from web apps such as Microsoft Teams. Before running I set:

export XDG_SESSION_TYPE=wayland
export XDG_CURRENT_DESKTOP=sway
/usr/lib/xdg-desktop-portal -r & /usr/lib/xdg-desktop-portal-wlr

This works for chromium, and idea what could be done to get it to work in weylus?

This means that Weylus only captures windows available via XWayland. Please check if https://gitlab.gnome.org/snippets/19 also fails.

Some more questions: How exactly does the process for window/desktop selection work with chromium, do you get some kind of popup window that lets you select what to capture and does that popup window also appear for Weylus?

Hi, also getting WARN Failed to get list of capturables via dbus/pipewire: Failed to obtain screen capture.. No native Wayland windows and only a desktop in the Capture list. The desktop option sets the correct aspect ratio but shows only black.

This means that Weylus only captures windows available via XWayland. Please check if https://gitlab.gnome.org/snippets/19 also fails.

Yes, this snippet works.

Some more questions: How exactly does the process for window/desktop selection work with chromium, do you get some kind of popup window that lets you select what to capture and does that popup window also appear for Weylus?

Don't know how chromium does this, but firefox (for example https://mozilla.github.io/webrtc-landing/gum_test.html) and the python snippet both change the mouse cursor to a cross and I need to click on a desktop I want to share. Actually, Weylus also does this.

Also, input forwarding to desktop does work (but there is no image).

EDIT: Arch Linux, Sway WM, AMD gpu

In Gnome/Mutter, the whole desktop works fine (Entire screen option)! It's pretty cool! Now I wish you could use Mutter's headless stuff and create a virtual screen! :)

@dron1885 @l-o-l I got it working under sway now, please check if the latest build works for you as well: https://github.com/H-M-H/Weylus/actions/runs/1012629238#artifacts

@mattpepin do you know how Mutter's headless stuff works? Perhaps I could add some examples or even an accompanying script to Weylus.

@H-M-H Can confirm that it's working. Thanks!
(But it seems it requires additional testing in multimonitor setup. From my quick testing, while Waylus was showing a second monitor all inputs were translated to the first one.)

Can confirm that it's working. Thanks!

Excellent, thanks for testing!

From my quick testing, while Waylus was showing a second monitor all inputs were translated to the first one.

This is a known problem and sadly not easily fixed. For monitors xdg-desktop-portal provides positions but unfortunately those are compositor dependent/the coordinate system is unspecified and there is no straightforward way to translate them into something useful. For windows the situation is even worse as there is no position given.

Unfortunately I do not see this being solved in the foreseeable future. In my opinion the whole screen recording/remote desktop situation around Wayland is most unfortunate. There have been discussions to add screen recording to Wayland for years with the result that the team behind Flatpak is now somehow responsible for this and there are no plans to extend Wayland itself or provide an official API/Protocol. This means the Flatpak devs effectively have full responsibility/control over how screen casting works. The problem now is that Flatpaks focus is not exactly to provide a nice screen cast/remote desktop interface for Wayland [1]. I can not and will not blame them for that, it is totally understandable if they deem some features required to get a nice remote desktop experience working unnecessary for Flatpak.
This is why I don't quite believe in this working anytime soon and the whole discussion doesn't even touch pipewire, which is still under heavy development.

If despite this anyone knows how to obtain the positions of pipewire streams in a way useful to Weylus I very much would like to hear that though.

[1]: There actually is a remote desktop interface. But among other things it doesn't handle resizing windows or pressure sensitivity of touch input.

Hello,

just a clarification. What does it mean that for Wayland there is no support for "capturing the cursor"?

If I move the stylus on my tablet, will the cursor move accordingly?

Please try this build here

@H-M-H
The crash is gone, but now I get: ERROR weylus::websocket: Got invalid id for capturable: 0.

I think something is missing in my setup, but i don't know what.
Why i keep getting Can't open display: wayland-1 when running whithout --no-gui? doesn't wayland-1 is the default socket under wayland?

@mgonzalezm alright, so autopilot was indeed the cause for the crash. Have you made sure xdg-desktop-portal-wlr is up an running? You can verify whether it is running by checking if screen capturing works with OBS or Firefox.

The error message you got means that Weylus could not find any screens or windows to capture. Have you made sure to start Weylus with --wayland-support?

Regarding Can't open display: wayland-1, it looks like fltk only supports X/XWayland. I guess I'd have to rewrite the gui with another library (this wouldn't be that bad as the gui is rather minimal anyways, the actual problem is finding a nice gui library).

@H-M-H sorry I forgot --wayland-support, my bad.

and yes xdg-desktop-portal-wlr is running:

$ systemctl --user status xdg-desktop-portal-wlr
● xdg-desktop-portal-wlr.service - Portal service (wlroots implementation)
Loaded: loaded (/usr/lib/systemd/user/xdg-desktop-portal-wlr.service; static)
Active: active (running) since Thu 2022-02-10 19:00:34 CST; 23h ago
....

Now I can connect, and select Pipewire Unknown, path:34 on the drop-down list (this is the only option), but after few seconds:

WARN weylus::websocket: Failed to init screen cast: Element failed to change its state!

and only a black screen is shown on the browser

I think that not having a wayland compatible gui is not a big deal, as long as the same options be available on the command line.

Thank you for your help and your excellent work

Hmm, this error stems from within gstreamer somewhere in this call to set_state:

@H-M-H Yes OBS works ok, I can see my screen.

I wouldn't now if the snippet works, after the pop-up to select the monitor to share, this is all I can see in the terminal

$ python xdp-screen-cast.py
session /org/freedesktop/portal/desktop/session/1_780/u1 created
sources selected
streams:
stream 36

@mgonzalezm I am sorry, I was a little busy with other things the last days. The script I linked should start recording your screen using gstreamer and then open a window that shows the recording. This typically results in some funny recursion to look at. If this is not the case this pretty much verifies that this is a gstreamer issue and not something related to pipewire only (OBS directly interfaces with pipewire, skipping gstreamer). What's the output of gst-inspect-1.0 pipewiresrc | grep Version. At least on my system using kwin_wayland (KDE) and pipewire version 0.3.47 streaming works fine using the script as well Weylus itself.

Hello,

I'm facing the same problem as @noneucat, i.e. I get a 2022-05-04T16:29:01.243927Z WARN weylus::websocket: Failed to init screen cast: Failed to create element from factory name! and screen sharing does not seem to work on my OS.

I'm using NixOS, and I tried running weylus with for example GST_PLUGIN_PATH=/nix/store/v8yddpay5ilm3m59yj64pdnc13gjkav1-pipewire-0.3.49-lib/lib/gstreamer-1.0/libgstpipewire.so weylus without much success (the module seems loaded according to strace, but I still get this message).

Any idea what could cause this / how to debug ?

Thanks.

Edit: Running weylus with GST_DEBUG=5, I get more information:

0:00:15.874726515 177259 0x56307e6b5490 DEBUG     GST_PLUGIN_LOADING gstpluginfeature.c:106:gst_plugin_feature_load: loading plugin for feature 0x56307e6b8920; 'pipeline'
0:00:15.874760196 177259 0x56307e6b5490 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:489:gst_element_factory_create_with_properties: creating element "pipeline"
0:00:15.874787417 177259 0x56307e6b5490 DEBUG                GST_BUS gstbus.c:256:gst_bus_init:<GstBus@0x56307e69de70> created
0:00:15.874805956 177259 0x56307e6b5490 DEBUG                    bin gstbin.c:498:gst_bin_init:<GstBin@0x7fc1bc004160> using bus <bus0> to listen to children
0:00:15.874822389 177259 0x56307e6b5490 DEBUG                GST_BUS gstbus.c:256:gst_bus_init:<GstBus@0x56307e69df30> created
0:00:15.874842136 177259 0x56307e6b5490 DEBUG               GST_POLL gstpoll.c:681:gst_poll_new: 0x56307e6a0e80: new controllable : 1
0:00:15.874863422 177259 0x56307e6b5490 DEBUG               GST_POLL gstpoll.c:848:gst_poll_add_fd_unlocked: 0x56307e6a0e80: fd (fd:29, idx:0)
0:00:15.874872373 177259 0x56307e6b5490 DEBUG               GST_POLL gstpoll.c:1014:gst_poll_fd_ctl_read_unlocked: 0x56307e6a0e80: fd (fd:29, idx:0), active : 1
0:00:15.874879547 177259 0x56307e6b5490 DEBUG                GST_BUS gstbus.c:310:gst_bus_new:<bus1> created new bus
0:00:15.874887630 177259 0x56307e6b5490 DEBUG          GST_PARENTAGE gstelement.c:3478:gst_element_set_bus_func:<GstPipeline@0x7fc1bc004160> setting bus to 0x56307e69df30
0:00:15.874895376 177259 0x56307e6b5490 DEBUG               pipeline gstpipeline.c:248:gst_pipeline_init:<GstPipeline@0x7fc1bc004160> set bus <bus1> on pipeline
0:00:15.874904199 177259 0x56307e6b5490 DEBUG    GST_ELEMENT_FACTORY gstelementfactory.c:522:gst_element_factory_create_with_properties: created element "pipeline"
0:00:15.874921808 177259 0x56307e6b5490 WARN     GST_ELEMENT_FACTORY gstelementfactory.c:701:gst_element_factory_make_with_properties: no such element factory "pipewiresrc"!
0:00:15.874930773 177259 0x56307e6b5490 DEBUG        GST_REFCOUNTING gstpipeline.c:258:gst_pipeline_dispose:<pipeline0> 0x7fc1bc004160 dispose
0:00:15.874945344 177259 0x56307e6b5490 DEBUG        GST_REFCOUNTING gstbin.c:517:gst_bin_dispose:<pipeline0> 0x7fc1bc004160 dispose
0:00:15.874968024 177259 0x56307e6b5490 INFO         GST_REFCOUNTING gstelement.c:3382:gst_element_dispose:<pipeline0> 0x7fc1bc004160 dispose
0:00:15.874976511 177259 0x56307e6b5490 DEBUG               GST_POLL gstpoll.c:772:gst_poll_free: 0x56307e6a0e80: freeing
0:00:15.874992833 177259 0x56307e6b5490 INFO         GST_REFCOUNTING gstelement.c:3428:gst_element_dispose:<pipeline0> 0x7fc1bc004160 parent class dispose
0:00:15.875000638 177259 0x56307e6b5490 INFO         GST_REFCOUNTING gstelement.c:3460:gst_element_finalize:<pipeline0> 0x7fc1bc004160 finalize
0:00:15.875007630 177259 0x56307e6b5490 INFO         GST_REFCOUNTING gstelement.c:3465:gst_element_finalize:<pipeline0> 0x7fc1bc004160 finalize parent
2022-05-06T09:50:27.815216Z  WARN weylus::websocket: Failed to init screen cast: Failed to create element from factory name!

Full log: log.txt

@rgrunbla: Thanks for looking into this! GST_PLUGIN_PATH should point to the directory with the plugin, not the shared library itself. What is the output of gst-inspect-1.0 pipewiresrc?

Also, if this problem persists, please open a separate issue.

Maybe this solution using vdl and deskreen could be inspiring for Wayland support ?

@H-M-H @zachlatta This issue should be closed as pipewire support has been added

It seems to me that, under Wayland (I am using sway), one cannot select a specific window, but "Pipewire unknown", which shows the current desktop. I am not sure whether it is possible to capture a specific window under Wayland, however.

To change the window, you can use the 'refresh sources' button, or reload the page, then choose the window you want to mirror on your screen.

I am not sure whether I misconfigured something. I compiled the latest Weylus without compiling ffmpeg (I failed to compile ffmpeg for some reasons). I am under sway.

I have exactly tried this: after clicking "refresh", the mouse icon becomes a cross, which seems to indicate me to select a windows, and after selection, the title of selected window does not appear in the drop list.

No, it is accessing the current display (although the cross icon seems to indicate that I was supposed to select a specific window).

Are you selecting Pipewire or X11 display? The default cursor on X11 is an X. You only choose the window on the actual computer, it should show up on the monitor, once you load the webpage. 25 Aug 2023 11:50:17 Iey4iej3 @.***>:

I believe that it is Pipewire (the cursor, more precisely being "+" in place of "X", is very different from the default cursor of X): it is triggered by "refresh list" or truly refreshing the site, and without clicking anything, the window list is empty.

After having searched, I see that it is an issue of xdg-desktop-portal-wlr: emersion/xdg-desktop-portal-wlr#107

To anyone getting WARN weylus::websocket: Failed to init screen cast: Failed to create element from factory name!: It appears that latest release is very old and does not contain proper Wayland support yet. Building from master seems to fix the issue.

For any Arch + PipeWire users getting the Failed to init screen cast: Failed to create element from factory name! error, installing the gst-plugin-pipewire package fixed it for me.

Is there a way for non Gnome and KDE wayland users to use weylus with video when it relies on org.freedesktop.impl.portal.RemoteDesktop? Only the KDE and Gnome portals have implemented it yet and I can't get them to work under Hyprland. It's not ideal if only these two IDE's can use video sharing on wayland

@dryBoneMarrow as far as I understand the Hyprland wiki about screen sharing it is supposed to work with xdg-desktop-portal and weylus should support that. This is the de-facto standard for screen sharing on Wayland, there is a list with supported DEs on the arch wiki: https://wiki.archlinux.org/title/XDG_Desktop_Portal#List_of_backends_and_interfaces

On KDE Neon 6.2, weylus just refuses to work: Failed to init screen cast!
Tried to connect tablet directly through adb usb, and it only works with janky Wayland to X11 bridge.