Stride Windowing Changes

[Doprez]

Perksey has brought up a discussion in the Discord for moving things over to the Silk.NET way of Windowing and Input. I wanted to bring up all of the findings I have for our current windowing implementation here since it isn't well documented anywhere I have seen.

With the work I was doing in trying to make the current windowing system work with any UI framework I have found a lot of possible areas of improvement we could move towards but they would be breaking changes.
PR: #2404
Avalonia POC: https://github.com/Doprez/custom-stride-window-poc/tree/avalonia-custom-platform
Custom SDL POC: https://github.com/Doprez/custom-stride-window-poc/tree/avalonia-custom-platform/MyGame3/SDL

First thing I wanted to do was layout all of the calls and features of the current GameWindow class as its an easy way to point out possible issues:

GameWindow Current Built-in Features

• Window title can be set through the property Title or the abstract method SetTitle(string)
• Can be Activated or Deactivated. Seems to pause and resume window rendering? This is only used internally that I can see and is only used to stop window rendering while it is being destroyed from the GameBase.
  This also includes public events that can be subscribed to.
• User resizing toggle can be changed through the property AllowUserResizing
• The Window surface bounds are controlled by the property ClientBounds this is used in the Game class to effect the graphics device back buffer.
• The display orientation can be seen through the property CurrentOrientation and can also send an event when it is changed with OnOrientationChanged.
• The window can be minimized with an abstract implementation and is accessed through the property IsMinimized.
• The window can be focused with an abstract implementation and is accessed through the property Focused.
• The mouse visibility can be change with the property IsMouseVisible.
• The window platform handle can be seen through the property NativeWindow as an abstract implementation.
• The window visibility can be changed through the property Visible as an abstract implementation.
• The window position can be changed through the property Position.
• The window border can be toggled through the property IsBorderLess as an abstract implementation.
• The window size can be changed with the property PreferredWindowedSize. This also seems to effect the back buffer size.
• The full screen size can be changed with the property PreferredFullscreenSize. This also seems to effect the back buffer size.
• Full screen borderless can be toggled with the property FullscreenIsBorderlessWindow.
• The window full screen can be toggled with the property IsFullscreen.

External Calls Effecting the Window

GraphicsDeviceManager.cs

• Gives information to the Window as to what orientations are supported through the internal abstract method GameWindow.SetSupportedOrientations(DisplayOrientation). Does not actually effect the orientation.
• Notifies the window that the graphics device is changing the back buffer size through Window.BeginScreenDeviceChange(bool) and tells the window if it will be true full screen or not. Maybe used to ensure there are no locks on resources?
• Notifies the window when the graphics device is done changing the size and the resulting size through Window.EndScreenDeviceChange(int, int).
• Also tells the window if the result is true full screen through the internal abstract call Window.SetIsReallyFullscreen(bool).

GamePlatform.cs

• Handles the Run of the window by calling GameWindow.Run() when the GamePlatform is run.
• Notifies the window that it should be exiting by calling GameWindow.Exiting.
• Sends the windows native platform handle to the Graphics Device to render the buffer.
• Resizes the GameWindow when the Graphics Device has been changed using GameWindow.Resize(int, int). This is likely to ensure that the backbuffer and window match or else it is handled improperly and can cause crashes if not called in order.
• Calls GameWindow.Dispose when the game platform has been disposed.

Silk.NET Windowing Changes

TBD I will add more onto this but there needs to be a direct comparison to the Silk.NET windowing to address those possibilities.

From a quick glance the Silk.NET 3.0 windowing proposal covers all that we need apart from:

• Orientation (Which isn't even used in Stride anyway)
• True full screen (Which maybe should be directly in windowing anyway)

Stride Game Management changes

• Remove direct calls to the GameWindow class in the graphicsDeviceManager and make this be handled by the GamePlatform instead.
• Make the GamePlatform the dedicated Window Manager. This way we will need to instantiate a GameContext and GameWindow separately and then join them in the platform instead. This will help avoid the current issue of cyclical dependencies here in the section "API improvement findings and thoughts".
• Make the window style an enum instead of multiple booleans. (Silk.NET windowing does this already if we use that). This will help make things clear as to what state the window is actually in. This is currently a problem with the way the GraphicsDeviceManager uses the method Window.BeginScreenDeviceChange(bool) since it isn't completely clear that SetIsReallyFullscreen(bool) is meant to be used on Graphics Device updates to tell the window it is taking over the display instead of using the window.

I'm sure there are more changes needed and I will need to go through the Silk windowing to gain a better understanding of how we could use the current one and possibly the 3.0 proposal once ready.

[Doprez]

Silk.NET 3.0 to Replace GameWindow

replacements

• GameWindow.ClientBounds -> ISurfaceWindow.ClientArea
• GameWindow.Title -> ISurfaceWindow.Title
• GameWindow.Exiting -> ISurfaceWindow.IsCloseRequested
• GameWindow.Focused -> ISurfaceWindow.IsFocused
• GameWindow.Visible -> ISurfaceWindow.IsVisible
• GameWindow.Position -> ISurfaceWindow.Position (Int2 to Vector2)
• GameWindow.IsBorderless -> ISurfaceWindow.Border (bool to enum)
• GameWindow.AllowUserResizing -> ISurfaceWindow.Border (bool to enum)
• GameWindow.IsMinimized -> ISurfaceWindow.State (bool to enum)
• GameWindow.FullscreenIsBorderlessWindow -> ISurfaceWindow.State (bool to enum)
• GameWindow.PreferredWindowedSize -> ISurfaceWindow.Size (Int2 to Rectangle floats)

No direct replacements

• GameWindow.CurrentOrientation (Not used in Stride anyway AFACIT)
• GameWindow.IsMouseVisible (Should probably be in Input)
• GameWindow.PreferredFullscreenSize (Likely not meant to be part of window but maybe the Silk.NET IDisplay could improve this)
• GameWindow.NativeHandle (Could use the INativeWindow interface)

[Doprez]

Maybe not used by Stride itself. But could the initial intent be that your application/game can check the orientation and react to its changes? Like, for example, rotating elements in the UI according to the orientation the user is holding the device.

I think you're right, orientation could make sense in many cases. I guess even since Windows 8 you could rotate on some devices not only mobile phones/tablets. It just needs to be implemented as it isnt right now.

This may indicate if the mouse cursor is visible when over the window (not globally). Right now it's not very important because there can be only one window. But maybe it would be important if we were to allow more than one window.

Should the window instead have an event to say that the cursor is over an active window in that case and still move the actual hiding of the cursor to Stride.Input?

[Perksey]

GameWindow.PreferredFullscreenSize (Likely not meant to be part of window but maybe the Silk.NET IDisplay could improve this)

I believe the scenarios this property would target (based on the name) is replaced by the IDisplay.VideMode/IDisplay.AvailableVideoModes API.

GameWindow.NativeHandle (Could use the INativeWindow interface)

Yeah the intention is this is what INativeWindow is for, you'd call TryGetPlatformInfo<Win32PlatformInfo> for example to get the HWND.

GameWindow.IsMouseVisible (Should probably be in Input)

This is CursorModes and associated APIs I think (in Silk.NET.Input, see the Multi-Backend Input proposal)

GameWindow.CurrentOrientation (Not used in Stride anyway AFACIT)

This is a good shout.

---

I've added a WindowedFullscreen and renamed Fullscreen to ExclusiveFullscreen

[Ethereal77]

Should the window instead have an event to say that the cursor is over an active window in that case and still move the actual hiding of the cursor to Stride.Input?

Something like the MouseOver event of WinForms? Yeah, that would do it I suppose. Although this being a visual thing that may depend on the each window, I don't know if moving out of GameWindow and into Input may obfuscate it a bit and make it difficult to find.
I mean, now Stride.Input is mainly for "reading input", so some people looking for a way to hide the cursor by first controlling if a specific window (and, if multi-window, not others) are being hovered may be confused, as that is a big departure from the current design (that comes all the way back from XNA).

[Doprez]

It's only weird to me since mouse locking is Input.Mouse.LockPosition() and Input.Mouse.UnlockPosition() but mouse hiding is Game.IsMouseVisible (Which accesses the window variable GameWindow.IsMouseVisible). It would be nice if it were consolidated into one library in my eyes. It does look like Silks input is in the same thought process like Perksey said above.

[Ethereal77]

Yeah, I understand that weirdiness and I agree. Just saying that the current way is there for convenience.

Suppose we (sometime in the future) support more than one window. Then if we somehow wanted to hide the mouse in some of them but not in others, right now is just a boolean, but with the new form, the Mouse.IsVisible will affect all of them (or you would need to go handling MouseOver events and hiding / showing per window), or there would be a method Mouse.SetVisible(Window) or something of that kind.

In summary, what I'm saying is that it makes sense to have Mouse.IsVisible, but Window.MouseVisible is a convenience.

[Doprez]

Game Startup issues

I have been meaning to make a quick layout of Strides current Startup process to give a visual example of the issues with making things modular. Below is a simplified graph of what I have understood, I duplicated nodes for classes to avoid making it unreadable in a graph.

Issue notes:

• Game -> GraphicsDeviceManager (Why do we have another reference for GraphicsDevice in there when we already have it in GameBase?)
• GameBase -> GameWindow (GameWindow is already part of GameContext why have both)
• GameWindow -> GameContext (If GameContext is only meant to send info to GameWindow why do we need a reference both ways)
• GameContext -> GameWindow (Same as above but maybe we leave Window creation to the GamePlatform?)
• GamPlatform -> GameBase (This feels wrong but is used to configure Services and a static method to create the relevant GamePlatform type. We should get the GamePlatform with the creation of the GameBase in the constructor instead I think.)
• GamePlatform -> GameWindow (This may be fine? Might need to be restructured but I think having the GamePlatform managing the window makes sense.)

Extra Issue notes semi unrelated:

• InputManager -> GameContext (This is only used to get the relevant input system for the platform. Its also hard coded so no one can make changes to the windowing without breaking this.)
• InputSystem -> InputManager (I dont know why this reference is needed but feels like InputManager isnt very useful to me.)

@xen2 sorry I keep tagging you in things but I think this may become semi relevant to what you were saying in the discussion about having different packages for windowing? #2493 (reply in thread)

[Doprez]

I agree, I think the main thing I am struggling to work out is specifically the single responsibility while working with the current structure.

From what I understood for the GamePlatform is that it is meant to determine the following things:

• Supported graphics backends (Determined by the project build settings so not used here)
• Graphics device
• The host window
• OS specific features

The thing I am struggling to figure out is how that is any better than having that same functionality managed by the GameBase class.

The other thing I thought of was maybe the purpose of the GamePlatform should purely be used to find the best features for the platform its running on? That way the class becomes much more focused and leaves the rest of the running and management features available through the GameBase class. It also becomes an optional utility rather than a seemingly extra layer with no major benefit.

I think the rest can make sense with those principles in mind:

• GameContext can contain Feature specific information including the currently configured platform settings
• GameSettings stays as is but I could add a list of GameSystems that can be added and removed to allow for the Game Studio to include systems dynamically rather than having it hard coded in the overridden Game class
• GameBase stays as is depending on what to do with the GamePlatform. If GamePlatform changes we may need to move window running to the GameBase Run method rather than leaving it in the GamePlatform Run method which seems unnecessary already.
• GameSystems stays as is, I may need to add a data contract to the class to allow Game Studio to add it to the GameSettings but it already takes in the IServiceProvider in its constructor(will need to see what can be done with the serializer since it needs an empty contructor).
• GameWindow stays as is, ignoring the possible change with Silk.Windowing I think this class only changes in the sense of what should reference and manage it. Either the GamePlatform or the GameBase.
• InputManager will have the GameContext reference removed and will instead be configured through Game. This solves a separate issue where the input had to be configured through Stride source rather then being allowed to add your own at runtime.

I think that covers most of the issues? At least with removing seemingly random references being everywhere and having a central point of truth. The last thing I need to look into is the cyclic reference between GameWindow and GameContext but I think with the above things getting figured out that will become easier to what should be managing the window.

[ericwj]

I can't comment very specifically, since I haven't had the code cloned for months and I was never intimately known with this part of Stride. But I can say a few things.

• It could well be that GamePlatform fulfills the role of a very specific, limited integrator.
  That is why it might be combining these things and have references to other things which themselves have a single responsibility.
  Once you start doing something useful or hard, like rendering, you might need to integrate multiple things. There is a tradeoff between utility and SRP in such specific circumstances.

• How GamePlatform relates to GameBase, without looking in the code, is probably that GamePlatform isn't "app code" and GameBase is meant to be the primary extension point for writing "app code". An "app", meaning what you can create deriving from GameBase, is just the extreme form of "integrator". A big pile of stuff, often mutually unrelated stuff, composed together to provide specific features and behavior. Usually not at all reusable.

• Notice I didn't say "executable", since it isn't required to be - Stride can be a component still in a larger app. Ideally possibly more than one such Stride. And I didn't say "window", or "surface" because even that is optional. Stride could make video's, or just images, as a server without UI, if it was really code only. Lets say you crank up detail, lighting, insanely, and do ray tracing, for the in-game video, which you ship as .mp4. Or replays; no UI, but 10 camera's, keyboards and mice, one for each player.
  So you don't have to premeditate that, but people can dream this sort of thing up if only Stride is flexible enough, through sound application of software engineering principles.

• So yes, windowing should be a Stride thing, not a Silk thing. Definitely.

• Also useful is to look at some very simple examples lets say in the DirectX Graphics Samples repository. Even if Stride is much more than a simple sample, the underlying basic structure of a rendering application should be there and possibly be available for replacement. Most samples have the basic structure:

  1. Create device dependent resources
  2. Create size dependent resources
  3. Listen for device lost to redo step 1
  4. Listen for resize to redo step 2
  5. Update/Render loop

  Which means the primitives absolutely required to start Stride should not be much more than that. I don't mean what you get when you make a game in the editor, but I mean what you need when you would create those objects relevant here yourself in say a console app. Maybe you can scrutinize what you have found in that light?

• This is one of the worst things ever happening to Stride imho. The #if and configuring the backend through MSBuild.

  • Supported graphics backends (Determined by the project build settings so not used here)

  The wanted state is that you can ship multiple backends and construct the desired one at runtime (through DI). Would you see opportunity to make this part of Stride more suitable to that end, I think that would be awesome.

[Ethereal77]

@ericwj I'm not saying XNA was design by the brightest minds and we should follow it to the letter. I'm saying XNA inspired SharpDX Toolkit, which then inspired Stride (Paradox back then), and that is a pretty flexible architecture for code-only (as that was the intent of XNA).

Also, as we currently have code-only users in the community and people who don't even like to launch Game Studio, let's not forget that. People even have started to use Stride for this because of lack of flexibility in other engines.

Right now you can say Stride is composed of several very different parts:

1. A game framework, the part that was inspired by XNA, etc. This includes things like Stride.Core, Stride.Core.Mathematics, or Stride.Graphics.
2. A game engine, the part that using the game framework compounds a game with its rendering pipeline, interaction, audio, streaming, etc.
3. A game content pipeline for consuming and compiling content, including also an editor (Game Studio).

I'm all in for improving and simplifying Stride where needed, but please don't destroy the core principles that make it so powerful and flexible at the same time. Just my opinion, of course.

[Doprez]

As someone who primarily uses the code only library and only opens Game Studio for scene editing. That was the main reasons I even decided to look into this in the first place.

The issues I have dealt with for using code only are:

• locked down APIs that didnt allow for custom window implementations.
• internal APIs that seem incomplete (probably why they're internal).
• internal calls in the Game class that prevented manual fixes to bugs.
• The content pipeline is painful for code only without a ton of added code Content Loading at runtime stride-community-toolkit#176

Definitely not my goal to change any of the core functions that make Stride flexible. Ideally nothing should change to the average users of Stride in terms of functionality or current access. The majority of this work is more about making things even more flexible by decoupling things that aren't needed by everyone like the GameWindow for server run games or Systems that are not configurable without rewriting the entire Game class which you can't even do because of internal classes.

The changes I want to make are for people who want:

• more customizable game startups
• to make their own UI framework implementations
• options to add systems easily at start

Most of the changes I have in mind are all within the Stride.Games library and should touch nothing else except to update any GameSystems, EntityProcessors, EntityComponents, etc to use the new references and even most of those are using the IServiceRegistry so a lot should already be OK.

[ericwj]

@Ethereal77 sure, I am first of all trying to make clear what the core principles even are, or how they came about. What I said about refactoring was towards cleanup of core principles where they might have become violated and improving flexibility, not at all for destruction.

I haven't commented on concrete things in Stride, or compared with XNA, because I don't have the knowledge - that's more you, reading your comments. Very valuable. We come at it from entirely different angles with the goal of improving understanding. Mine being almost entirely abstract.

Other than expecting such a Microsoft project to be meticulously designed and reviewed by at least very decent architects, I have no opinion on the quality of XNA, or its problems. But I like what you say - that one of its core design principles was code-first. However that works though, the other parts of Stride will have to remain able to deal with it. I don't think that is much of a restriction though.