~eliasnaur/gio

4 3

Injecting custom frame update to the flow?

Details
Message ID
<CAJJC9nfJDtirW3S8CA1FqtPYbPRSfem522dk2VNyJq1n0A-uaA@mail.gmail.com>
DKIM signature
pass
Download raw message
Hi guys,

I am trying to use gio to build a multi window chat client to have
main chat UI in one
window and run OpenGL ES game in another window which I hope to use Gio
Window as well as the event loop and frame updating mechanism.

The game has its own "drawFrame" method which calls standard OpenGL ES APIs.
And does not use gio widgets yet (might need to add some later).
I am wondering how I can inject the game's drawFrame into Gio window's frame
update loop.

I spent some time digging into the code, it looks to me current frame
drawing happens
in gio/gpu/GPU.Frame() method. So seems not possible without API changes.
Is this true? If so, any advice on how to do this?

Thanks in advance for your help.

- Zhao Wang
Details
Message ID
<lfvkIdqV4a4fWLp-Lg--3XjSG7XOCAavFxM3bZjZNx8F9Hf2ECoolx5XIPuroajiY16vk1ywmooXyCvGgZ1cwccBndXzAyX8P-loRqbTr5U=@protonmail.ch>
In-Reply-To
<CAJJC9nfJDtirW3S8CA1FqtPYbPRSfem522dk2VNyJq1n0A-uaA@mail.gmail.com> (view parent)
DKIM signature
pass
Download raw message
It should be possible to take the pixels the other code is generating and put them
in an image op and update the window, does the other GL library have a headless
functionality? I assume the game environment captures events of interest also to
the proposed Gio GUI you want to frame/overlay on it.

If you can avoid the two things drawing into the same box there might be some way to
make the box for the GL paint directly as it normally does and Gio just render nothing
in the same place. There might be synchronisation issues with this also, but it probably
would only make some screen garbage/flicker etc. idk. just thinking out loud.

I have done a little work with headless rendering. I was initially going to go for a
more traditional offscreen private bitmap, using the headless context, to make a scrolling view,
but found I was able to skirt around doing so much blitting by running the widgets to
generate dimensions, as with dimensions computing how to render the view. That's how my scrollbar
widget works.

It's the best scrollbar implementation to date for Gio, maybe looking at what I did will
give you some ideas? https://github.com/p9c/pod/blob/master/pkg/gui/list.go

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Sunday, January 17th, 2021 at 08:33, Zhao Wang <sunny.open@gmail.com> wrote:

> Hi guys,
>
> I am trying to use gio to build a multi window chat client to have
>
> main chat UI in one
>
> window and run OpenGL ES game in another window which I hope to use Gio
>
> Window as well as the event loop and frame updating mechanism.
>
> The game has its own "drawFrame" method which calls standard OpenGL ES APIs.
>
> And does not use gio widgets yet (might need to add some later).
>
> I am wondering how I can inject the game's drawFrame into Gio window's frame
>
> update loop.
>
> I spent some time digging into the code, it looks to me current frame
>
> drawing happens
>
> in gio/gpu/GPU.Frame() method. So seems not possible without API changes.
>
> Is this true? If so, any advice on how to do this?
>
> Thanks in advance for your help.
>
> -   Zhao Wang
Details
Message ID
<C8LBTLBQW1K8.3V5RLMYHVKR3F@testmac>
In-Reply-To
<CAJJC9nfJDtirW3S8CA1FqtPYbPRSfem522dk2VNyJq1n0A-uaA@mail.gmail.com> (view parent)
DKIM signature
fail
Download raw message
DKIM signature: fail
On Sun Jan 17, 2021 at 8:33 AM CET, Zhao Wang wrote:
> Hi guys,
>
> I am trying to use gio to build a multi window chat client to have
> main chat UI in one
> window and run OpenGL ES game in another window which I hope to use Gio
> Window as well as the event loop and frame updating mechanism.
>
> The game has its own "drawFrame" method which calls standard OpenGL ES
> APIs.
> And does not use gio widgets yet (might need to add some later).
> I am wondering how I can inject the game's drawFrame into Gio window's
> frame
> update loop.
>
> I spent some time digging into the code, it looks to me current frame
> drawing happens
> in gio/gpu/GPU.Frame() method. So seems not possible without API
> changes.
> Is this true? If so, any advice on how to do this?
>

Is the game's OpenGL ES commands supposed to happen before, after, or in
between Gio drawing commands? If before or after, I believe you can use
the glfw example as inspiration. Specifically, use GLFW for the native
window and OpenGL context, and let both the game and Gio use the same
context. In the glfw example you can also see how the main program
forwards mouse/keyboard input to Gio. That's where you can route input
to the game as well.

A more portable variant is to use an OpenGL context for the game and a
separate context for Gio. If you bind a FBO with a texture target before
calling Gio's Frame, you can use the resulting texture to draw on top of
the game. This works because you can set up contexts so they share
resources (textures etc.) without sharing state.

If you haven't already considered it, I'd like to mention
https://github.com/ocornut/imgui. Imgui is immediate mode like Gio,
designed to be embedded, and used by many game studios for their GUI.

These are all high-level ideas. You'll probably hit various issues on
your way, all of which we're happy to help resolve. Portability is an
important Gio feature, and your embedding use case is an important kind
of portability.

Elias
Details
Message ID
<CAJJC9ne_iJrDHfJhnPyn54kPawYcxCMTZ+2JJimAgOTCx97ViQ@mail.gmail.com>
In-Reply-To
<lfvkIdqV4a4fWLp-Lg--3XjSG7XOCAavFxM3bZjZNx8F9Hf2ECoolx5XIPuroajiY16vk1ywmooXyCvGgZ1cwccBndXzAyX8P-loRqbTr5U=@protonmail.ch> (view parent)
DKIM signature
pass
Download raw message
Hi Loki,

Thanks for the help. It is helpful, I will try the approaches
you mentioned below.

Besides my response below, here is our current app demo,
just in case it helps to understand what I am talking about.

https://drive.google.com/file/d/1HTJ5L8l4MoRz5llK_WhR7ObyKMtvJK6X/view

The gif above shows both chat view (native iOS UIKit view)
and game view (custom iOS UIView with a EAGLContext
with DisplayLink)


On Sun, Jan 17, 2021 at 12:50 AM Loki Verloren
<stalker.loki@protonmail.ch> wrote:
>
> It should be possible to take the pixels the other code is generating and put them
> in an image op and update the window, does the other GL library have a headless
> functionality? I assume the game environment captures events of interest also to
> the proposed Gio GUI you want to frame/overlay on it.

Yeah, the other GL library we have is headless, it expect the native
view/window to provide the context and pass input events to game. Game
libraries will handle input events.

> If you can avoid the two things drawing into the same box there might be some way to
> make the box for the GL paint directly as it normally does and Gio just render nothing
> in the same place. There might be synchronisation issues with this also, but it probably
> would only make some screen garbage/flicker etc. idk. just thinking out loud.

Yeah, I was thinking to use GIO's window and context to draw game
stuff. Gio widget can be drawn at top of game drawings. See game demo
above, in game view, the back button and menu button are native iOS
UIKit buttons and the chat view are UIKit view, they are not draw by
game libraries.

> I have done a little work with headless rendering. I was initially going to go for a
> more traditional offscreen private bitmap, using the headless context, to make a scrolling view,
> but found I was able to skirt around doing so much blitting by running the widgets to
> generate dimensions, as with dimensions computing how to render the view. That's how my scrollbar
> widget works.
>
> It's the best scrollbar implementation to date for Gio, maybe looking at what I did will
> give you some ideas? https://github.com/p9c/pod/blob/master/pkg/gui/list.go
>
> Sent with ProtonMail Secure Email.
>

This sounds great, if I can make game view as a widget, it might solve
the problem well. I will definitely check this out and see if I can
figure out this way.
Details
Message ID
<CAJJC9ndRxe0erMUf+vfZA4oxsBRnjrup+eQOKBpkm9GNCx8X0w@mail.gmail.com>
In-Reply-To
<C8LBTLBQW1K8.3V5RLMYHVKR3F@testmac> (view parent)
DKIM signature
pass
Download raw message
Hi Elias,

Please see my response below. Thank you!

On Sun, Jan 17, 2021 at 1:30 AM Elias Naur <mail@eliasnaur.com> wrote:
>
> On Sun Jan 17, 2021 at 8:33 AM CET, Zhao Wang wrote:
> > Hi guys,
> >
> > I am trying to use gio to build a multi window chat client to have
> > main chat UI in one
> > window and run OpenGL ES game in another window which I hope to use Gio
> > Window as well as the event loop and frame updating mechanism.
> >
> > The game has its own "drawFrame" method which calls standard OpenGL ES
> > APIs.
> > And does not use gio widgets yet (might need to add some later).
> > I am wondering how I can inject the game's drawFrame into Gio window's
> > frame
> > update loop.
> >
> > I spent some time digging into the code, it looks to me current frame
> > drawing happens
> > in gio/gpu/GPU.Frame() method. So seems not possible without API
> > changes.
> > Is this true? If so, any advice on how to do this?
> >
>
> Is the game's OpenGL ES commands supposed to happen before, after, or in
> between Gio drawing commands? If before or after, I believe you can use
> the glfw example as inspiration. Specifically, use GLFW for the native
> window and OpenGL context, and let both the game and Gio use the same
> context. In the glfw example you can also see how the main program
> forwards mouse/keyboard input to Gio. That's where you can route input
> to the game as well.

I think the GL ES commands are supposed to happen in between Gio
drawing commands. They run to generate every frame just like Gio's
Frame method. The glfw example can be what I need for game view alone,
but I am not sure how to make it work together with the main Gio
window (for chat interfaces) . Also I might need multiple glfw+gio
windows at the same time (besides the main gio window for chat
clients) as we should support users playing multiple games at the same
time in desktop clients. Please see this gif
https://drive.google.com/file/d/1HTJ5L8l4MoRz5llK_WhR7ObyKMtvJK6X/view
for how our iOS app works between chat view and game view.

> A more portable variant is to use an OpenGL context for the game and a
> separate context for Gio. If you bind a FBO with a texture target before
> calling Gio's Frame, you can use the resulting texture to draw on top of
> the game. This works because you can set up contexts so they share
> resources (textures etc.) without sharing state.

Yeah, this is exactly what I was looking for. I want to have multiple
windows, one Gio only window using all Gio widgets and components.
Then another Gio window with a different context to draw game contents
and some limited gio widgets on top of the game view. However, right
now I am not sure where to put my custom drawing commands with the
current Gio API, it seems I need to pass Ops to draw, which means I
then can not use our custom libs.

> If you haven't already considered it, I'd like to mention
> https://github.com/ocornut/imgui. Imgui is immediate mode like Gio,
> designed to be embedded, and used by many game studios for their GUI.

Will definitely check out this. By any chance, do you know if the go
wrapper for IMGui is good to try?
https://github.com/inkyblackness/imgui-go

> These are all high-level ideas. You'll probably hit various issues on
> your way, all of which we're happy to help resolve. Portability is an
> important Gio feature, and your embedding use case is an important kind
> of portability.

Thank you for all the ideas, definitely very helpful. I will keep
trying and updating back when I have progress. I really hope Gio can
support my case if not yet.

> Elias
Reply to thread Export thread (mbox)