~eliasnaur/gio

17 3

Gio and accessibility

Matt Campbell
Details
Message ID
<6e41cbe8-0d59-917d-337e-3eb3debecda4@pobox.com>
DKIM signature
missing
Download raw message
Hello:

Are there any plans to add accessibility for users with disabilities 
(e.g. blind users who need to use a screen reader, or users who need 
alternative methods of input) to Gio? I imagine this would be a 
challenge for an immediate-mode toolkit, since all of the platform 
accessibility APIs assume a persistent tree of UI elements. Also, it 
looks to me like there are some basic widgets, like buttons, that aren't 
defined by Gio itself, but have to be defined by each application. I 
hope you've considered these things, because Scatter in particular looks 
like a useful application, and it would be unfortunate if some of us 
were unable to use it because of the way the GUI was implemented.

Thanks,

Matt
Details
Message ID
<BXM57J1B6AH4.3SRC01XON5SQW@toolbox>
In-Reply-To
<6e41cbe8-0d59-917d-337e-3eb3debecda4@pobox.com> (view parent)
DKIM signature
missing
Download raw message
On Thu Oct 10, 2019 at 8:55 AM Matt Campbell wrote:
> Are there any plans to add accessibility for users with disabilities 
> (e.g. blind users who need to use a screen reader, or users who need 
> alternative methods of input) to Gio?

I'm glad you asked. Two big features included in any general purpose toolkit
not yet covered by Gio are accessibility and right-to-left layouts.

I have very little experience in either field, but it's my intention that Gio
1.0 will address both.

> I imagine this would be a 
> challenge for an immediate-mode toolkit, since all of the platform 
> accessibility APIs assume a persistent tree of UI elements.

You still have persistent UI state in immediate mode programs; state is just
explicit rather than hidden inside the toolkit.

One approach to acessibility is to add special accessibility operations and
emit them during widget layouts. For example, an access.DescriptionOp could
serve the same purpose as the "alt" tag from HTML.

At some point, Gio will have to tackle keyboard navigation between input
fields. That functionality should prove useful for screen readers.

A more crazy idea is to make the accessibility metadata so detailed that a HTML
rendering of a Gio UI could be generated automatically. That would be useful in
itself, and thus motivate programmers to ensure their accessibility metadata is
complete.

> Also, it 
> looks to me like there are some basic widgets, like buttons, that aren't 
> defined by Gio itself, but have to be defined by each application.

That's just because I wasn't yet happy with the API for the widgets :)

In fact, I've been working on a theme and widget design that I'm
days away from releasing. When it's out I'll port the example programs
as well as Scatter.

-- elias
Matt Campbell
Details
Message ID
<f95d551b-6a65-1392-8eb6-4cdcf341b47e@pobox.com>
In-Reply-To
<BXM57J1B6AH4.3SRC01XON5SQW@toolbox> (view parent)
DKIM signature
missing
Download raw message
I'm happy to hear that you're planning to work on accessibility. I think 
making your cross-platform accessibility support rich enough to enable 
HTML output is a good way to start. That will also help with the 
WebAssembly target.

When you're eventually ready to implement platform-specific 
accessibility APIs, I can help with the Windows UI Automation API. I 
work at Microsoft, on the Windows accessibility team, but I would be 
doing this on my own time.

Matt
Details
Message ID
<BXMIS8FJK4V3.16B76W2URRAOP@toolbox>
In-Reply-To
<f95d551b-6a65-1392-8eb6-4cdcf341b47e@pobox.com> (view parent)
DKIM signature
missing
Download raw message
On Thu Oct 10, 2019 at 8:29 PM Matt Campbell wrote:
> When you're eventually ready to implement platform-specific 
> accessibility APIs, I can help with the Windows UI Automation API. I 
> work at Microsoft, on the Windows accessibility team, but I would be 
> doing this on my own time.
> 

Great, thanks!

-- elias
Details
Message ID
<02a8a098-d3fd-76f4-3f17-0adf87865fcc@pobox.com>
In-Reply-To
<BXM57J1B6AH4.3SRC01XON5SQW@toolbox> (view parent)
DKIM signature
pass
Download raw message
Hi Elias,

I wonder if you have any plans to implement accessibility in Gio in the 
near future. I'll be happy to help, especially with the Windows 
implementation (BTW, I'm no longer at Microsoft). But this will require 
some design work, which you'll obviously need to at least approve. I 
like your earlier idea of emitting accessibility-specific operations 
during widget layout.

Gio is now being used in a couple of Android applications that I'd like 
to be able to use (Tailscale and Wormhole William). And I depend on the 
TalkBack screen reader when using my Android phone. So I now have a more 
concrete interest in seeing this gap filled.

Thanks,
Matt
Details
Message ID
<CAFcc3FTM4zAMinos8WMSAqGHy7KNursKGZFuvtdQvJ+iZsZ-Mw@mail.gmail.com>
In-Reply-To
<02a8a098-d3fd-76f4-3f17-0adf87865fcc@pobox.com> (view parent)
DKIM signature
pass
Download raw message
Hi Matt!

> Hi Elias,

I'm (obviously) not Elias, but...

> I wonder if you have any plans to implement accessibility in Gio in the
> near future. I'll be happy to help, especially with the Windows
> implementation (BTW, I'm no longer at Microsoft). But this will require
> some design work, which you'll obviously need to at least approve. I
> like your earlier idea of emitting accessibility-specific operations
> during widget layout.

We have spoken about this relatively recently (I think during the
office hours after a community call). I believe that the conclusion
was that it's a significant priority, probably right behind rolling
out the compute rendering backend as the default renderer (Elias,
please correct me if I'm wrong). We're all very eager to properly
support assistive technologies, and I know that I personally would
help as time allows (but I'm not currently in a position to spearhead
that effort myself).

> Gio is now being used in a couple of Android applications that I'd like
> to be able to use (Tailscale and Wormhole William). And I depend on the
> TalkBack screen reader when using my Android phone. So I now have a more
> concrete interest in seeing this gap filled.

Well I'm glad that some Gio-based apps are out there being useful for
you! Let's definitely fix this.

Cheers,
Chris
Details
Message ID
<CC39BIG7N6XK.TB0XIQYESTW2@testmac>
In-Reply-To
<CAFcc3FTM4zAMinos8WMSAqGHy7KNursKGZFuvtdQvJ+iZsZ-Mw@mail.gmail.com> (view parent)
DKIM signature
pass
Download raw message
On Sun Jun 13, 2021 at 20:23 CEST, Chris Waldon wrote:
> Hi Matt!
>
> > Hi Elias,
>
> I'm (obviously) not Elias, but...
>
> > I wonder if you have any plans to implement accessibility in Gio in the
> > near future. I'll be happy to help, especially with the Windows
> > implementation (BTW, I'm no longer at Microsoft). But this will require
> > some design work, which you'll obviously need to at least approve. I
> > like your earlier idea of emitting accessibility-specific operations
> > during widget layout.
>
> We have spoken about this relatively recently (I think during the
> office hours after a community call). I believe that the conclusion
> was that it's a significant priority, probably right behind rolling
> out the compute rendering backend as the default renderer (Elias,
> please correct me if I'm wrong). We're all very eager to properly
> support assistive technologies, and I know that I personally would
> help as time allows (but I'm not currently in a position to spearhead
> that effort myself).
>

Yes, the motivating issue is Tailscale's #1004[0]. That's for Android,
but I'm sure that most of the work will be figuring out how to add an
accessibility framework to Gio. I'm no expert here, so your input for
design is valuable, Matt, and once we have a basic design a Windows
implementation would be great to ensure that we haven't sneaked in any
platform idiosyncrasies.

Elias

[0] https://github.com/tailscale/tailscale/issues/1004
Details
Message ID
<6f50a6a7-8749-08cb-e405-e11616496be4@pobox.com>
In-Reply-To
<CC39BIG7N6XK.TB0XIQYESTW2@testmac> (view parent)
DKIM signature
pass
Download raw message
Elias, you mentioned the issue with Tailscale for Android. I mistakenly 
opened a duplicate of that issue yesterday, and on my issue, someone 
mentioned that they were planning a UI rewrite, likely using native 
widgets [1]. Do you know anything about this? Of course, accessibility 
in Gio is important for more than just Tailscale.

Would you rather design Gio's accessibility support yourself and have me 
review it, or vice versa?

Thanks,
Matt

[1]: 
https://github.com/tailscale/tailscale/issues/2118#issuecomment-860240140
Details
Message ID
<CC3CVPFKA014.2JYYUM0MYS7I6@testmac>
In-Reply-To
<6f50a6a7-8749-08cb-e405-e11616496be4@pobox.com> (view parent)
DKIM signature
pass
Download raw message
On Mon Jun 14, 2021 at 14:08 CEST, Matt Campbell wrote:
> Elias, you mentioned the issue with Tailscale for Android. I mistakenly 
> opened a duplicate of that issue yesterday, and on my issue, someone 
> mentioned that they were planning a UI rewrite, likely using native 
> widgets [1]. Do you know anything about this? Of course, accessibility 
> in Gio is important for more than just Tailscale.
>

This is the first I've heard of it :) I recently resigned my role as
maintainer of the Tailscale Android app, I suppose they chose to rewrite
it rather than keep using Gio.

> Would you rather design Gio's accessibility support yourself and have me 
> review it, or vice versa?
>

I think you have a better chance arriving at a design that fits the platforms
APIs for acccessibility. I'm happy to review what you come up with.

Elias
Details
Message ID
<6170ac0b-d883-5965-5c37-ac8bb4dba6df@pobox.com>
In-Reply-To
<CC3CVPFKA014.2JYYUM0MYS7I6@testmac> (view parent)
DKIM signature
pass
Download raw message
On 6/14/2021 7:56 AM, Elias Naur wrote:

> I think you have a better chance arriving at a design that fits the 
> platforms
> APIs for acccessibility. I'm happy to review what you come up with.
Agreed.

One problem we're going to encounter with implementing platform 
accessibility APIs is that it will require much more cross-language glue 
than you have had to implement so far. The Windows UI Automation API 
requires you to not merely consume COM interfaces, but implement them. 
Likewise for Cocoa with Objective-C, and Android with Java. I'm guessing 
that doing all of this in a Go project would be quite tedious.

Also, I'm all too aware that there are many UI toolkits out there that 
need work on accessibility, not just Gio. So, if possible, I want to 
focus my efforts on work that will benefit multiple toolkits.

So I've been thinking about starting my own open-source project to make 
it easier for toolkits like Gio to implement accessibility. The working 
name for my project is AccessKit, though I haven't committed to that 
yet. At a high level, AccessKit would consist of the following components:

 1. A message schema for defining UI semantics (a.k.a. an accessibility
    tree), probably using Protocol Buffers.
 2. Libraries that consume messages in this schema and implement the
    various platform accessibility APIs. Each of these libraries would
    be written in the platform's native language, e.g. C++ for Windows,
    Objective-C or Swift for Apple platforms, Java for Android, and
    JavaScript for web.
 3. Documentation on how to provide semantics for a UI. Not just
    reference documentation for the schema, but conceptual documentation
    for developers that aren't already experts in accessibility.

I think this approach would be a good fit for Gio. As I understand it, 
you're already serializing your operations to a byte buffer. My only 
question is whether you'd be willing to take a dependency on both 
Protobuf and my platform-specific libraries. Protobuf is BSD-licensed, 
and I'm planning to use the same license for all code in AccessKit. So I 
guess the primary concern is that my non-Go libraries would complicate 
the build process for Gio users.

What do you think?

Matt
Details
Message ID
<CC3F7JJ3BMGG.2AK2ITRF3U80C@testmac>
In-Reply-To
<6170ac0b-d883-5965-5c37-ac8bb4dba6df@pobox.com> (view parent)
DKIM signature
pass
Download raw message
On Mon Jun 14, 2021 at 16:15 CEST, Matt Campbell wrote:
> On 6/14/2021 7:56 AM, Elias Naur wrote:
>
> > I think you have a better chance arriving at a design that fits the 
> > platforms
> > APIs for acccessibility. I'm happy to review what you come up with.
> Agreed.
>
> One problem we're going to encounter with implementing platform 
> accessibility APIs is that it will require much more cross-language glue 
> than you have had to implement so far. The Windows UI Automation API 
> requires you to not merely consume COM interfaces, but implement them. 
> Likewise for Cocoa with Objective-C, and Android with Java. I'm guessing 
> that doing all of this in a Go project would be quite tedious.
>
> Also, I'm all too aware that there are many UI toolkits out there that 
> need work on accessibility, not just Gio. So, if possible, I want to 
> focus my efforts on work that will benefit multiple toolkits.
>
> So I've been thinking about starting my own open-source project to make 
> it easier for toolkits like Gio to implement accessibility. The working 
> name for my project is AccessKit, though I haven't committed to that 
> yet. At a high level, AccessKit would consist of the following components:
>
>  1. A message schema for defining UI semantics (a.k.a. an accessibility
>     tree), probably using Protocol Buffers.
>  2. Libraries that consume messages in this schema and implement the
>     various platform accessibility APIs. Each of these libraries would
>     be written in the platform's native language, e.g. C++ for Windows,
>     Objective-C or Swift for Apple platforms, Java for Android, and
>     JavaScript for web.
>  3. Documentation on how to provide semantics for a UI. Not just
>     reference documentation for the schema, but conceptual documentation
>     for developers that aren't already experts in accessibility.
>
> I think this approach would be a good fit for Gio. As I understand it, 
> you're already serializing your operations to a byte buffer. My only 
> question is whether you'd be willing to take a dependency on both 
> Protobuf and my platform-specific libraries. Protobuf is BSD-licensed, 
> and I'm planning to use the same license for all code in AccessKit. So I 
> guess the primary concern is that my non-Go libraries would complicate 
> the build process for Gio users.
>
> What do you think?
>

Well, beggars can't be choosers :) More seriously, I think your plan
sounds great, and support for multiple toolkits will make your project
better and spread the maintenance burden. In fact, the issues in
accessibility support sound similar to what we face to implement support
for complex text scripts. We have the #ui-text-rendering Gopher Slack
channel and the beginning of a shared library at
https://github.com/go-text/shaping. I encourage you to initiate
something similar for accessibility and bring in the Fyne people.

Elias
Details
Message ID
<6d7f7675-7055-14d0-bc22-fce8bf6bfa7b@pobox.com>
In-Reply-To
<CC3F7JJ3BMGG.2AK2ITRF3U80C@testmac> (view parent)
DKIM signature
pass
Download raw message
On 6/14/2021 9:45 AM, Elias Naur wrote:

> Well, beggars can't be choosers :) More seriously, I think your plan
> sounds great, and support for multiple toolkits will make your project
> better and spread the maintenance burden. In fact, the issues in
> accessibility support sound similar to what we face to implement support
> for complex text scripts. We have the #ui-text-rendering Gopher Slack
> channel and the beginning of a shared library at
> https://github.com/go-text/shaping. I encourage you to initiate
> something similar for accessibility and bring in the Fyne people.
Thanks. To be clear though, I want my AccessKit project to be 
language-independent, not just for Go. That's why I propose using a 
Protobuf schema as the glue between the cross-platform toolkit code (Gio 
in this case) and the platform-specific accessibility implementations, 
which I'll call platform adapters. Also, those platform adapters would 
not be written in Go. I suppose that's not a problem for Mac, iOS, 
Linux, or Android, since Gio's support for those platforms isn't pure Go 
anyway. But I guess that's more of a problem for Windows. I could ship 
my Windows platform adapter library as a DLL with a C ABI, and you could 
call that easily enough from Go. Would it be a problem if developers had 
to ship an extra DLL with their application on Windows?

Matt
Details
Message ID
<CC3FW2QJSINK.X5JRLW1FARLS@testmac>
In-Reply-To
<6d7f7675-7055-14d0-bc22-fce8bf6bfa7b@pobox.com> (view parent)
DKIM signature
pass
Download raw message
On Mon Jun 14, 2021 at 17:06 CEST, Matt Campbell wrote:
> On 6/14/2021 9:45 AM, Elias Naur wrote:
>
> > Well, beggars can't be choosers :) More seriously, I think your plan
> > sounds great, and support for multiple toolkits will make your project
> > better and spread the maintenance burden. In fact, the issues in
> > accessibility support sound similar to what we face to implement support
> > for complex text scripts. We have the #ui-text-rendering Gopher Slack
> > channel and the beginning of a shared library at
> > https://github.com/go-text/shaping. I encourage you to initiate
> > something similar for accessibility and bring in the Fyne people.
> Thanks. To be clear though, I want my AccessKit project to be 
> language-independent, not just for Go. That's why I propose using a 
> Protobuf schema as the glue between the cross-platform toolkit code (Gio 
> in this case) and the platform-specific accessibility implementations, 
> which I'll call platform adapters. Also, those platform adapters would 
> not be written in Go. I suppose that's not a problem for Mac, iOS, 
> Linux, or Android, since Gio's support for those platforms isn't pure Go 
> anyway. But I guess that's more of a problem for Windows. I could ship 
> my Windows platform adapter library as a DLL with a C ABI, and you could 
> call that easily enough from Go. Would it be a problem if developers had 
> to ship an extra DLL with their application on Windows?
>

Not an insurmountable problem. If possible, you may have success in
compiling AccessKit to an object file and embed it as a self-contained
.syso file in which case you can avoid the extra DLL.

At some point I plan to work on avoiding Cgo at build time[2], but
that's pie-in-the-sky and just a FYI for now.

Elias

[0] https://github.com/golang/go/wiki/GcToolchainTricks
[1] https://github.com/golang/go/issues/38917
Details
Message ID
<d4ac5beb-e4c5-5e28-91f1-bb4ef2bc8da7@pobox.com>
In-Reply-To
<CC3FW2QJSINK.X5JRLW1FARLS@testmac> (view parent)
DKIM signature
pass
Download raw message
One more question for now: The Gio architecture document briefly 
mentions that Gio has a "zero allocation" design. How stringent is this 
requirement? Have you written more about it somewhere else?

My AccessKit project is still in the early design phase, so I want to 
make sure that it will be usable for Gio and other similar toolkits.

Thanks,
Matt
Details
Message ID
<CC3HF8JFRTI0.2V2ILU1E8V9LA@testmac>
In-Reply-To
<d4ac5beb-e4c5-5e28-91f1-bb4ef2bc8da7@pobox.com> (view parent)
DKIM signature
pass
Download raw message
On Mon Jun 14, 2021 at 18:14 CEST, Matt Campbell wrote:
> One more question for now: The Gio architecture document briefly 
> mentions that Gio has a "zero allocation" design. How stringent is this 
> requirement? Have you written more about it somewhere else?
>

Zero allocation design refers to the performance-critical FrameEvent
processing, which needs to comfortably hit 60+ Hz without garbage
collector hitches. That's the reason ops (op.InvalidateOp,
paint.PaintOp etc.) are encoded to a byte slice when added to gtx.Ops.

Allocating in AccessKit or in the Gio glue is fine as long as it's
amortized and not per frame.

> My AccessKit project is still in the early design phase, so I want to 
> make sure that it will be usable for Gio and other similar toolkits.
>

If AccessKit is implemented as ops (say, access.LabelOp for textual
representation) which can be encoded to a byte stream, I think we'll be
fine. We can also add an accessibility context to Gio programs to enable
caching of expensive computations, if needed. That's what text shaping
does with gioui.org/text.Cache, hidden behind the gioui.org/text.Shaper
interface.

Elias
Details
Message ID
<00695dbd-592c-ee74-a4c3-90882925e8f2@pobox.com>
In-Reply-To
<CC3FW2QJSINK.X5JRLW1FARLS@testmac> (view parent)
DKIM signature
pass
Download raw message
On 6/14/2021 10:17 AM, Elias Naur wrote:

> Not an insurmountable problem. If possible, you may have success in
> compiling AccessKit to an object file and embed it as a self-contained
> .syso file in which case you can avoid the extra DLL.
Would this require the AccessKit Windows library to be compilable with a 
GNU toolchain (meaning mingw-w64)? Or can the Go linker handle .obj 
files produced by MSVC?

Also, I see that Gio's CI uses Wine to run Windows tests on Linux. For 
what it's worth, Wine's implementation of UI Automation, the current 
Windows accessibility API, is only a stub. And it's implementation of 
Active Accessibility, the legacy accessibility API, is also very 
incomplete. So this means that the AccessKit Windows library can't make 
any accessibility API calls unless an assistive technology is actually 
running, or if it does have to proactively make any API calls, it should 
fail gracefully. That shouldn't be a problem. It also means that 
end-to-end accessibility testing won't be possible on Gio's CI. That 
should also be OK, since AccessKit will have its own CI (probably GitHub 
Actions), which will have access to the actual proprietary platforms.

Matt
Details
Message ID
<CC47ZD3W6FCO.2S4W112K2V0MU@testmac>
In-Reply-To
<00695dbd-592c-ee74-a4c3-90882925e8f2@pobox.com> (view parent)
DKIM signature
pass
Download raw message
On Tue Jun 15, 2021 at 15:15 CEST, Matt Campbell wrote:
> On 6/14/2021 10:17 AM, Elias Naur wrote:
>
> > Not an insurmountable problem. If possible, you may have success in
> > compiling AccessKit to an object file and embed it as a self-contained
> > .syso file in which case you can avoid the extra DLL.
> Would this require the AccessKit Windows library to be compilable with a 
> GNU toolchain (meaning mingw-w64)? Or can the Go linker handle .obj 
> files produced by MSVC?
>

I don't think the Go linker can handle MSVC object files. However,
because AccessKit is meant to be used on other platforms, won't you need
gcc/clang support anyway?

> Also, I see that Gio's CI uses Wine to run Windows tests on Linux. For 
> what it's worth, Wine's implementation of UI Automation, the current 
> Windows accessibility API, is only a stub. And it's implementation of 
> Active Accessibility, the legacy accessibility API, is also very 
> incomplete. So this means that the AccessKit Windows library can't make 
> any accessibility API calls unless an assistive technology is actually 
> running, or if it does have to proactively make any API calls, it should 
> fail gracefully. That shouldn't be a problem. It also means that 
> end-to-end accessibility testing won't be possible on Gio's CI. That 
> should also be OK, since AccessKit will have its own CI (probably GitHub 
> Actions), which will have access to the actual proprietary platforms.
>

Or Gio could upgrade its CI :)

Elias
Details
Message ID
<7fc22ec9-63a3-6b74-ce25-a1c35b874625@pobox.com>
In-Reply-To
<CC47ZD3W6FCO.2S4W112K2V0MU@testmac> (view parent)
DKIM signature
pass
Download raw message
On 6/15/2021 8:18 AM, Elias Naur wrote:

> I don't think the Go linker can handle MSVC object files. However,
> because AccessKit is meant to be used on other platforms, won't you need
> gcc/clang support anyway?
The AccessKit libraries for other C-based platforms will certainly 
compile with GCC and clang. (The Android one will be all Java, and the 
web one will be all JavaScript or TypeScript.) But I haven't yet decided 
about the Windows library. Yes, each platform will have its own library; 
the one thing they will all have in common is the message schema 
(probably using Protobuf). The libraries for C-based platforms may also 
export the same C ABI; I haven't decided on that yet.
> Or Gio could upgrade its CI :) 
I didn't want to get into an argument over what I thought might be an 
ideological decision. I saw that you went to some trouble to do Windows 
tests and Apple builds on Linux, so I figured I shouldn't try to 
convince you to do otherwise.

Matt
Reply to thread Export thread (mbox)