~eliasnaur/gio

3 2

Unicode support?

Details
Message ID
<CALGqR9J1Z2CAvGG-zJNesSQtKS2wp=qi4HYUzF+GRz11nhSMiA@mail.gmail.com>
DKIM signature
pass
Download raw message
Hi all,

Can I just confirm whether there is in fact support for Unicode?

In our project[1] if you type in some Unicode characters like those
typically found on the macOS Emoji keyboard (like 😅) are rendered as
squares 😂

Kind regards

James

[1]: https://git.mills.io/saltyim/app

-- James Mills / prologic

Join Yarn.social today! The only decentralised social media that
respects your privacy and freedoms!

E:    prologic@shortcircuit.net.au
W:    prologic.shortcircuit.net.au
Blog: Read my Blog
Yarn: @prologic@twtxt.net
Details
Message ID
<CAFcc3FR9GrboKnMsUUG+UHAY2mDOQVc1PdLgmLetniTWFB5AnQ@mail.gmail.com>
In-Reply-To
<CALGqR9J1Z2CAvGG-zJNesSQtKS2wp=qi4HYUzF+GRz11nhSMiA@mail.gmail.com> (view parent)
DKIM signature
pass
Download raw message
Hey James,

> Can I just confirm whether there is in fact support for Unicode?
>
> In our project[1] if you type in some Unicode characters like those
> typically found on the macOS Emoji keyboard (like 😅) are rendered as
> squares 😂

This is a slightly pedantic response, but I think the distinction is
important. We totally support unicode, and our recent upgrades to our
text shaper mean that we can shape most opentype fonts (even really
complex ones).

The reason that you see those box characters isn't a lack of support
for displaying them, but a lack of font fallback support. Basically,
Gio selects a single font to display your string of text, and it does
not fall back to searching for other fonts when it encounters a
character that is not available within the primary font. Since your
primary font is intended to display latin text, it does not contain
the glyphs for emoji. This means you see the placeholder character
instead.

We certainly want and intend to have good font fallback support, but I
haven't been able to add it. I wrote a recent slack post detailing the
work required to enable this feature here:

https://gophers.slack.com/archives/CM87SNCGM/p1651670531994129?thread_ts=1651657472.794499&cid=CM87SNCGM

I'll quote it below for those who do not use slack:

> Since the current state of things is my fault, let me tell you about it: when I upgraded the text shaper to support complex scripts, I had to drop support for font fallback temporarily in order to reduce the complexity of the design. Gio used to search all of the fonts loaded into your theme for a font appropriate to each unicode code point and use that font for each codepoint. This model doesn't translate very well to our new text shaper, which instead requires dividing the text into "runs" of consecutive text which each use symbols from a different font.
>
> Right now, Gio only can use a single font for each label/text widget, and (as Lucas says above) most fonts don't contain both letters and emoji. This means that either you can see the letters or the emoji. As Lucas correctly points out, you can use font combining tools to merge fonts so that you have a font with support for both the languages and emoji you want, but it's certainly not ideal.
>
> The reason that I haven't already re-implemented font fallback is that it requires changes to our line wrapping algorithm. The line wrapper would need to be modified to do two things:
>
> - accept multiple shaped output runs as input (currently it accepts only one)
> - track the font for each output separately (right now all outputs are assumed to use the same font)
>
> That code is kinda tricky, and I haven't had enough time to tackle it. If we had the changes I described, we could then divide the text by font coverage and shape each run in the first appropriate font we could find.
>
> The line wrapper code is linked below. We'd need to change its signature to accept multiple Output (the shaped text type), as well as change how it's invoked to actually shape text with multiple different fonts.
>
> https://git.sr.ht/~eliasnaur/gio/tree/main/item/font/opentype/internal/shaping.go#L296
>
> So yeah... right now this situation isn't great. Sorry! I'll work on it when I can, but I don't know when that will be. I'd be happy to help someone else do it if they wanted to try.

You can work around the problem by using a font that supports both
latin text and emoji, but I should warn you that I believe we do have
some outstanding work to do for color emoji fonts. If you're okay with
black and white emoji, this font should work well:

https://developers.googleblog.com/2022/04/what-is-black-and-white-and-read-all.html?m=1

Cheers,
Chris
Details
Message ID
<CALGqR9JxXBpWAeF3C7tsXyO3r=EjZJix8q4NyU7djQj3jT061w@mail.gmail.com>
In-Reply-To
<CAFcc3FR9GrboKnMsUUG+UHAY2mDOQVc1PdLgmLetniTWFB5AnQ@mail.gmail.com> (view parent)
DKIM signature
pass
Download raw message
Thanks Chris! 🤗

I know you have your daughter coming along any day now, but given that
this is very likely beyond the scope of my expertise, I'm not sure I
could pick this code up and do what you've described 😅

How can we help convince you to re-add the font fallback feature? 🙇‍♂️

cheers
James

-- James Mills / prologic

Join Yarn.social today! The only decentralised social media that
respects your privacy and freedoms!

E:    prologic@shortcircuit.net.au
W:    prologic.shortcircuit.net.au
Blog: Read my Blog
Yarn: @prologic@twtxt.net

On Fri, May 6, 2022 at 1:09 AM Chris Waldon
<christopher.waldon.dev@gmail.com> wrote:
>
> Hey James,
>
> > Can I just confirm whether there is in fact support for Unicode?
> >
> > In our project[1] if you type in some Unicode characters like those
> > typically found on the macOS Emoji keyboard (like 😅) are rendered as
> > squares 😂
>
> This is a slightly pedantic response, but I think the distinction is
> important. We totally support unicode, and our recent upgrades to our
> text shaper mean that we can shape most opentype fonts (even really
> complex ones).
>
> The reason that you see those box characters isn't a lack of support
> for displaying them, but a lack of font fallback support. Basically,
> Gio selects a single font to display your string of text, and it does
> not fall back to searching for other fonts when it encounters a
> character that is not available within the primary font. Since your
> primary font is intended to display latin text, it does not contain
> the glyphs for emoji. This means you see the placeholder character
> instead.
>
> We certainly want and intend to have good font fallback support, but I
> haven't been able to add it. I wrote a recent slack post detailing the
> work required to enable this feature here:
>
> https://gophers.slack.com/archives/CM87SNCGM/p1651670531994129?thread_ts=1651657472.794499&cid=CM87SNCGM
>
> I'll quote it below for those who do not use slack:
>
> > Since the current state of things is my fault, let me tell you about it: when I upgraded the text shaper to support complex scripts, I had to drop support for font fallback temporarily in order to reduce the complexity of the design. Gio used to search all of the fonts loaded into your theme for a font appropriate to each unicode code point and use that font for each codepoint. This model doesn't translate very well to our new text shaper, which instead requires dividing the text into "runs" of consecutive text which each use symbols from a different font.
> >
> > Right now, Gio only can use a single font for each label/text widget, and (as Lucas says above) most fonts don't contain both letters and emoji. This means that either you can see the letters or the emoji. As Lucas correctly points out, you can use font combining tools to merge fonts so that you have a font with support for both the languages and emoji you want, but it's certainly not ideal.
> >
> > The reason that I haven't already re-implemented font fallback is that it requires changes to our line wrapping algorithm. The line wrapper would need to be modified to do two things:
> >
> > - accept multiple shaped output runs as input (currently it accepts only one)
> > - track the font for each output separately (right now all outputs are assumed to use the same font)
> >
> > That code is kinda tricky, and I haven't had enough time to tackle it. If we had the changes I described, we could then divide the text by font coverage and shape each run in the first appropriate font we could find.
> >
> > The line wrapper code is linked below. We'd need to change its signature to accept multiple Output (the shaped text type), as well as change how it's invoked to actually shape text with multiple different fonts.
> >
> > https://git.sr.ht/~eliasnaur/gio/tree/main/item/font/opentype/internal/shaping.go#L296
> >
> > So yeah... right now this situation isn't great. Sorry! I'll work on it when I can, but I don't know when that will be. I'd be happy to help someone else do it if they wanted to try.
>
> You can work around the problem by using a font that supports both
> latin text and emoji, but I should warn you that I believe we do have
> some outstanding work to do for color emoji fonts. If you're okay with
> black and white emoji, this font should work well:
>
> https://developers.googleblog.com/2022/04/what-is-black-and-white-and-read-all.html?m=1
>
> Cheers,
> Chris
Details
Message ID
<CALGqR9+F3EJrTx=yQA_qjhpQ94eqa0wj0rmx4LzeFyUCZja9Og@mail.gmail.com>
In-Reply-To
<CAFcc3FR9GrboKnMsUUG+UHAY2mDOQVc1PdLgmLetniTWFB5AnQ@mail.gmail.com> (view parent)
DKIM signature
pass
Download raw message
Hi all,

This is probably foolish of me to suggest this, being legally blind
and all, I have no business doing anything UI/UX related, but I'm
going to see if I can wrap my puny little head around this section of
code and figure out how to get multiple fonts working again 😅

I may need your help!

cheers
James

-- James Mills / prologic

Join Yarn.social today! The only decentralised social media that
respects your privacy and freedoms!

E:    prologic@shortcircuit.net.au
W:    prologic.shortcircuit.net.au
Blog: Read my Blog
Yarn: @prologic@twtxt.net

On Fri, May 6, 2022 at 1:09 AM Chris Waldon
<christopher.waldon.dev@gmail.com> wrote:
>
> Hey James,
>
> > Can I just confirm whether there is in fact support for Unicode?
> >
> > In our project[1] if you type in some Unicode characters like those
> > typically found on the macOS Emoji keyboard (like 😅) are rendered as
> > squares 😂
>
> This is a slightly pedantic response, but I think the distinction is
> important. We totally support unicode, and our recent upgrades to our
> text shaper mean that we can shape most opentype fonts (even really
> complex ones).
>
> The reason that you see those box characters isn't a lack of support
> for displaying them, but a lack of font fallback support. Basically,
> Gio selects a single font to display your string of text, and it does
> not fall back to searching for other fonts when it encounters a
> character that is not available within the primary font. Since your
> primary font is intended to display latin text, it does not contain
> the glyphs for emoji. This means you see the placeholder character
> instead.
>
> We certainly want and intend to have good font fallback support, but I
> haven't been able to add it. I wrote a recent slack post detailing the
> work required to enable this feature here:
>
> https://gophers.slack.com/archives/CM87SNCGM/p1651670531994129?thread_ts=1651657472.794499&cid=CM87SNCGM
>
> I'll quote it below for those who do not use slack:
>
> > Since the current state of things is my fault, let me tell you about it: when I upgraded the text shaper to support complex scripts, I had to drop support for font fallback temporarily in order to reduce the complexity of the design. Gio used to search all of the fonts loaded into your theme for a font appropriate to each unicode code point and use that font for each codepoint. This model doesn't translate very well to our new text shaper, which instead requires dividing the text into "runs" of consecutive text which each use symbols from a different font.
> >
> > Right now, Gio only can use a single font for each label/text widget, and (as Lucas says above) most fonts don't contain both letters and emoji. This means that either you can see the letters or the emoji. As Lucas correctly points out, you can use font combining tools to merge fonts so that you have a font with support for both the languages and emoji you want, but it's certainly not ideal.
> >
> > The reason that I haven't already re-implemented font fallback is that it requires changes to our line wrapping algorithm. The line wrapper would need to be modified to do two things:
> >
> > - accept multiple shaped output runs as input (currently it accepts only one)
> > - track the font for each output separately (right now all outputs are assumed to use the same font)
> >
> > That code is kinda tricky, and I haven't had enough time to tackle it. If we had the changes I described, we could then divide the text by font coverage and shape each run in the first appropriate font we could find.
> >
> > The line wrapper code is linked below. We'd need to change its signature to accept multiple Output (the shaped text type), as well as change how it's invoked to actually shape text with multiple different fonts.
> >
> > https://git.sr.ht/~eliasnaur/gio/tree/main/item/font/opentype/internal/shaping.go#L296
> >
> > So yeah... right now this situation isn't great. Sorry! I'll work on it when I can, but I don't know when that will be. I'd be happy to help someone else do it if they wanted to try.
>
> You can work around the problem by using a font that supports both
> latin text and emoji, but I should warn you that I believe we do have
> some outstanding work to do for color emoji fonts. If you're okay with
> black and white emoji, this font should work well:
>
> https://developers.googleblog.com/2022/04/what-is-black-and-white-and-read-all.html?m=1
>
> Cheers,
> Chris
Reply to thread Export thread (mbox)