Hey all, This patchset totally overhauls text in Gio to support important features like font fallback and bidirectional text. As part of that restructure, the text shaping stack gets noticeably faster [0] in most cases. The cases in which it gets slower are those in which support for bidirectional text means the new shaper is doing significantly more work than the old one. [0] https://paste.sr.ht/~whereswaldon/a5b314a3277c8f3f63d9d90f09cab95339322da7 I've included a summary of the structural changes in the commit that actually modifies the text packages. I look forward to your thoughts, Chris Chris Waldon (4): io/system: implement Stringer on TextDirection widget: define text rendering benchmarks widget: fix oob-access in editor buffer font/{gofont,opentype},text,widget{,/material}: [API] add font fallback and bidi support app/window.go | 2 +- font/gofont/gofont.go | 4 +- font/opentype/internal/shaping.go | 521 ------ font/opentype/internal/shaping_test.go | 1522 ----------------- font/opentype/opentype.go | 609 ++++++- font/opentype/opentype_test.go | 853 ++++++++- ...15eb6104cf09159c7d756cc3530ccaa9007c0a2c06 | 5 + ...6a67df3aa3a562e2f2f1cfb665a650f60e7e0ab4e6 | 5 + ...a74279d7e3e486d8ac211e45049cfa4833257ef236 | 5 + ...077b6f4ba91b040119d82d55991d568014f4ba671e | 5 + ...1bc74e2377c0664f64bbf2ac8b1cdd0ce9f55461b3 | 5 + ...857d5487d1a89309f7312ccde5ff8b94fcd29521eb | 5 + ...75cb8900f97fc1c20e19de09fc0ffca8ff0c8d9e5a | 5 + ...39baaace51fa4348bed0e3b662dd50aa341f67d10c | 5 + ...f4ea99e50a6ecd0d4134e3a77234172d13787ac4ea | 5 + ...83f58a3ca4eba895087316b31e598f0b05c978d87c | 5 + ...60599b281593098b0aad864231d756c3e9699029c1 | 5 + go.mod | 8 +- go.sum | 16 +- io/system/locale.go | 9 + text/lru.go | 59 +- text/lru_test.go | 4 +- text/shaper.go | 237 ++- text/shaper_test.go | 91 +- text/text.go | 75 +- widget/buffer.go | 2 +- widget/editor.go | 506 ++++-- widget/editor_test.go | 34 +- widget/label.go | 57 +- widget/material/theme.go | 4 +- ...120fe7b82978ac0e5c16afb519beb080f73c470d3f | 4 + ...645d025ac15316f1210d4aa307cde333cc87fdad55 | 4 + .../FuzzEditorEditing/clusterIndexForCrash1 | 4 + .../FuzzEditorEditing/clusterIndexForCrash2 | 4 + widget/text_bench_test.go | 374 ++++ widget/text_test.go | 1075 ++++++++++-- 36 files changed, 3607 insertions(+), 2526 deletions(-) delete mode 100644 font/opentype/internal/shaping.go delete mode 100644 font/opentype/internal/shaping_test.go create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/2a7730fcbcc3550718b37415eb6104cf09159c7d756cc3530ccaa9007c0a2c06 create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/3fc3ee939f0df44719bee36a67df3aa3a562e2f2f1cfb665a650f60e7e0ab4e6 create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/594e4fda2e3462061d50b6a74279d7e3e486d8ac211e45049cfa4833257ef236 create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/6b452fd81f16c000dbe525077b6f4ba91b040119d82d55991d568014f4ba671e create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/6b5a1e9cd750a9aa8dafe11bc74e2377c0664f64bbf2ac8b1cdd0ce9f55461b3 create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/be56090b98f3c84da6ff9e857d5487d1a89309f7312ccde5ff8b94fcd29521eb create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/dda958d1b1bb9df71f81c575cb8900f97fc1c20e19de09fc0ffca8ff0c8d9e5a create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/de31ec6b1ac7797daefecf39baaace51fa4348bed0e3b662dd50aa341f67d10c create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/e79034d6c7a6ce74d7a689f4ea99e50a6ecd0d4134e3a77234172d13787ac4ea create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/f1f0611baadc20469863e483f58a3ca4eba895087316b31e598f0b05c978d87c create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/f2ebea678c72f6c394d7f860599b281593098b0aad864231d756c3e9699029c1 create mode 100644 widget/testdata/fuzz/FuzzEditorEditing/18c534da60e6b61361786a120fe7b82978ac0e5c16afb519beb080f73c470d3f create mode 100644 widget/testdata/fuzz/FuzzEditorEditing/a489f2d9f9226d13b55846645d025ac15316f1210d4aa307cde333cc87fdad55 create mode 100644 widget/testdata/fuzz/FuzzEditorEditing/clusterIndexForCrash1 create mode 100644 widget/testdata/fuzz/FuzzEditorEditing/clusterIndexForCrash2 create mode 100644 widget/text_bench_test.go -- 2.34.5
Thanks, merged. Elias
Thanks, merged! Elias
This looks like you're papering over another bug. e.pos should never be > e.len().
Great work! Comments below
Thanks for the review! Replies below. Chris
Needs a Fixes/References header?
Huh. We don't really have a ticket for either of these features. The closest I can find are: https://todo.sr.ht/~eliasnaur/gio/425 https://todo.sr.ht/~eliasnaur/gio/211
Elias
Always returning an opentype.Shaper makes material.NewTheme easier to use, but I don't like that all Collection functions of every font package need to decide on and return a shaper. Perhaps we should just bite the bullet and let material.Theme use opentype.Shaper if its Shaper field is nil. Users wanting a custom shaper probably don't mind the binary size hit. And if they do, they probably also want a custom theme tailored for weak devices.
This exposes go-text/typesetting/font to our users. Do we want that?
Nit: I think we can spell this `any` now.
Ack
However, this seems like a code smell: if Face is opaque, why does package text have to deal with it at all? Why not deal only in Fonts and let the shaper implementation map them to Faces?
It seems to me needsVisual can be dropped as a result and ge redefined to positionVisuallyContrains in case needsVisual is true. Perhaps positionGreaterOrEqual should be renamed.
On Wed, Nov 9, 2022 at 1:46 PM Chris Waldon <christopher.waldon.dev@gmail.com> wrote:
Copy & paste the following snippet into your terminal to import this patchset into git:
curl -s https://lists.sr.ht/~eliasnaur/gio-patches/patches/36683/mbox | git am -3Learn more about email & git
From: Chris Waldon <christopher.waldon.dev@gmail.com> Signed-off-by: Chris Waldon <christopher.waldon.dev@gmail.com> --- io/system/locale.go | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/io/system/locale.go b/io/system/locale.go index 7bc75817..388d2e5b 100644 --- a/io/system/locale.go +++ b/io/system/locale.go @@ -34,6 +34,15 @@ func (d TextDirection) Progression() TextProgression { return TextProgression((d & (1 << progressionShift)) >> progressionShift) } +func (d TextDirection) String() string { + switch d { + case RTL: + return "RTL" + default: + return "LTR" + } +} + // TextAxis defines the layout axis of text. type TextAxis byte -- 2.34.5
Thanks, merged. Elias
From: Chris Waldon <christopher.waldon.dev@gmail.com> This commit adds a series of benchmarks for text rendering. They are intended to capture the performance of static and continuously changing text within labels and editors, and will serve as a baseline to compare the post-bidi text stack against. Signed-off-by: Chris Waldon <christopher.waldon.dev@gmail.com> --- widget/text_bench_test.go | 274 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 274 insertions(+) create mode 100644 widget/text_bench_test.go diff --git a/widget/text_bench_test.go b/widget/text_bench_test.go new file mode 100644 index 00000000..68e41b85 --- /dev/null +++ b/widget/text_bench_test.go @@ -0,0 +1,274 @@ +package widget + +import ( + "fmt" + "image" + "math/rand" + "testing" + "time" + + "gioui.org/font/gofont" + "gioui.org/io/system" + "gioui.org/layout" + "gioui.org/op" + "gioui.org/text" + "gioui.org/unit" +) + +var ( + documents = map[string]string{ + "latin": latinDocument, + "arabic": arabicDocument, + "complex": complexDocument, + } + sizes = []int{10, 100, 1000} + locales = []system.Locale{arabic, english} + benchFonts = func() []text.FontFace { + gofonts := gofont.Collection() + return append(arabicCollection, gofonts...) + }() +) + +func init() { + rand.Seed(int64(time.Now().Nanosecond())) +} + +func BenchmarkLabelStatic(b *testing.B) { + for _, locale := range locales { + for _, runes := range sizes { + for textType, txt := range documents { + b.Run(fmt.Sprintf("%drunes-%s-%s", runes, locale.Direction, textType), func(b *testing.B) { + gtx := layout.Context{ + Ops: new(op.Ops), + Constraints: layout.Constraints{ + Max: image.Pt(200, 1000), + }, + Locale: locale, + } + cache := text.NewCache(benchFonts) + fontSize := unit.Sp(10) + font := text.Font{} + runes := []rune(txt)[:runes] + runesStr := string(runes) + l := Label{} + b.ResetTimer() + for i := 0; i < b.N; i++ { + l.Layout(gtx, cache, font, fontSize, runesStr) + gtx.Ops.Reset() + } + }) + } + } + } +} + +func BenchmarkLabelDynamic(b *testing.B) { + for _, locale := range locales { + for _, runes := range sizes { + for textType, txt := range documents { + b.Run(fmt.Sprintf("%drunes-%s-%s", runes, locale.Direction, textType), func(b *testing.B) { + gtx := layout.Context{ + Ops: new(op.Ops), + Constraints: layout.Constraints{ + Max: image.Pt(200, 1000), + }, + Locale: locale, + } + cache := text.NewCache(benchFonts) + fontSize := unit.Sp(10) + font := text.Font{} + runes := []rune(txt)[:runes] + l := Label{} + b.ResetTimer() + for i := 0; i < b.N; i++ { + // simulate a constantly changing string + a := rand.Intn(len(runes)) + b := rand.Intn(len(runes)) + runes[a], runes[b] = runes[b], runes[a] + l.Layout(gtx, cache, font, fontSize, string(runes)) + gtx.Ops.Reset() + } + }) + } + } + } +} + +func BenchmarkEditorStatic(b *testing.B) { + for _, locale := range locales { + for _, runes := range sizes { + for textType, txt := range documents { + b.Run(fmt.Sprintf("%drunes-%s-%s", runes, locale.Direction, textType), func(b *testing.B) { + gtx := layout.Context{ + Ops: new(op.Ops), + Constraints: layout.Constraints{ + Max: image.Pt(200, 1000), + }, + Locale: locale, + } + cache := text.NewCache(benchFonts) + fontSize := unit.Sp(10) + font := text.Font{} + runes := []rune(txt)[:runes] + runesStr := string(runes) + e := Editor{} + e.SetText(runesStr) + b.ResetTimer() + for i := 0; i < b.N; i++ { + e.Layout(gtx, cache, font, fontSize, func(gtx layout.Context) layout.Dimensions { + e.PaintSelection(gtx) + e.PaintText(gtx) + e.PaintCaret(gtx) + return layout.Dimensions{Size: gtx.Constraints.Min} + }) + gtx.Ops.Reset() + } + }) + } + } + } +} + +func BenchmarkEditorDynamic(b *testing.B) { + for _, locale := range locales { + for _, runes := range sizes { + for textType, txt := range documents { + b.Run(fmt.Sprintf("%drunes-%s-%s", runes, locale.Direction, textType), func(b *testing.B) { + gtx := layout.Context{ + Ops: new(op.Ops), + Constraints: layout.Constraints{ + Max: image.Pt(200, 1000), + }, + Locale: locale, + } + cache := text.NewCache(benchFonts) + fontSize := unit.Sp(10) + font := text.Font{} + runes := []rune(txt)[:runes] + e := Editor{} + e.SetText(string(runes)) + b.ResetTimer() + for i := 0; i < b.N; i++ { + // simulate a constantly changing string + a := rand.Intn(e.Len()) + b := rand.Intn(e.Len()) + e.SetCaret(a, a+1) + takeStr := e.SelectedText() + e.Insert("") + e.SetCaret(b, b) + e.Insert(takeStr) + e.Layout(gtx, cache, font, fontSize, func(gtx layout.Context) layout.Dimensions { + e.PaintSelection(gtx) + e.PaintText(gtx) + e.PaintCaret(gtx) + return layout.Dimensions{Size: gtx.Constraints.Min} + }) + gtx.Ops.Reset() + } + }) + } + } + } +} + +const ( + latinDocument = `Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +Porttitor eget dolor morbi non arcu risus quis. +Nibh sit amet commodo nulla. +Posuere ac ut consequat semper viverra nam libero justo. +Risus in hendrerit gravida rutrum quisque. +Natoque penatibus et magnis dis parturient montes nascetur. +In metus vulputate eu scelerisque felis imperdiet proin fermentum. +Mattis rhoncus urna neque viverra. +Elit pellentesque habitant morbi tristique. +Nisl nunc mi ipsum faucibus vitae aliquet nec. +Sed augue lacus viverra vitae congue eu consequat. +At quis risus sed vulputate odio ut. +Sit amet volutpat consequat mauris nunc congue nisi. +Dignissim cras tincidunt lobortis feugiat. +Faucibus turpis in eu mi bibendum. +Odio aenean sed adipiscing diam donec adipiscing tristique. +Fermentum leo vel orci porta non pulvinar. +Ut venenatis tellus in metus vulputate eu scelerisque felis imperdiet. +Et netus et malesuada fames ac turpis. +Venenatis urna cursus eget nunc scelerisque viverra mauris in. +Risus ultricies tristique nulla aliquet enim tortor. +Risus pretium quam vulputate dignissim suspendisse in. +Interdum velit euismod in pellentesque massa placerat duis ultricies lacus. +Proin gravida hendrerit lectus a. +Auctor augue mauris augue neque gravida in fermentum et. +Laoreet sit amet cursus sit amet dictum. +In fermentum et sollicitudin ac orci phasellus egestas tellus rutrum. +Tempus imperdiet nulla malesuada pellentesque elit eget gravida. +Consequat id porta nibh venenatis cras sed. +Vulputate ut pharetra sit amet aliquam. +Congue mauris rhoncus aenean vel elit. +Risus quis varius quam quisque id diam vel quam elementum. +Pretium lectus quam id leo in vitae. +Sed sed risus pretium quam vulputate dignissim suspendisse in est. +Velit laoreet id donec ultrices. +Nunc sed velit dignissim sodales ut. +Nunc scelerisque viverra mauris in aliquam sem fringilla ut. +Sed enim ut sem viverra aliquet eget sit. +Convallis posuere morbi leo urna molestie at. +Aliquam id diam maecenas ultricies mi eget mauris. +Ipsum dolor sit amet consectetur adipiscing elit ut aliquam. +Accumsan tortor posuere ac ut consequat semper. +Viverra vitae congue eu consequat ac felis donec et odio. +Scelerisque in dictum non consectetur a. +Consequat nisl vel pretium lectus quam id leo in vitae. +Morbi tristique senectus et netus et malesuada fames ac turpis. +Ac orci phasellus egestas tellus. +Tempus egestas sed sed risus. +Ullamcorper morbi tincidunt ornare massa eget egestas purus. +Nibh venenatis cras sed felis eget velit.` + arabicDocument = `و سأعرض مثال حي لهذا، من منا لم يتحمل جهد بدني شاق إلا من أجل الحصول على ميزة أو فائدة؟ ولكن من لديه الحق أن ينتقد شخص ما أراد أن يشعر بالسعادة التي لا تشوبها عواقب أليمة أو آخر أراد أن يتجنب الألم الذي ربما تنجم عنه بعض المتعة ؟ علي الجانب الآخر نشجب ونستنكر هؤلاء الرجال المفتونون بنشوة اللحظة الهائمون في رغباتهم فلا يدركون ما يعقبها من الألم والأسي المحتم، واللوم كذلك يشمل هؤلاء الذين أخفقوا في واجباتهم نتيجة لضعف إرادتهم فيتساوي مع هؤلاء الذين يتجنبون وينأون عن تحمل الكدح والألم . +من المفترض أن نفرق بين هذه الحالات بكل سهولة ومرونة. +في ذاك الوقت عندما تكون قدرتنا علي الاختيار غير مقيدة بشرط وعندما لا نجد ما يمنعنا أن نفعل الأفضل فها نحن نرحب بالسرور والسعادة ونتجنب كل ما يبعث إلينا الألم. +في بعض الأحيان ونظراً للالتزامات التي يفرضها علينا الواجب والعمل سنتنازل غالباً ونرفض الشعور بالسرور ونقبل ما يجلبه إلينا الأسى. +الإنسان الحكيم عليه أن يمسك زمام الأمور ويختار إما أن يرفض مصادر السعادة من أجل ما هو أكثر أهمية أو يتحمل الألم من أجل ألا يتحمل ما هو أسوأ. +و سأعرض مثال حي لهذا، من منا لم يتحمل جهد بدني شاق إلا من أجل الحصول على ميزة أو فائدة؟ ولكن من لديه الحق أن ينتقد شخص ما أراد أن يشعر بالسعادة التي لا تشوبها عواقب أليمة أو آخر أراد أن يتجنب الألم الذي ربما تنجم عنه بعض المتعة ؟ علي الجانب الآخر نشجب ونستنكر هؤلاء الرجال المفتونون بنشوة اللحظة الهائمون في رغباتهم فلا يدركون ما يعقبها من الألم والأسي المحتم، واللوم كذلك يشمل هؤلاء الذين أخفقوا في واجباتهم نتيجة لضعف إرادتهم فيتساوي مع هؤلاء الذين يتجنبون وينأون عن تحمل الكدح والألم . +من المفترض أن نفرق بين هذه الحالات بكل سهولة ومرونة. +في ذاك الوقت عندما تكون قدرتنا علي الاختيار غير مقيدة بشرط وعندما لا نجد ما يمنعنا أن نفعل الأفضل فها نحن نرحب بالسرور والسعادة ونتجنب كل ما يبعث إلينا الألم. +في بعض الأحيان ونظراً للالتزامات التي يفرضها علينا الواجب والعمل سنتنازل غالباً ونرفض الشعور بالسرور ونقبل ما يجلبه إلينا الأسى. +الإنسان الحكيم عليه أن يمسك زمام الأمور ويختار إما أن يرفض مصادر السعادة من أجل ما هو أكثر أهمية أو يتحمل الألم من أجل ألا يتحمل ما هو أسوأ.` + complexDocument = `و سأعرض مثال dolor sit amet, لم يتحمل جهد adipiscing elit, sed do الحصول على ميزة incididunt ut labore أن ينتقد magna aliqua. +Porttitor إرادتهم فيتساوي morbi non arcu يدركون ما يعقبها . +Nibh نشجب ونستنكر commodo nulla. +بكل سهولة ومرونة ut consequat لهذا، من منا nam libero justo. +Risus in hendrerit علينا الواجب والعمل. +Natoque تكون قدرتنا علي magnis dis parturient يمسك زمام الأمور ويختار. +In نجد ما يمنعنا eu scelerisque ونظراً للالتزامات التي fermentum. +Mattis ة بشرط وعندما لا neque viverra. +يمسك زمام الأمور habitant لهذا، من. +Nisl تي يفرضها علينا faucibus ،من منا لم nec. +Sed augue علي الاختيار غير vitae congue eu consequat. +At quis risus سك زمام الأمور ويختار. +Sit amet volutpat consequat mauris الأمور ويختار إما nisi. +Dignissim لواجب والعمل tincidunt سنتنازل feugiat. +Faucibus التزامات in eu mi bibendum. +Odio ويختار إما أن يرفض مصادر السعادة sed adipiscing ذا، من منا لم tristique. +Fermentum leo vel ور ويختار إما pulvinar. +Ut ر إما أن يرفض مصادر السعادة من in metus تكون قدرتنا علي felis imperdiet. +ي الاختيار غير مقيدة بشرط et malesuada fames ac turpis. +Venenatis على ميزة أو فائدة؟ ولكن eget nunc scelerisque سك زمام الأمور ويختار إما in. +رتنا ultricies tristique ي الاختيار غير مقيدة بشرط enim tortor. +Risus اختيار غير مقيدة بشرط وعندما quam سان الحكيم عليه أن suspendisse in. +Interdum velit ونظراً للالتزامات التي pellentesque massa placerat لأمور ويختار إما أن يرفض lacus. +Proin دما تكون قدرتنا علي الاختيار lectus a. +Auctor الوقت عندما تكون augue neque ض مثال حي fermentum et. +Laoreet مسك زمام الأمور ويختار amet cursus لم يتحمل جهد dictum. +In fermentum et sollicitudin ac orci phasellus علي الاختيار غير rutrum. +Tempus imperdiet المفترض أن نفرق pellentesque ت بكل سهولة eget gravida. +Consequat id portaمصادر السعادة cras sed. +Vulputate علي الاختيار غير مقيدة sit amet aliquam. +Congue mauris حيان ونظراً للالتزامات التي vel elit. +Risus quis varius quam quisque id ار غير مقيدة بشرط elementum. +Pretium تي يفرضها علينا الواجب leo in vitae. + شاق إلا من أجل pretium quam الحكيم عليه أن يمسك suspendisse in est. +Velit ونظراً للالتزامات التي يفرضها ultrices. + الوقت عندما تكون velit dignissim يه أن يمسك . +Nunc scelerisque viverra mauris in aliquam sem ر إما أن ut. +السعادة من أجل ما هو أكثر أهمية أو يتحمل الألم +Convallis posuere morbi leo urna molestie at.` +) -- 2.34.5
Thanks, merged! Elias
From: Chris Waldon <christopher.waldon.dev@gmail.com> This fixes an out-of-bounds crash that I found using the fuzz tests added in the rest of this patchset. I've included this fix before the tests that trigger it so that I'm not introducing a commit that fails the tests. Signed-off-by: Chris Waldon <christopher.waldon.dev@gmail.com> --- widget/buffer.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/widget/buffer.go b/widget/buffer.go index 40e02a86..96e80146 100644 --- a/widget/buffer.go +++ b/widget/buffer.go @@ -135,7 +135,7 @@ func (e *editBuffer) Read(p []byte) (int, error) { } func (e *editBuffer) ReadRune() (rune, int, error) { - if e.pos == e.len() { + if e.pos >= e.len() { return 0, 0, io.EOF } r, s := e.runeAt(e.pos) -- 2.34.5
This looks like you're papering over another bug. e.pos should never be > e.len().
From: Chris Waldon <christopher.waldon.dev@gmail.com> This commit restructures the entire text shaping stack to enable lines of shaped text to have non-homogeneous properties like which font face they belong to and which direction a segment of text is going. This required restructuring both text.Line and the text shaping cache extensively. text.Line now holds a slice of text.Layout, each of which is a run of homogeneous text (all one font, all one direction). Because text.Layout is now a slice element referenced by a text.Line instead of a nested struct, it is no longer safe to modify the contents of a text.Layout in order to clip away glyphs that shouldn't be visible, so the strategy for hiding off-screen glyphs changed to informing the shaper which range of glyphs should be displayed and allowing it to iterate glyphs as needed. Speaking of the shaper, I've defined a second shaping interface. text.Shaper is the externally-consumed-by-widgets interface for text shaping, but text.FaceShaper is the interface implemented by a text shaping backend like opentype. This interface accepts a prioritized list of faces to use when shaping (instead of a text.Font). This separation allows the font selection algorithm to live in text.Cache and be reused across multiple shaping backends. In order to compensate for the dramatic increase in text shaping complexity, I've modified the text caching strategy to cache at the paragraph level instead of caching the entire input. This allows for dramatic performance improvements when re-shaping partially modified large text input. However, this required changes to text.Line because its rune accounting now cannot reference its absolute position within the document (that can change when shaping the contents of an editor, for example). Rewriting the rune offsets when we retrieve a line from the cache is error-prone and ultimately unnecessary. We only really needed to know how many runes a line represented, not where they were within the input text, so text.Line.Runes is now just text.Line.RuneCount. Similarly, all line-internal rune accounting is now relative to the start of the line instead of the start of the text input. This commit consumes the upstream github.com/go-text/typesetting/shaping API now that my prior work is merged there, removing the need for the font/opentype/internal package entirely. As part of my efforts, I fuzzed both the low-level text shaping stack and the editor widget extensively. I've committed regression tests found that way into the appropriate testdata files to ensure the fuzzer re-checks them. Signed-off-by: Chris Waldon <christopher.waldon.dev@gmail.com> --- app/window.go | 2 +- font/gofont/gofont.go | 4 +- font/opentype/internal/shaping.go | 521 ------ font/opentype/internal/shaping_test.go | 1522 ----------------- font/opentype/opentype.go | 609 ++++++- font/opentype/opentype_test.go | 853 ++++++++- ...15eb6104cf09159c7d756cc3530ccaa9007c0a2c06 | 5 + ...6a67df3aa3a562e2f2f1cfb665a650f60e7e0ab4e6 | 5 + ...a74279d7e3e486d8ac211e45049cfa4833257ef236 | 5 + ...077b6f4ba91b040119d82d55991d568014f4ba671e | 5 + ...1bc74e2377c0664f64bbf2ac8b1cdd0ce9f55461b3 | 5 + ...857d5487d1a89309f7312ccde5ff8b94fcd29521eb | 5 + ...75cb8900f97fc1c20e19de09fc0ffca8ff0c8d9e5a | 5 + ...39baaace51fa4348bed0e3b662dd50aa341f67d10c | 5 + ...f4ea99e50a6ecd0d4134e3a77234172d13787ac4ea | 5 + ...83f58a3ca4eba895087316b31e598f0b05c978d87c | 5 + ...60599b281593098b0aad864231d756c3e9699029c1 | 5 + go.mod | 8 +- go.sum | 16 +- text/lru.go | 59 +- text/lru_test.go | 4 +- text/shaper.go | 237 ++- text/shaper_test.go | 91 +- text/text.go | 75 +- widget/editor.go | 506 ++++-- widget/editor_test.go | 34 +- widget/label.go | 57 +- widget/material/theme.go | 4 +- ...120fe7b82978ac0e5c16afb519beb080f73c470d3f | 4 + ...645d025ac15316f1210d4aa307cde333cc87fdad55 | 4 + .../FuzzEditorEditing/clusterIndexForCrash1 | 4 + .../FuzzEditorEditing/clusterIndexForCrash2 | 4 + widget/text_bench_test.go | 110 +- widget/text_test.go | 1075 ++++++++++-- 34 files changed, 3328 insertions(+), 2530 deletions(-) delete mode 100644 font/opentype/internal/shaping.go delete mode 100644 font/opentype/internal/shaping_test.go create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/2a7730fcbcc3550718b37415eb6104cf09159c7d756cc3530ccaa9007c0a2c06 create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/3fc3ee939f0df44719bee36a67df3aa3a562e2f2f1cfb665a650f60e7e0ab4e6 create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/594e4fda2e3462061d50b6a74279d7e3e486d8ac211e45049cfa4833257ef236 create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/6b452fd81f16c000dbe525077b6f4ba91b040119d82d55991d568014f4ba671e create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/6b5a1e9cd750a9aa8dafe11bc74e2377c0664f64bbf2ac8b1cdd0ce9f55461b3 create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/be56090b98f3c84da6ff9e857d5487d1a89309f7312ccde5ff8b94fcd29521eb create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/dda958d1b1bb9df71f81c575cb8900f97fc1c20e19de09fc0ffca8ff0c8d9e5a create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/de31ec6b1ac7797daefecf39baaace51fa4348bed0e3b662dd50aa341f67d10c create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/e79034d6c7a6ce74d7a689f4ea99e50a6ecd0d4134e3a77234172d13787ac4ea create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/f1f0611baadc20469863e483f58a3ca4eba895087316b31e598f0b05c978d87c create mode 100644 font/opentype/testdata/fuzz/FuzzLayout/f2ebea678c72f6c394d7f860599b281593098b0aad864231d756c3e9699029c1 create mode 100644 widget/testdata/fuzz/FuzzEditorEditing/18c534da60e6b61361786a120fe7b82978ac0e5c16afb519beb080f73c470d3f create mode 100644 widget/testdata/fuzz/FuzzEditorEditing/a489f2d9f9226d13b55846645d025ac15316f1210d4aa307cde333cc87fdad55 create mode 100644 widget/testdata/fuzz/FuzzEditorEditing/clusterIndexForCrash1 create mode 100644 widget/testdata/fuzz/FuzzEditorEditing/clusterIndexForCrash2 diff --git a/app/window.go b/app/window.go index dac0b9e6..90938945 100644 --- a/app/window.go +++ b/app/window.go @@ -143,7 +143,7 @@ func NewWindow(options ...Option) *Window { // Measure decoration height. deco := new(widget.Decorations) face, _ := opentype.Parse(goregular.TTF) - theme := material.NewTheme([]text.FontFace{{Font: text.Font{Typeface: "Go"}, Face: face}}) + theme := material.NewTheme([]text.FontFace{{Font: text.Font{Typeface: "Go"}, Face: face}}, &opentype.Shaper{}) decoStyle := material.Decorations(theme, deco, 0, "") gtx := layout.Context{ Ops: new(op.Ops), diff --git a/font/gofont/gofont.go b/font/gofont/gofont.go index a06efeb8..a2fc9bdb 100644 --- a/font/gofont/gofont.go +++ b/font/gofont/gofont.go @@ -33,7 +33,7 @@ var ( collection []text.FontFace ) -func Collection() []text.FontFace { +func Collection() ([]text.FontFace, text.FaceShaper) { once.Do(func() { register(text.Font{}, goregular.TTF) register(text.Font{Style: text.Italic}, goitalic.TTF) @@ -51,7 +51,7 @@ func Collection() []text.FontFace { n := len(collection) collection = collection[:n:n] }) - return collection + return collection, &opentype.Shaper{} } func register(fnt text.Font, ttf []byte) { diff --git a/font/opentype/internal/shaping.go b/font/opentype/internal/shaping.go deleted file mode 100644 index 66571d3a..00000000 --- a/font/opentype/internal/shaping.go @@ -1,521 +0,0 @@ -package internal - -import ( - "io" - - "gioui.org/io/system" - "gioui.org/text" - "github.com/benoitkugler/textlayout/language" - "github.com/gioui/uax/segment" - "github.com/gioui/uax/uax14" - "github.com/go-text/typesetting/di" - "github.com/go-text/typesetting/font" - "github.com/go-text/typesetting/shaping" - "golang.org/x/image/math/fixed" -) - -// computeGlyphClusters populates the Clusters field of a Layout. -// The order of the clusters is visual, meaning -// that the first cluster is the leftmost cluster displayed even when -// the cluster is part of RTL text. -func computeGlyphClusters(l *text.Layout) { - clusters := make([]text.GlyphCluster, 0, len(l.Glyphs)+1) - if len(l.Glyphs) < 1 { - if l.Runes.Count > 0 { - // Empty line corresponding to a newline character. - clusters = append(clusters, text.GlyphCluster{ - Runes: text.Range{ - Count: 1, - Offset: l.Runes.Offset, - }, - }) - } - l.Clusters = clusters - return - } - rtl := l.Direction == system.RTL - - // Check for trailing whitespace characters and synthesize - // GlyphClusters to represent them. - lastGlyph := l.Glyphs[len(l.Glyphs)-1] - if rtl { - lastGlyph = l.Glyphs[0] - } - trailingNewline := lastGlyph.ClusterIndex+lastGlyph.RuneCount < l.Runes.Count+l.Runes.Offset - newlineCluster := text.GlyphCluster{ - Runes: text.Range{ - Count: 1, - Offset: l.Runes.Count + l.Runes.Offset - 1, - }, - Glyphs: text.Range{ - Offset: len(l.Glyphs), - }, - } - - var ( - i int = 0 - inc int = 1 - runesProcessed int = 0 - glyphsProcessed int = 0 - ) - - if rtl { - i = len(l.Glyphs) - 1 - inc = -inc - glyphsProcessed = len(l.Glyphs) - 1 - newlineCluster.Glyphs.Offset = 0 - } - // Construct clusters from the line's glyphs. - for ; i < len(l.Glyphs) && i >= 0; i += inc { - g := l.Glyphs[i] - xAdv := g.XAdvance * fixed.Int26_6(inc) - for k := 0; k < g.GlyphCount-1 && k < len(l.Glyphs); k++ { - i += inc - xAdv += l.Glyphs[i].XAdvance * fixed.Int26_6(inc) - } - - startRune := runesProcessed - runeIncrement := g.RuneCount - startGlyph := glyphsProcessed - glyphIncrement := g.GlyphCount * inc - if rtl { - startGlyph = glyphsProcessed + glyphIncrement + 1 - } - clusters = append(clusters, text.GlyphCluster{ - Advance: xAdv, - Runes: text.Range{ - Count: g.RuneCount, - Offset: startRune + l.Runes.Offset, - }, - Glyphs: text.Range{ - Count: g.GlyphCount, - Offset: startGlyph, - }, - }) - runesProcessed += runeIncrement - glyphsProcessed += glyphIncrement - } - // Insert synthetic clusters at the right edge of the line. - if trailingNewline { - clusters = append(clusters, newlineCluster) - } - l.Clusters = clusters -} - -// langConfig describes the language and writing system of a body of text. -type langConfig struct { - // Language the text is written in. - language.Language - // Writing system used to represent the text. - language.Script - // Direction of the text, usually driven by the writing system. - di.Direction -} - -// mapRunesToClusterIndices returns a slice. Each index within that slice corresponds -// to an index within the runes input slice. The value stored at that index is the -// index of the glyph at the start of the corresponding glyph cluster shaped by -// harfbuzz. -func mapRunesToClusterIndices(runes []rune, glyphs []shaping.Glyph) []int { - mapping := make([]int, len(runes)) - glyphCursor := 0 - if len(runes) == 0 { - return nil - } - // If the final cluster values are lower than the starting ones, - // the text is RTL. - rtl := len(glyphs) > 0 && glyphs[len(glyphs)-1].ClusterIndex < glyphs[0].ClusterIndex - if rtl { - glyphCursor = len(glyphs) - 1 - } - for i := range runes { - for glyphCursor >= 0 && glyphCursor < len(glyphs) && - ((rtl && glyphs[glyphCursor].ClusterIndex <= i) || - (!rtl && glyphs[glyphCursor].ClusterIndex < i)) { - if rtl { - glyphCursor-- - } else { - glyphCursor++ - } - } - if rtl { - glyphCursor++ - } else if (glyphCursor >= 0 && glyphCursor < len(glyphs) && - glyphs[glyphCursor].ClusterIndex > i) || - (glyphCursor == len(glyphs) && len(glyphs) > 1) { - glyphCursor-- - targetClusterIndex := glyphs[glyphCursor].ClusterIndex - for glyphCursor-1 >= 0 && glyphs[glyphCursor-1].ClusterIndex == targetClusterIndex { - glyphCursor-- - } - } - if glyphCursor < 0 { - glyphCursor = 0 - } else if glyphCursor >= len(glyphs) { - glyphCursor = len(glyphs) - 1 - } - mapping[i] = glyphCursor - } - return mapping -} - -// inclusiveGlyphRange returns the inclusive range of runes and glyphs matching -// the provided start and breakAfter rune positions. -// runeToGlyph must be a valid mapping from the rune representation to the -// glyph reprsentation produced by mapRunesToClusterIndices. -// numGlyphs is the number of glyphs in the output representing the runes -// under consideration. -func inclusiveGlyphRange(start, breakAfter int, runeToGlyph []int, numGlyphs int) (glyphStart, glyphEnd int) { - rtl := runeToGlyph[len(runeToGlyph)-1] < runeToGlyph[0] - runeStart := start - runeEnd := breakAfter - if rtl { - glyphStart = runeToGlyph[runeEnd] - if runeStart-1 >= 0 { - glyphEnd = runeToGlyph[runeStart-1] - 1 - } else { - glyphEnd = numGlyphs - 1 - } - } else { - glyphStart = runeToGlyph[runeStart] - if runeEnd+1 < len(runeToGlyph) { - glyphEnd = runeToGlyph[runeEnd+1] - 1 - } else { - glyphEnd = numGlyphs - 1 - } - } - return -} - -// breakOption represets a location within the rune slice at which -// it may be safe to break a line of text. -type breakOption struct { - // breakAtRune is the index at which it is safe to break. - breakAtRune int - // penalty is the cost of breaking at this index. Negative - // penalties mean that the break is beneficial, and a penalty - // of uax14.PenaltyForMustBreak means a required break. - penalty int -} - -// getBreakOptions returns a slice of line break candidates for the -// text in the provided slice. -func getBreakOptions(text []rune) []breakOption { - // Collect options for breaking the lines in a slice. - var options []breakOption - const adjust = -1 - breaker := uax14.NewLineWrap() - segmenter := segment.NewSegmenter(breaker) - segmenter.InitFromSlice(text) - runeOffset := 0 - brokeAtEnd := false - for segmenter.Next() { - penalty, _ := segmenter.Penalties() - // Determine the indices of the breaking runes in the runes - // slice. Would be nice if the API provided this. - currentSegment := segmenter.Runes() - runeOffset += len(currentSegment) - - // Collect all break options. - options = append(options, breakOption{ - penalty: penalty, - breakAtRune: runeOffset + adjust, - }) - if options[len(options)-1].breakAtRune == len(text)-1 { - brokeAtEnd = true - } - } - if len(text) > 0 && !brokeAtEnd { - options = append(options, breakOption{ - penalty: uax14.PenaltyForMustBreak, - breakAtRune: len(text) - 1, - }) - } - return options -} - -type Shaper func(shaping.Input) (shaping.Output, error) - -// paragraph shapes a single paragraph of text, breaking it into multiple lines -// to fit within the provided maxWidth. -func paragraph(shaper Shaper, face font.Face, ppem fixed.Int26_6, maxWidth int, lc langConfig, paragraph []rune) ([]output, error) { - // TODO: handle splitting bidi text here - - // Shape the text. - input := toInput(face, ppem, lc, paragraph) - out, err := shaper(input) - if err != nil { - return nil, err - } - // Get a mapping from input runes to output glyphs. - runeToGlyph := mapRunesToClusterIndices(paragraph, out.Glyphs) - - // Fetch line break candidates. - breaks := getBreakOptions(paragraph) - - return lineWrap(out, input.Direction, paragraph, runeToGlyph, breaks, maxWidth), nil -} - -// shouldKeepSegmentOnLine decides whether the segment of text from the current -// end of the line to the provided breakOption should be kept on the current -// line. It should be called successively with each available breakOption, -// and the line should be broken (without keeping the current segment) -// whenever it returns false. -// -// The parameters require some explanation: -// - out - the shaping.Output that is being line-broken. -// - runeToGlyph - a mapping where accessing the slice at the index of a rune -// into out will yield the index of the first glyph corresponding to that rune. -// - lineStartRune - the index of the first rune in the line. -// - b - the line break candidate under consideration. -// - curLineWidth - the amount of space total in the current line. -// - curLineUsed - the amount of space in the current line that is already used. -// - nextLineWidth - the amount of space available on the next line. -// -// This function returns both a valid shaping.Output broken at b and a boolean -// indicating whether the returned output should be used. -func shouldKeepSegmentOnLine(out shaping.Output, runeToGlyph []int, lineStartRune int, b breakOption, curLineWidth, curLineUsed, nextLineWidth int) (candidateLine shaping.Output, keep bool) { - // Convert the break target to an inclusive index. - glyphStart, glyphEnd := inclusiveGlyphRange(lineStartRune, b.breakAtRune, runeToGlyph, len(out.Glyphs)) - - // Construct a line out of the inclusive glyph range. - candidateLine = out - candidateLine.Glyphs = candidateLine.Glyphs[glyphStart : glyphEnd+1] - candidateLine.RecomputeAdvance() - candidateAdvance := candidateLine.Advance.Ceil() - if candidateAdvance > curLineWidth && candidateAdvance-curLineUsed <= nextLineWidth { - // If it fits on the next line, put it there. - return candidateLine, false - } - - return candidateLine, true -} - -// lineWrap wraps the shaped glyphs of a paragraph to a particular max width. -func lineWrap(out shaping.Output, dir di.Direction, paragraph []rune, runeToGlyph []int, breaks []breakOption, maxWidth int) []output { - var outputs []output - if len(breaks) == 0 { - // Pass empty lines through as empty. - outputs = append(outputs, output{ - Shaped: out, - RuneRange: text.Range{ - Count: len(paragraph), - }, - }) - return outputs - } - - for i := 0; i < len(breaks); i++ { - b := breaks[i] - if b.breakAtRune+1 < len(runeToGlyph) { - // Check if this break is valid. - gIdx := runeToGlyph[b.breakAtRune] - g2Idx := runeToGlyph[b.breakAtRune+1] - cIdx := out.Glyphs[gIdx].ClusterIndex - c2Idx := out.Glyphs[g2Idx].ClusterIndex - if cIdx == c2Idx { - // This break is within a harfbuzz cluster, and is - // therefore invalid. - copy(breaks[i:], breaks[i+1:]) - breaks = breaks[:len(breaks)-1] - i-- - } - } - } - - start := 0 - runesProcessed := 0 - for i := 0; i < len(breaks); i++ { - b := breaks[i] - // Always keep the first segment on a line. - good, _ := shouldKeepSegmentOnLine(out, runeToGlyph, start, b, maxWidth, 0, maxWidth) - end := b.breakAtRune - innerLoop: - for k := i + 1; k < len(breaks); k++ { - bb := breaks[k] - candidate, ok := shouldKeepSegmentOnLine(out, runeToGlyph, start, bb, maxWidth, good.Advance.Ceil(), maxWidth) - if ok { - // Use this new, longer segment. - good = candidate - end = bb.breakAtRune - i++ - } else { - break innerLoop - } - } - numRunes := end - start + 1 - outputs = append(outputs, output{ - Shaped: good, - RuneRange: text.Range{ - Count: numRunes, - Offset: runesProcessed, - }, - }) - runesProcessed += numRunes - start = end + 1 - } - return outputs -} - -// output is a run of shaped text with metadata about its position -// within a text document. -type output struct { - Shaped shaping.Output - RuneRange text.Range -} - -func toSystemDirection(d di.Direction) system.TextDirection { - switch d { - case di.DirectionLTR: - return system.LTR - case di.DirectionRTL: - return system.RTL - } - return system.LTR -} - -// toGioGlyphs converts text shaper glyphs into the minimal representation -// that Gio needs. -func toGioGlyphs(in []shaping.Glyph) []text.Glyph { - out := make([]text.Glyph, 0, len(in)) - for _, g := range in { - out = append(out, text.Glyph{ - ID: g.GlyphID, - ClusterIndex: g.ClusterIndex, - RuneCount: g.RuneCount, - GlyphCount: g.GlyphCount, - XAdvance: g.XAdvance, - YAdvance: g.YAdvance, - XOffset: g.XOffset, - YOffset: g.YOffset, - }) - } - return out -} - -// ToLine converts the output into a text.Line -func (o output) ToLine() text.Line { - layout := text.Layout{ - Glyphs: toGioGlyphs(o.Shaped.Glyphs), - Runes: o.RuneRange, - Direction: toSystemDirection(o.Shaped.Direction), - } - return text.Line{ - Layout: layout, - Bounds: fixed.Rectangle26_6{ - Min: fixed.Point26_6{ - Y: -o.Shaped.LineBounds.Ascent, - }, - Max: fixed.Point26_6{ - X: o.Shaped.Advance, - Y: -o.Shaped.LineBounds.Ascent + o.Shaped.LineBounds.LineHeight(), - }, - }, - Width: o.Shaped.Advance, - Ascent: o.Shaped.LineBounds.Ascent, - Descent: -o.Shaped.LineBounds.Descent + o.Shaped.LineBounds.Gap, - } -} - -func mapDirection(d system.TextDirection) di.Direction { - switch d { - case system.LTR: - return di.DirectionLTR - case system.RTL: - return di.DirectionRTL - } - return di.DirectionLTR -} - -// Document shapes text using the given font, ppem, maximum line width, language, -// and sequence of runes. It returns a slice of lines corresponding to the txt, -// broken to fit within maxWidth and on paragraph boundaries. -func Document(shaper Shaper, face font.Face, ppem fixed.Int26_6, maxWidth int, lc system.Locale, txt io.RuneReader) []text.Line { - var ( - outputs []text.Line - startByte int - startRune int - paragraphText []rune - done bool - langs = make(map[language.Script]int) - ) - for !done { - var ( - bytes int - runes int - ) - newlineAdjust := 0 - paragraphLoop: - for r, sz, re := txt.ReadRune(); !done; r, sz, re = txt.ReadRune() { - if re != nil { - done = true - continue - } - paragraphText = append(paragraphText, r) - script := language.LookupScript(r) - langs[script]++ - bytes += sz - runes++ - if r == '\n' { - newlineAdjust = 1 - break paragraphLoop - } - } - var ( - primary language.Script - primaryTotal int - ) - for script, total := range langs { - if total > primaryTotal { - primary = script - primaryTotal = total - } - } - if lc.Language == "" { - lc.Language = "EN" - } - lcfg := langConfig{ - Language: language.NewLanguage(lc.Language), - Script: primary, - Direction: mapDirection(lc.Direction), - } - lines, _ := paragraph(shaper, face, ppem, maxWidth, lcfg, paragraphText[:len(paragraphText)-newlineAdjust]) - for i := range lines { - // Update the offsets of each paragraph to be correct within the - // whole document. - lines[i].RuneRange.Offset += startRune - // Update the cluster values to be rune indices within the entire - // document. - for k := range lines[i].Shaped.Glyphs { - lines[i].Shaped.Glyphs[k].ClusterIndex += startRune - } - outputs = append(outputs, lines[i].ToLine()) - } - // If there was a trailing newline update the byte counts to include - // it on the last line of the paragraph. - if newlineAdjust > 0 { - outputs[len(outputs)-1].Layout.Runes.Count += newlineAdjust - } - paragraphText = paragraphText[:0] - startByte += bytes - startRune += runes - } - for i := range outputs { - computeGlyphClusters(&outputs[i].Layout) - } - return outputs -} - -// toInput converts its parameters into a shaping.Input. -func toInput(face font.Face, ppem fixed.Int26_6, lc langConfig, runes []rune) shaping.Input { - var input shaping.Input - input.Direction = lc.Direction - input.Text = runes - input.Size = ppem - input.Face = face - input.Language = lc.Language - input.Script = lc.Script - input.RunStart = 0 - input.RunEnd = len(runes) - return input -} diff --git a/font/opentype/internal/shaping_test.go b/font/opentype/internal/shaping_test.go deleted file mode 100644 index ecbd89e0..00000000 --- a/font/opentype/internal/shaping_test.go @@ -1,1522 +0,0 @@ -package internal - -import ( - "bytes" - "reflect" - "sort" - "testing" - "testing/quick" - - "gioui.org/io/system" - "gioui.org/text" - "github.com/go-text/typesetting/di" - "github.com/go-text/typesetting/shaping" - "golang.org/x/image/math/fixed" -) - -// glyph returns a glyph with the given cluster. Its dimensions -// are a square sitting atop the baseline, with 10 units to a side. -func glyph(cluster int) shaping.Glyph { - return shaping.Glyph{ - XAdvance: fixed.I(10), - YAdvance: fixed.I(10), - Width: fixed.I(10), - Height: fixed.I(10), - YBearing: fixed.I(10), - ClusterIndex: cluster, - } -} - -func max(a, b int) int { - if a > b { - return a - } - return b -} -func min(a, b int) int { - if a < b { - return a - } - return b -} - -// glyphs returns a slice of glyphs with clusters from start to -// end. If start is greater than end, the glyphs will be returned -// with descending cluster values. -func glyphs(start, end int) []shaping.Glyph { - inc := 1 - if start > end { - inc = -inc - } - num := max(start, end) - min(start, end) + 1 - g := make([]shaping.Glyph, 0, num) - for i := start; i >= 0 && i <= max(start, end); i += inc { - g = append(g, glyph(i)) - } - return g -} - -func TestMapRunesToClusterIndices(t *testing.T) { - type testcase struct { - name string - runes []rune - glyphs []shaping.Glyph - expected []int - } - for _, tc := range []testcase{ - { - name: "simple", - runes: make([]rune, 5), - glyphs: []shaping.Glyph{ - glyph(0), - glyph(1), - glyph(2), - glyph(3), - glyph(4), - }, - expected: []int{0, 1, 2, 3, 4}, - }, - { - name: "simple rtl", - runes: make([]rune, 5), - glyphs: []shaping.Glyph{ - glyph(4), - glyph(3), - glyph(2), - glyph(1), - glyph(0), - }, - expected: []int{4, 3, 2, 1, 0}, - }, - { - name: "fused clusters", - runes: make([]rune, 5), - glyphs: []shaping.Glyph{ - glyph(0), - glyph(0), - glyph(2), - glyph(3), - glyph(3), - }, - expected: []int{0, 0, 2, 3, 3}, - }, - { - name: "fused clusters rtl", - runes: make([]rune, 5), - glyphs: []shaping.Glyph{ - glyph(3), - glyph(3), - glyph(2), - glyph(0), - glyph(0), - }, - expected: []int{3, 3, 2, 0, 0}, - }, - { - name: "ligatures", - runes: make([]rune, 5), - glyphs: []shaping.Glyph{ - glyph(0), - glyph(2), - glyph(3), - }, - expected: []int{0, 0, 1, 2, 2}, - }, - { - name: "ligatures rtl", - runes: make([]rune, 5), - glyphs: []shaping.Glyph{ - glyph(3), - glyph(2), - glyph(0), - }, - expected: []int{2, 2, 1, 0, 0}, - }, - { - name: "expansion", - runes: make([]rune, 5), - glyphs: []shaping.Glyph{ - glyph(0), - glyph(1), - glyph(1), - glyph(1), - glyph(2), - glyph(3), - glyph(4), - }, - expected: []int{0, 1, 4, 5, 6}, - }, - { - name: "expansion rtl", - runes: make([]rune, 5), - glyphs: []shaping.Glyph{ - glyph(4), - glyph(3), - glyph(2), - glyph(1), - glyph(1), - glyph(1), - glyph(0), - }, - expected: []int{6, 3, 2, 1, 0}, - }, - } { - t.Run(tc.name, func(t *testing.T) { - mapping := mapRunesToClusterIndices(tc.runes, tc.glyphs) - if !reflect.DeepEqual(tc.expected, mapping) { - t.Errorf("expected %v, got %v", tc.expected, mapping) - } - }) - } -} - -func TestInclusiveRange(t *testing.T) { - type testcase struct { - name string - // inputs - start int - breakAfter int - runeToGlyph []int - numGlyphs int - // expected outputs - gs, ge int - } - for _, tc := range []testcase{ - { - name: "simple at start", - numGlyphs: 5, - start: 0, - breakAfter: 2, - runeToGlyph: []int{0, 1, 2, 3, 4}, - gs: 0, - ge: 2, - }, - { - name: "simple in middle", - numGlyphs: 5, - start: 1, - breakAfter: 3, - runeToGlyph: []int{0, 1, 2, 3, 4}, - gs: 1, - ge: 3, - }, - { - name: "simple at end", - numGlyphs: 5, - start: 2, - breakAfter: 4, - runeToGlyph: []int{0, 1, 2, 3, 4}, - gs: 2, - ge: 4, - }, - { - name: "simple at start rtl", - numGlyphs: 5, - start: 0, - breakAfter: 2, - runeToGlyph: []int{4, 3, 2, 1, 0}, - gs: 2, - ge: 4, - }, - { - name: "simple in middle rtl", - numGlyphs: 5, - start: 1, - breakAfter: 3, - runeToGlyph: []int{4, 3, 2, 1, 0}, - gs: 1, - ge: 3, - }, - { - name: "simple at end rtl", - numGlyphs: 5, - start: 2, - breakAfter: 4, - runeToGlyph: []int{4, 3, 2, 1, 0}, - gs: 0, - ge: 2, - }, - { - name: "fused clusters at start", - numGlyphs: 5, - start: 0, - breakAfter: 1, - runeToGlyph: []int{0, 0, 2, 3, 3}, - gs: 0, - ge: 1, - }, - { - name: "fused clusters start and middle", - numGlyphs: 5, - start: 0, - breakAfter: 2, - runeToGlyph: []int{0, 0, 2, 3, 3}, - gs: 0, - ge: 2, - }, - { - name: "fused clusters middle and end", - numGlyphs: 5, - start: 2, - breakAfter: 4, - runeToGlyph: []int{0, 0, 2, 3, 3}, - gs: 2, - ge: 4, - }, - { - name: "fused clusters at end", - numGlyphs: 5, - start: 3, - breakAfter: 4, - runeToGlyph: []int{0, 0, 2, 3, 3}, - gs: 3, - ge: 4, - }, - { - name: "fused clusters at start rtl", - numGlyphs: 5, - start: 0, - breakAfter: 1, - runeToGlyph: []int{3, 3, 2, 0, 0}, - gs: 3, - ge: 4, - }, - { - name: "fused clusters start and middle rtl", - numGlyphs: 5, - start: 0, - breakAfter: 2, - runeToGlyph: []int{3, 3, 2, 0, 0}, - gs: 2, - ge: 4, - }, - { - name: "fused clusters middle and end rtl", - numGlyphs: 5, - start: 2, - breakAfter: 4, - runeToGlyph: []int{3, 3, 2, 0, 0}, - gs: 0, - ge: 2, - }, - { - name: "fused clusters at end rtl", - numGlyphs: 5, - start: 3, - breakAfter: 4, - runeToGlyph: []int{3, 3, 2, 0, 0}, - gs: 0, - ge: 1, - }, - { - name: "ligatures at start", - numGlyphs: 3, - start: 0, - breakAfter: 2, - runeToGlyph: []int{0, 0, 1, 2, 2}, - gs: 0, - ge: 1, - }, - { - name: "ligatures in middle", - numGlyphs: 3, - start: 2, - breakAfter: 2, - runeToGlyph: []int{0, 0, 1, 2, 2}, - gs: 1, - ge: 1, - }, - { - name: "ligatures at end", - numGlyphs: 3, - start: 2, - breakAfter: 4, - runeToGlyph: []int{0, 0, 1, 2, 2}, - gs: 1, - ge: 2, - }, - { - name: "ligatures at start rtl", - numGlyphs: 3, - start: 0, - breakAfter: 2, - runeToGlyph: []int{2, 2, 1, 0, 0}, - gs: 1, - ge: 2, - }, - { - name: "ligatures in middle rtl", - numGlyphs: 3, - start: 2, - breakAfter: 2, - runeToGlyph: []int{2, 2, 1, 0, 0}, - gs: 1, - ge: 1, - }, - { - name: "ligatures at end rtl", - numGlyphs: 3, - start: 2, - breakAfter: 4, - runeToGlyph: []int{2, 2, 1, 0, 0}, - gs: 0, - ge: 1, - }, - { - name: "expansion at start", - numGlyphs: 7, - start: 0, - breakAfter: 2, - runeToGlyph: []int{0, 1, 4, 5, 6}, - gs: 0, - ge: 4, - }, - { - name: "expansion in middle", - numGlyphs: 7, - start: 1, - breakAfter: 3, - runeToGlyph: []int{0, 1, 4, 5, 6}, - gs: 1, - ge: 5, - }, - { - name: "expansion at end", - numGlyphs: 7, - start: 2, - breakAfter: 4, - runeToGlyph: []int{0, 1, 4, 5, 6}, - gs: 4, - ge: 6, - }, - { - name: "expansion at start rtl", - numGlyphs: 7, - start: 0, - breakAfter: 2, - runeToGlyph: []int{6, 3, 2, 1, 0}, - gs: 2, - ge: 6, - }, - { - name: "expansion in middle rtl", - numGlyphs: 7, - start: 1, - breakAfter: 3, - runeToGlyph: []int{6, 3, 2, 1, 0}, - gs: 1, - ge: 5, - }, - { - name: "expansion at end rtl", - numGlyphs: 7, - start: 2, - breakAfter: 4, - runeToGlyph: []int{6, 3, 2, 1, 0}, - gs: 0, - ge: 2, - }, - } { - t.Run(tc.name, func(t *testing.T) { - gs, ge := inclusiveGlyphRange(tc.start, tc.breakAfter, tc.runeToGlyph, tc.numGlyphs) - if gs != tc.gs { - t.Errorf("glyphStart mismatch, got %d, expected %d", gs, tc.gs) - } - if ge != tc.ge { - t.Errorf("glyphEnd mismatch, got %d, expected %d", ge, tc.ge) - } - }) - } -} - -var ( - // Assume the simple case of 1:1:1 glyph:rune:byte for this input. - text1 = "text one is ltr" - shapedText1 = shaping.Output{ - Advance: fixed.I(10 * len([]rune(text1))), - LineBounds: shaping.Bounds{ - Ascent: fixed.I(10), - Descent: fixed.I(5), - // No line gap. - }, - GlyphBounds: shaping.Bounds{ - Ascent: fixed.I(10), - // No glyphs descend. - }, - Glyphs: glyphs(0, 14), - } - text1Trailing = text1 + " " - shapedText1Trailing = func() shaping.Output { - out := shapedText1 - out.Glyphs = append(out.Glyphs, glyph(len(out.Glyphs))) - out.RecalculateAll() - return out - }() - // Test M:N:O glyph:rune:byte for this input. - // The substring `lig` is shaped as a ligature. - // The substring `DROP` is not shaped at all. - text2 = "안П你 ligDROP 안П你 ligDROP" - shapedText2 = shaping.Output{ - // There are 11 glyphs shaped for this string. - Advance: fixed.I(10 * 11), - LineBounds: shaping.Bounds{ - Ascent: fixed.I(10), - Descent: fixed.I(5), - // No line gap. - }, - GlyphBounds: shaping.Bounds{ - Ascent: fixed.I(10), - // No glyphs descend. - }, - Glyphs: []shaping.Glyph{ - 0: glyph(0), // 안 - 4 bytes - 1: glyph(1), // П - 3 bytes - 2: glyph(2), // 你 - 4 bytes - 3: glyph(3), // <space> - 1 byte - 4: glyph(4), // lig - 3 runes, 3 bytes - // DROP - 4 runes, 4 bytes - 5: glyph(11), // <space> - 1 byte - 6: glyph(12), // 안 - 4 bytes - 7: glyph(13), // П - 3 bytes - 8: glyph(14), // 你 - 4 bytes - 9: glyph(15), // <space> - 1 byte - 10: glyph(16), // lig - 3 runes, 3 bytes - // DROP - 4 runes, 4 bytes - }, - } - // Test RTL languages. - text3 = "שלום أهلا שלום أهلا" - shapedText3 = shaping.Output{ - // There are 15 glyphs shaped for this string. - Advance: fixed.I(10 * 15), - LineBounds: shaping.Bounds{ - Ascent: fixed.I(10), - Descent: fixed.I(5), - // No line gap. - }, - GlyphBounds: shaping.Bounds{ - Ascent: fixed.I(10), - // No glyphs descend. - }, - Glyphs: []shaping.Glyph{ - 0: glyph(16), // LIGATURE of three runes: - // ا - 3 bytes - // ل - 3 bytes - // ه - 3 bytes - 1: glyph(15), // أ - 3 bytes - 2: glyph(14), // <space> - 1 byte - 3: glyph(13), // ם - 3 bytes - 4: glyph(12), // ו - 3 bytes - 5: glyph(11), // ל - 3 bytes - 6: glyph(10), // ש - 3 bytes - 7: glyph(9), // <space> - 1 byte - 8: glyph(6), // LIGATURE of three runes: - // ا - 3 bytes - // ل - 3 bytes - // ه - 3 bytes - 9: glyph(5), // أ - 3 bytes - 10: glyph(4), // <space> - 1 byte - 11: glyph(3), // ם - 3 bytes - 12: glyph(2), // ו - 3 bytes - 13: glyph(1), // ל - 3 bytes - 14: glyph(0), // ש - 3 bytes - }, - } -) - -// splitShapedAt splits a single shaped output into multiple. It splits -// on each provided glyph index in indices, with the index being the end of -// a slice range (so it's exclusive). You can think of the index as the -// first glyph of the next output. -func splitShapedAt(shaped shaping.Output, direction di.Direction, indices ...int) []shaping.Output { - numOut := len(indices) + 1 - outputs := make([]shaping.Output, 0, numOut) - start := 0 - for _, i := range indices { - newOut := shaped - newOut.Glyphs = newOut.Glyphs[start:i] - newOut.RecalculateAll() - outputs = append(outputs, newOut) - start = i - } - newOut := shaped - newOut.Glyphs = newOut.Glyphs[start:] - newOut.RecalculateAll() - outputs = append(outputs, newOut) - return outputs -} - -func TestEngineLineWrap(t *testing.T) { - type testcase struct { - name string - direction di.Direction - shaped shaping.Output - paragraph []rune - maxWidth int - expected []output - } - for _, tc := range []testcase{ - { - // This test case verifies that no line breaks occur if they are not - // necessary, and that the proper Offsets are reported in the output. - name: "all one line", - shaped: shapedText1, - direction: di.DirectionLTR, - paragraph: []rune(text1), - maxWidth: 1000, - expected: []output{ - { - RuneRange: text.Range{ - Count: len([]rune(text1)), - }, - Shaped: shapedText1, - }, - }, - }, - { - // This test case verifies that trailing whitespace characters on a - // line do not just disappear if it's the first line. - name: "trailing whitespace", - shaped: shapedText1Trailing, - direction: di.DirectionLTR, - paragraph: []rune(text1Trailing), - maxWidth: 1000, - expected: []output{ - { - RuneRange: text.Range{ - Count: len([]rune(text1)) + 1, - }, - Shaped: shapedText1Trailing, - }, - }, - }, - { - // This test case verifies that the line wrapper rejects line break - // candidates that would split a glyph cluster. - name: "reject mid-cluster line breaks", - shaped: shaping.Output{ - Advance: fixed.I(10 * 3), - LineBounds: shaping.Bounds{ - Ascent: fixed.I(10), - Descent: fixed.I(5), - // No line gap. - }, - GlyphBounds: shaping.Bounds{ - Ascent: fixed.I(10), - // No glyphs descend. - }, - Glyphs: []shaping.Glyph{ - simpleGlyph(0), - complexGlyph(1, 2, 2), - complexGlyph(1, 2, 2), - }, - }, - direction: di.DirectionLTR, - // This unicode data was discovered in a testing/quick failure - // for widget.Editor. It has the property that the middle two - // runes form a harfbuzz cluster but also have a legal UAX#14 - // segment break between them. - paragraph: []rune{0xa8e58, 0x3a4fd, 0x119dd}, - maxWidth: 20, - expected: []output{ - { - RuneRange: text.Range{ - Count: 1, - }, - Shaped: shaping.Output{ - Direction: di.DirectionLTR, - Advance: fixed.I(10), - LineBounds: shaping.Bounds{ - Ascent: fixed.I(10), - Descent: fixed.I(5), - }, - GlyphBounds: shaping.Bounds{ - Ascent: fixed.I(10), - }, - Glyphs: []shaping.Glyph{ - simpleGlyph(0), - }, - }, - }, - { - RuneRange: text.Range{ - Count: 2, - Offset: 1, - }, - Shaped: shaping.Output{ - Direction: di.DirectionLTR, - Advance: fixed.I(20), - LineBounds: shaping.Bounds{ - Ascent: fixed.I(10), - Descent: fixed.I(5), - }, - GlyphBounds: shaping.Bounds{ - Ascent: fixed.I(10), - }, - Glyphs: []shaping.Glyph{ - complexGlyph(1, 2, 2), - complexGlyph(1, 2, 2), - }, - }, - }, - }, - }, - { - // This test case verifies that line breaking does occur, and that - // all lines have proper offsets. - name: "line break on last word", - shaped: shapedText1, - direction: di.DirectionLTR, - paragraph: []rune(text1), - maxWidth: 120, - expected: []output{ - { - RuneRange: text.Range{ - Count: len([]rune(text1)) - 3, - }, - Shaped: splitShapedAt(shapedText1, di.DirectionLTR, 12)[0], - }, - { - RuneRange: text.Range{ - Offset: len([]rune(text1)) - 3, - Count: 3, - }, - Shaped: splitShapedAt(shapedText1, di.DirectionLTR, 12)[1], - }, - }, - }, - { - // This test case verifies that many line breaks still result in - // correct offsets. This test also ensures that leading whitespace - // is correctly hidden on lines after the first. - name: "line break several times", - shaped: shapedText1, - direction: di.DirectionLTR, - paragraph: []rune(text1), - maxWidth: 70, - expected: []output{ - { - RuneRange: text.Range{ - Count: 5, - }, - Shaped: splitShapedAt(shapedText1, di.DirectionLTR, 5)[0], - }, - { - RuneRange: text.Range{ - Offset: 5, - Count: 7, - }, - Shaped: splitShapedAt(shapedText1, di.DirectionLTR, 5, 12)[1], - }, - { - RuneRange: text.Range{ - Offset: 12, - Count: 3, - }, - Shaped: splitShapedAt(shapedText1, di.DirectionLTR, 12)[1], - }, - }, - }, - { - // This test case verifies baseline offset math for more complicated input. - name: "all one line 2", - shaped: shapedText2, - direction: di.DirectionLTR, - paragraph: []rune(text2), - maxWidth: 1000, - expected: []output{ - { - RuneRange: text.Range{ - Count: len([]rune(text2)), - }, - Shaped: shapedText2, - }, - }, - }, - { - // This test case verifies that offset accounting correctly handles complex - // input across line breaks. It is legal to line-break within words composed - // of more than one script, so this test expects that to occur. - name: "line break several times 2", - shaped: shapedText2, - direction: di.DirectionLTR, - paragraph: []rune(text2), - maxWidth: 40, - expected: []output{ - { - RuneRange: text.Range{ - Count: len([]rune("안П你 ")), - }, - Shaped: splitShapedAt(shapedText2, di.DirectionLTR, 4)[0], - }, - { - RuneRange: text.Range{ - Count: len([]rune("ligDROP 안П")), - Offset: len([]rune("안П你 ")), - }, - Shaped: splitShapedAt(shapedText2, di.DirectionLTR, 4, 8)[1], - }, - { - RuneRange: text.Range{ - Count: len([]rune("你 ligDROP")), - Offset: len([]rune("안П你 ligDROP 안П")), - }, - Shaped: splitShapedAt(shapedText2, di.DirectionLTR, 8, 11)[1], - }, - }, - }, - { - // This test case verifies baseline offset math for complex RTL input. - name: "all one line 3", - shaped: shapedText3, - direction: di.DirectionLTR, - paragraph: []rune(text3), - maxWidth: 1000, - expected: []output{ - { - RuneRange: text.Range{ - Count: len([]rune(text3)), - }, - Shaped: shapedText3, - }, - }, - }, - { - // This test case verifies line wrapping logic in RTL mode. - name: "line break once [RTL]", - shaped: shapedText3, - direction: di.DirectionRTL, - paragraph: []rune(text3), - maxWidth: 100, - expected: []output{ - { - RuneRange: text.Range{ - Count: len([]rune("שלום أهلا ")), - }, - Shaped: splitShapedAt(shapedText3, di.DirectionRTL, 7)[1], - }, - { - RuneRange: text.Range{ - Count: len([]rune("שלום أهلا")), - Offset: len([]rune("שלום أهلا ")), - }, - Shaped: splitShapedAt(shapedText3, di.DirectionRTL, 7)[0], - }, - }, - }, - { - // This test case verifies line wrapping logic in RTL mode. - name: "line break several times [RTL]", - shaped: shapedText3, - direction: di.DirectionRTL, - paragraph: []rune(text3), - maxWidth: 50, - expected: []output{ - { - RuneRange: text.Range{ - Count: len([]rune("שלום ")), - }, - Shaped: splitShapedAt(shapedText3, di.DirectionRTL, 10)[1], - }, - { - RuneRange: text.Range{ - Count: len([]rune("أهلا ")), - Offset: len([]rune("שלום ")), - }, - Shaped: splitShapedAt(shapedText3, di.DirectionRTL, 7, 10)[1], - }, - { - RuneRange: text.Range{ - Count: len([]rune("שלום ")), - Offset: len([]rune("שלום أهلا ")), - }, - Shaped: splitShapedAt(shapedText3, di.DirectionRTL, 2, 7)[1], - }, - { - RuneRange: text.Range{ - Count: len([]rune("أهلا")), - Offset: len([]rune("שלום أهلا שלום ")), - }, - Shaped: splitShapedAt(shapedText3, di.DirectionRTL, 2)[0], - }, - }, - }, - } { - t.Run(tc.name, func(t *testing.T) { - // Get a mapping from input runes to output glyphs. - runeToGlyph := mapRunesToClusterIndices(tc.paragraph, tc.shaped.Glyphs) - - // Fetch line break candidates. - breaks := getBreakOptions(tc.paragraph) - - outs := lineWrap(tc.shaped, tc.direction, tc.paragraph, runeToGlyph, breaks, tc.maxWidth) - if len(tc.expected) != len(outs) { - t.Errorf("expected %d lines, got %d", len(tc.expected), len(outs)) - } - for i := range tc.expected { - e := tc.expected[i] - o := outs[i] - lenE := len(e.Shaped.Glyphs) - lenO := len(o.Shaped.Glyphs) - if lenE != lenO { - t.Errorf("line %d: expected %d glyphs, got %d", i, lenE, lenO) - } else { - for k := range e.Shaped.Glyphs { - e := e.Shaped.Glyphs[k] - o := o.Shaped.Glyphs[k] - if !reflect.DeepEqual(e, o) { - t.Errorf("line %d: glyph mismatch at index %d, expected: %#v, got %#v", i, k, e, o) - } - } - } - if e.RuneRange != o.RuneRange { - t.Errorf("line %d: expected %#v offsets, got %#v", i, e.RuneRange, o.RuneRange) - } - if e.Shaped.Direction != o.Shaped.Direction { - t.Errorf("line %d: expected %v direction, got %v", i, e.Shaped.Direction, o.Shaped.Direction) - } - // Reduce the verbosity of the reflect mismatch since we already - // compared the glyphs. - e.Shaped.Glyphs = nil - o.Shaped.Glyphs = nil - if !reflect.DeepEqual(e.Shaped, o.Shaped) { - t.Errorf("line %d: expected: %#v, got %#v", i, e, o) - } - } - }) - } -} - -func TestEngineDocument(t *testing.T) { - const doc = `Rutrum quisque non tellus orci ac auctor augue. -At risus viverra adipiscing at.` - english := system.Locale{ - Language: "EN", - Direction: system.LTR, - } - docRunes := len([]rune(doc)) - - // Override the shaping engine with one that will return a simple - // square glyph info for each rune in the input. - shaper := func(in shaping.Input) (shaping.Output, error) { - o := shaping.Output{ - // TODO: ensure that this is either inclusive or exclusive - Glyphs: glyphs(in.RunStart, in.RunEnd), - } - o.RecalculateAll() - return o, nil - } - - lines := Document(shaper, nil, 10, 100, english, bytes.NewBufferString(doc)) - - lineRunes := 0 - for i, line := range lines { - t.Logf("Line %d: runeOffset %d, runes %d", - i, line.Layout.Runes.Offset, line.Layout.Runes.Count) - if line.Layout.Runes.Offset != lineRunes { - t.Errorf("expected line %d to start at byte %d, got %d", i, lineRunes, line.Layout.Runes.Offset) - } - lineRunes += line.Layout.Runes.Count - } - if lineRunes != docRunes { - t.Errorf("unexpected count: expected %d runes, got %d runes", - docRunes, lineRunes) - } -} - -// simpleGlyph returns a simple square glyph with the provided cluster -// value. -func simpleGlyph(cluster int) shaping.Glyph { - return complexGlyph(cluster, 1, 1) -} - -// ligatureGlyph returns a simple square glyph with the provided cluster -// value and number of runes. -func ligatureGlyph(cluster, runes int) shaping.Glyph { - return complexGlyph(cluster, runes, 1) -} - -// expansionGlyph returns a simple square glyph with the provided cluster -// value and number of glyphs. -func expansionGlyph(cluster, glyphs int) shaping.Glyph { - return complexGlyph(cluster, 1, glyphs) -} - -// complexGlyph returns a simple square glyph with the provided cluster -// value, number of associated runes, and number of glyphs in the cluster. -func complexGlyph(cluster, runes, glyphs int) shaping.Glyph { - return shaping.Glyph{ - Width: fixed.I(10), - Height: fixed.I(10), - XAdvance: fixed.I(10), - YAdvance: fixed.I(10), - YBearing: fixed.I(10), - ClusterIndex: cluster, - GlyphCount: glyphs, - RuneCount: runes, - } -} - -func simpleCluster(runeOffset, glyphOffset int, ltr bool) text.GlyphCluster { - g := text.GlyphCluster{ - Advance: fixed.I(10), - Runes: text.Range{ - Count: 1, - Offset: runeOffset, - }, - Glyphs: text.Range{ - Count: 1, - Offset: glyphOffset, - }, - } - if !ltr { - g.Advance = -g.Advance - } - return g -} - -func TestLayoutComputeClusters(t *testing.T) { - type testcase struct { - name string - line text.Layout - expected []text.GlyphCluster - } - for _, tc := range []testcase{ - { - name: "empty", - expected: []text.GlyphCluster{}, - }, - { - name: "just newline", - line: text.Layout{ - Direction: system.LTR, - Glyphs: toGioGlyphs([]shaping.Glyph{}), - Runes: text.Range{ - Count: 1, - }, - }, - expected: []text.GlyphCluster{ - { - Runes: text.Range{ - Count: 1, - }, - }, - }, - }, - { - name: "simple", - line: text.Layout{ - Direction: system.LTR, - Glyphs: toGioGlyphs([]shaping.Glyph{ - simpleGlyph(0), - simpleGlyph(1), - simpleGlyph(2), - }), - Runes: text.Range{ - Count: 3, - }, - }, - expected: []text.GlyphCluster{ - simpleCluster(0, 0, true), - simpleCluster(1, 1, true), - simpleCluster(2, 2, true), - }, - }, - { - name: "simple with newline", - line: text.Layout{ - Direction: system.LTR, - Glyphs: toGioGlyphs([]shaping.Glyph{ - simpleGlyph(0), - simpleGlyph(1), - simpleGlyph(2), - }), - Runes: text.Range{ - Count: 4, - }, - }, - expected: []text.GlyphCluster{ - simpleCluster(0, 0, true), - simpleCluster(1, 1, true), - simpleCluster(2, 2, true), - { - Runes: text.Range{ - Count: 1, - Offset: 3, - }, - Glyphs: text.Range{ - Offset: 3, - }, - }, - }, - }, - { - name: "ligature", - line: text.Layout{ - Direction: system.LTR, - Glyphs: toGioGlyphs([]shaping.Glyph{ - ligatureGlyph(0, 2), - simpleGlyph(2), - simpleGlyph(3), - }), - Runes: text.Range{ - Count: 4, - }, - }, - expected: []text.GlyphCluster{ - { - Advance: fixed.I(10), - Runes: text.Range{ - Count: 2, - }, - Glyphs: text.Range{ - Count: 1, - }, - }, - simpleCluster(2, 1, true), - simpleCluster(3, 2, true), - }, - }, - { - name: "ligature with newline", - line: text.Layout{ - Direction: system.LTR, - Glyphs: toGioGlyphs([]shaping.Glyph{ - ligatureGlyph(0, 2), - }), - Runes: text.Range{ - Count: 3, - }, - }, - expected: []text.GlyphCluster{ - { - Advance: fixed.I(10), - Runes: text.Range{ - Count: 2, - }, - Glyphs: text.Range{ - Count: 1, - }, - }, - { - Runes: text.Range{ - Count: 1, - Offset: 2, - }, - Glyphs: text.Range{ - Offset: 1, - }, - }, - }, - }, - { - name: "expansion", - line: text.Layout{ - Direction: system.LTR, - Glyphs: toGioGlyphs([]shaping.Glyph{ - expansionGlyph(0, 2), - expansionGlyph(0, 2), - simpleGlyph(1), - simpleGlyph(2), - }), - Runes: text.Range{ - Count: 3, - }, - }, - expected: []text.GlyphCluster{ - { - Advance: fixed.I(20), - Runes: text.Range{ - Count: 1, - }, - Glyphs: text.Range{ - Count: 2, - }, - }, - simpleCluster(1, 2, true), - simpleCluster(2, 3, true), - }, - }, - { - name: "deletion", - line: text.Layout{ - Direction: system.LTR, - Glyphs: toGioGlyphs([]shaping.Glyph{ - simpleGlyph(0), - ligatureGlyph(1, 2), - simpleGlyph(3), - simpleGlyph(4), - }), - Runes: text.Range{ - Count: 5, - }, - }, - expected: []text.GlyphCluster{ - simpleCluster(0, 0, true), - { - Advance: fixed.I(10), - Runes: text.Range{ - Count: 2, - Offset: 1, - }, - Glyphs: text.Range{ - Count: 1, - Offset: 1, - }, - }, - simpleCluster(3, 2, true), - simpleCluster(4, 3, true), - }, - }, - { - name: "simple rtl", - line: text.Layout{ - Direction: system.RTL, - Glyphs: toGioGlyphs([]shaping.Glyph{ - simpleGlyph(2), - simpleGlyph(1), - simpleGlyph(0), - }), - Runes: text.Range{ - Count: 3, - }, - }, - expected: []text.GlyphCluster{ - simpleCluster(0, 2, false), - simpleCluster(1, 1, false), - simpleCluster(2, 0, false), - }, - }, - { - name: "simple rtl with newline", - line: text.Layout{ - Direction: system.RTL, - Glyphs: toGioGlyphs([]shaping.Glyph{ - simpleGlyph(2), - simpleGlyph(1), - simpleGlyph(0), - }), - Runes: text.Range{ - Count: 4, - }, - }, - expected: []text.GlyphCluster{ - simpleCluster(0, 2, false), - simpleCluster(1, 1, false), - simpleCluster(2, 0, false), - { - Runes: text.Range{ - Count: 1, - Offset: 3, - }, - }, - }, - }, - { - name: "ligature rtl", - line: text.Layout{ - Direction: system.RTL, - Glyphs: toGioGlyphs([]shaping.Glyph{ - simpleGlyph(3), - simpleGlyph(2), - ligatureGlyph(0, 2), - }), - Runes: text.Range{ - Count: 4, - }, - }, - expected: []text.GlyphCluster{ - { - Advance: fixed.I(-10), - Runes: text.Range{ - Count: 2, - }, - Glyphs: text.Range{ - Count: 1, - Offset: 2, - }, - }, - simpleCluster(2, 1, false), - simpleCluster(3, 0, false), - }, - }, - { - name: "ligature rtl with newline", - line: text.Layout{ - Direction: system.RTL, - Glyphs: toGioGlyphs([]shaping.Glyph{ - ligatureGlyph(0, 2), - }), - Runes: text.Range{ - Count: 3, - }, - }, - expected: []text.GlyphCluster{ - { - Advance: fixed.I(-10), - Runes: text.Range{ - Count: 2, - }, - Glyphs: text.Range{ - Count: 1, - }, - }, - { - Runes: text.Range{ - Count: 1, - Offset: 2, - }, - }, - }, - }, - { - name: "expansion rtl", - line: text.Layout{ - Direction: system.RTL, - Glyphs: toGioGlyphs([]shaping.Glyph{ - simpleGlyph(2), - simpleGlyph(1), - expansionGlyph(0, 2), - expansionGlyph(0, 2), - }), - Runes: text.Range{ - Count: 3, - }, - }, - expected: []text.GlyphCluster{ - { - Advance: fixed.I(-20), - Runes: text.Range{ - Count: 1, - }, - Glyphs: text.Range{ - Count: 2, - Offset: 2, - }, - }, - simpleCluster(1, 1, false), - simpleCluster(2, 0, false), - }, - }, - { - name: "deletion rtl", - line: text.Layout{ - Direction: system.RTL, - Glyphs: toGioGlyphs([]shaping.Glyph{ - simpleGlyph(4), - simpleGlyph(3), - ligatureGlyph(1, 2), - simpleGlyph(0), - }), - Runes: text.Range{ - Count: 5, - }, - }, - expected: []text.GlyphCluster{ - simpleCluster(0, 3, false), - { - Advance: fixed.I(-10), - Runes: text.Range{ - Count: 2, - Offset: 1, - }, - Glyphs: text.Range{ - Count: 1, - Offset: 2, - }, - }, - simpleCluster(3, 1, false), - simpleCluster(4, 0, false), - }, - }, - } { - t.Run(tc.name, func(t *testing.T) { - computeGlyphClusters(&tc.line) - actual := tc.line.Clusters - if !reflect.DeepEqual(actual, tc.expected) { - t.Errorf("expected %v, got %v", tc.expected, actual) - } - }) - } -} - -func TestGetBreakOptions(t *testing.T) { - if err := quick.Check(func(runes []rune) bool { - options := getBreakOptions(runes) - // Ensure breaks are in valid range. - for _, o := range options { - if o.breakAtRune < 0 || o.breakAtRune > len(runes)-1 { - return false - } - } - // Ensure breaks are sorted. - if !sort.SliceIsSorted(options, func(i, j int) bool { - return options[i].breakAtRune < options[j].breakAtRune - }) { - return false - } - - // Ensure breaks are unique. - m := make([]bool, len(runes)) - for _, o := range options { - if m[o.breakAtRune] { - return false - } else { - m[o.breakAtRune] = true - } - } - - return true - }, nil); err != nil { - t.Errorf("generated invalid break options: %v", err) - } -} - -func TestLayoutSlice(t *testing.T) { - type testcase struct { - name string - in text.Layout - expected text.Layout - start, end int - } - - ltrGlyphs := toGioGlyphs([]shaping.Glyph{ - simpleGlyph(0), - complexGlyph(1, 2, 2), - complexGlyph(1, 2, 2), - simpleGlyph(3), - simpleGlyph(4), - simpleGlyph(5), - ligatureGlyph(6, 3), - simpleGlyph(9), - simpleGlyph(10), - }) - rtlGlyphs := toGioGlyphs([]shaping.Glyph{ - simpleGlyph(10), - simpleGlyph(9), - ligatureGlyph(6, 3), - simpleGlyph(5), - simpleGlyph(4), - simpleGlyph(3), - complexGlyph(1, 2, 2), - complexGlyph(1, 2, 2), - simpleGlyph(0), - }) - - for _, tc := range []testcase{ - { - name: "ltr", - in: func() text.Layout { - l := text.Layout{ - Glyphs: ltrGlyphs, - Direction: system.LTR, - Runes: text.Range{ - Count: 11, - }, - } - computeGlyphClusters(&l) - return l - }(), - expected: func() text.Layout { - l := text.Layout{ - Glyphs: ltrGlyphs[5:], - Direction: system.LTR, - Runes: text.Range{ - Count: 6, - Offset: 5, - }, - } - return l - }(), - start: 4, - end: 8, - }, - { - name: "ltr different range", - in: func() text.Layout { - l := text.Layout{ - Glyphs: ltrGlyphs, - Direction: system.LTR, - Runes: text.Range{ - Count: 11, - }, - } - computeGlyphClusters(&l) - return l - }(), - expected: func() text.Layout { - l := text.Layout{ - Glyphs: ltrGlyphs[3:7], - Direction: system.LTR, - Runes: text.Range{ - Count: 6, - Offset: 3, - }, - } - return l - }(), - start: 2, - end: 6, - }, - { - name: "ltr zero len", - in: func() text.Layout { - l := text.Layout{ - Glyphs: ltrGlyphs, - Direction: system.LTR, - Runes: text.Range{ - Count: 11, - }, - } - computeGlyphClusters(&l) - return l - }(), - expected: text.Layout{}, - start: 0, - end: 0, - }, - { - name: "rtl", - in: func() text.Layout { - l := text.Layout{ - Glyphs: rtlGlyphs, - Direction: system.RTL, - Runes: text.Range{ - Count: 11, - }, - } - computeGlyphClusters(&l) - return l - }(), - expected: func() text.Layout { - l := text.Layout{ - Glyphs: rtlGlyphs[:4], - Direction: system.RTL, - Runes: text.Range{ - Count: 6, - Offset: 5, - }, - } - return l - }(), - start: 4, - end: 8, - }, - } { - t.Run(tc.name, func(t *testing.T) { - out := tc.in.Slice(tc.start, tc.end) - if len(out.Glyphs) != len(tc.expected.Glyphs) { - t.Errorf("expected %v glyphs, got %v", len(tc.expected.Glyphs), len(out.Glyphs)) - } - if len(out.Clusters) != len(tc.expected.Clusters) { - t.Errorf("expected %v clusters, got %v", len(tc.expected.Clusters), len(out.Clusters)) - } - if out.Runes != tc.expected.Runes { - t.Errorf("expected %#+v, got %#+v", tc.expected.Runes, out.Runes) - } - if out.Direction != tc.expected.Direction { - t.Errorf("expected %#+v, got %#+v", tc.expected.Direction, out.Direction) - } - }) - } -} diff --git a/font/opentype/opentype.go b/font/opentype/opentype.go index 87d76507..ecd4d220 100644 --- a/font/opentype/opentype.go +++ b/font/opentype/opentype.go @@ -7,72 +7,272 @@ package opentype import ( "bytes" "fmt" - "image" - "io" "github.com/benoitkugler/textlayout/fonts" "github.com/benoitkugler/textlayout/fonts/truetype" - "github.com/benoitkugler/textlayout/harfbuzz" + "github.com/benoitkugler/textlayout/language" + "github.com/go-text/typesetting/di" + "github.com/go-text/typesetting/font" "github.com/go-text/typesetting/shaping" - "golang.org/x/image/font" + "golang.org/x/exp/slices" "golang.org/x/image/math/fixed" + "golang.org/x/text/unicode/bidi" "gioui.org/f32" - "gioui.org/font/opentype/internal" "gioui.org/io/system" "gioui.org/op" "gioui.org/op/clip" "gioui.org/text" ) -// Font implements the text.Shaper interface using a rich text -// shaping engine. -type Font struct { - font *truetype.Font +type Face font.Face + +// Shaper implements the shaping and line-wrapping of opentype fonts. +type Shaper struct { + shaper shaping.HarfbuzzShaper + wrapper shaping.LineWrapper + bidiParagraph bidi.Paragraph + fallbackFaces []font.Face + + // Scratch buffers used to avoid re-allocating slices during routine internal + // shaping operations. + splitScratch1, splitScratch2 []shaping.Input + outScratchBuf []shaping.Output } +var _ text.FaceShaper = (*Shaper)(nil) + // Parse constructs a Font from source bytes. -func Parse(src []byte) (*Font, error) { +func Parse(src []byte) (Face, error) { face, err := truetype.Parse(bytes.NewReader(src)) if err != nil { return nil, fmt.Errorf("failed parsing truetype font: %w", err) } - return &Font{ - font: face, - }, nil + return face, nil } -func (f *Font) Layout(ppem fixed.Int26_6, maxWidth int, lc system.Locale, txt io.RuneReader) ([]text.Line, error) { - return internal.Document(shaping.Shape, f.font, ppem, maxWidth, lc, txt), nil +func scriptToDir(script language.Script, documentDir di.Direction) di.Direction { + // List of RTL scripts sourced from: + // https://en.wikipedia.org/wiki/Right-to-left_script#List_of_RTL_scripts + switch script { + case language.Arabic: + fallthrough + case language.Hebrew: + fallthrough + case language.Thaana: + fallthrough + case language.Syriac: + fallthrough + case language.Mandaic: + fallthrough + case language.Samaritan: + fallthrough + case language.Mende_Kikakui: + fallthrough + case language.Nko: + fallthrough + case language.Adlam: + fallthrough + case language.Hanifi_Rohingya: + fallthrough + case language.Yezidi: + return di.DirectionRTL + case language.Common: + return documentDir + default: + return di.DirectionLTR + } } -func (f *Font) Shape(ppem fixed.Int26_6, str text.Layout) clip.PathSpec { - return textPath(ppem, f, str) +// splitByScriptAndDirection divides the inputs into new, smaller inputs on script boundaries +// and correctly sets the text direction per-script. This is not as correct as the unicode +// bidirectional text algorithm, but is simple and good enough as a starting point. It will +// use buf as the backing memory for the returned slice if buf is non-nil. +func splitByScript(inputs []shaping.Input, documentDir di.Direction, buf []shaping.Input) []shaping.Input { + var splitInputs []shaping.Input + if buf == nil { + splitInputs = make([]shaping.Input, 0, len(inputs)) + } else { + splitInputs = buf + } + for _, input := range inputs { + currentInput := input + if input.RunStart == input.RunEnd { + return []shaping.Input{input} + } + firstNonCommonRune := input.RunStart + for i := firstNonCommonRune; i < input.RunEnd; i++ { + if language.LookupScript(input.Text[i]) != language.Common { + firstNonCommonRune = i + break + } + } + // Adapted from github.com/go-text/typesetting/shaping.SplitByFace() under the terms of the + // UNLICENSE. + currentInput.Script = language.LookupScript(input.Text[firstNonCommonRune]) + for i := firstNonCommonRune + 1; i < input.RunEnd; i++ { + r := input.Text[i] + runeScript := language.LookupScript(r) + + if runeScript == language.Common || runeScript == currentInput.Script { + continue + } + + if i != input.RunStart { + currentInput.RunEnd = i + splitInputs = append(splitInputs, currentInput) + } + + currentInput = input + currentInput.RunStart = i + currentInput.Script = runeScript + // In the future, it may make sense to try to guess the language of the text here as well, + // but this is a complex process. + } + // close and add the last input + currentInput.RunEnd = input.RunEnd + splitInputs = append(splitInputs, currentInput) + } + + return splitInputs } -func (f *Font) Metrics(ppem fixed.Int26_6) font.Metrics { - metrics := font.Metrics{} - font := harfbuzz.NewFont(f.font) - font.XScale = int32(ppem.Ceil()) << 6 - font.YScale = font.XScale - // Use any horizontal direction. - fontExtents := font.ExtentsForDirection(harfbuzz.LeftToRight) - ascender := fixed.I(int(fontExtents.Ascender * 64)) - descender := fixed.I(int(fontExtents.Descender * 64)) - gap := fixed.I(int(fontExtents.LineGap * 64)) - metrics.Height = ascender + descender + gap - metrics.Ascent = ascender - metrics.Descent = descender - // These three are not readily available. - // TODO(whereswaldon): figure out how to get these values. - metrics.XHeight = ascender - metrics.CapHeight = ascender - metrics.CaretSlope = image.Pt(0, 1) +func (s *Shaper) splitBidi(input shaping.Input) []shaping.Input { + var splitInputs []shaping.Input + if input.Direction.Axis() != di.Horizontal || input.RunStart == input.RunEnd { + return []shaping.Input{input} + } + def := bidi.LeftToRight + if input.Direction.Progression() == di.TowardTopLeft { + def = bidi.RightToLeft + } + s.bidiParagraph.SetString(string(input.Text), bidi.DefaultDirection(def)) + out, err := s.bidiParagraph.Order() + if err != nil { + return []shaping.Input{input} + } + for i := 0; i < out.NumRuns(); i++ { + currentInput := input + run := out.Run(i) + dir := run.Direction() + _, endRune := run.Pos() + currentInput.RunEnd = endRune + 1 + if dir == bidi.RightToLeft { + currentInput.Direction = di.DirectionRTL + } else { + currentInput.Direction = di.DirectionLTR + } + splitInputs = append(splitInputs, currentInput) + input.RunStart = currentInput.RunEnd + } + return splitInputs +} - return metrics +// splitByFaces divides the inputs by font coverage in the provided faces. It will use the slice provided in buf +// as the backing storage of the returned slice if buf is non-nil. +func (s *Shaper) splitByFaces(inputs []shaping.Input, faces []font.Face, buf []shaping.Input) []shaping.Input { + var split []shaping.Input + if buf == nil { + split = make([]shaping.Input, 0, len(inputs)) + } else { + split = buf + } + for _, input := range inputs { + split = append(split, shaping.SplitByFontGlyphs(input, faces)...) + } + return split } -func textPath(ppem fixed.Int26_6, font *Font, str text.Layout) clip.PathSpec { +// shapeText invokes the text shaper and returns the raw text data in the shaper's native +// format. It does not wrap lines. +func (s *Shaper) shapeText(faces []text.Face, ppem fixed.Int26_6, lc system.Locale, txt []rune) []shaping.Output { + if len(faces) < 1 { + return nil + } + if needed := len(faces) - len(s.fallbackFaces); needed > 0 { + s.fallbackFaces = slices.Grow(s.fallbackFaces, needed) + } + s.fallbackFaces = s.fallbackFaces[:0] + // Unpack fallback fonts into the proper type. + for i := range faces { + if asTTF, ok := faces[i].(Face); ok { + s.fallbackFaces = append(s.fallbackFaces, asTTF) + } + } + lcfg := langConfig{ + Language: language.NewLanguage(lc.Language), + Direction: mapDirection(lc.Direction), + } + // Create an initial input. + input := toInput(s.fallbackFaces[0], ppem, lcfg, txt) + // Break input on font glyph coverage. + inputs := s.splitBidi(input) + inputs = s.splitByFaces(inputs, s.fallbackFaces, s.splitScratch1[:0]) + inputs = splitByScript(inputs, lcfg.Direction, s.splitScratch2[:0]) + // Shape all inputs. + if needed := len(inputs) - len(s.outScratchBuf); needed > 0 { + s.outScratchBuf = slices.Grow(s.outScratchBuf, needed) + } + s.outScratchBuf = s.outScratchBuf[:len(inputs)] + for i := range inputs { + s.outScratchBuf[i] = s.shaper.Shape(inputs[i]) + } + return s.outScratchBuf +} + +// shapeAndWrapText invokes the text shaper and returns wrapped lines in the shaper's native format. +func (s *Shaper) shapeAndWrapText(faces []text.Face, ppem fixed.Int26_6, maxWidth int, lc system.Locale, txt []rune) []shaping.Line { + // Wrap outputs into lines. + return s.wrapper.WrapParagraph(maxWidth, txt, s.shapeText(faces, ppem, lc, txt)...) +} + +// replaceControlCharacters replaces problematic unicode +// code points with spaces to ensure proper rune accounting. +func replaceControlCharacters(in []rune) []rune { + for i, r := range in { + switch r { + // ASCII File separator. + case '\u001C': + // ASCII Group separator. + case '\u001D': + // ASCII Record separator. + case '\u001E': + case '\r': + case '\n': + // Unicode "next line" character. + case '\u0085': + // Unicode "paragraph separator". + case '\u2029': + default: + continue + } + in[i] = ' ' + } + return in +} + +// Layout shapes and wraps the text, and returns the result in Gio's shaped text format. +func (s *Shaper) Layout(faces []text.Face, ppem fixed.Int26_6, maxWidth int, lc system.Locale, txt []rune) []text.Line { + lines := s.shapeAndWrapText(faces, ppem, maxWidth, lc, replaceControlCharacters(txt)) + // Convert to text.Lines. + textLines := make([]text.Line, len(lines)) + for i := range lines { + textLines[i] = ToLine(lines[i], lc.Direction) + } + return textLines +} + +// Shape converts the provided line into a path. It trims the output glyphs to only +// contain glyphs from the runs between startRun and endRun (inclusive), with the +// actual start/end positions within those runs determined by startCluster and +// endCluster, respectively. startCluster and endCluster are permitted to be equal +// to the length of their respective Clusters slices. +func (s *Shaper) Shape(ppem fixed.Int26_6, str text.Line, startRun, startGlyph, endRun, endGlyph int) clip.PathSpec { + var startPosition, endPosition int + if len(str.Runs) > 0 {
startCluster/endCluster from the comment don't refer to any argument. Also, this is public API, so it would be nice to have an indexing scheme with only one start index and one end index.
+ startPosition = str.Runs[startRun].VisualPosition + endPosition = str.Runs[endRun].VisualPosition + } var lastPos f32.Point var builder clip.Path ops := new(op.Ops) @@ -81,58 +281,305 @@ func textPath(ppem fixed.Int26_6, font *Font, str text.Layout) clip.PathSpec { rune := 0 ppemInt := ppem.Round() ppem16 := uint16(ppemInt) - scaleFactor := float32(ppemInt) / float32(font.font.Upem()) - for _, g := range str.Glyphs { - advance := g.XAdvance - outline, ok := font.font.GlyphData(g.ID, ppem16, ppem16).(fonts.GlyphOutline) + // Runs are ordered logically. Iterate them visually based on the line's + // overall direction. + for visualPosition, logicalIndex := range str.VisualOrder { + if visualPosition < startPosition || visualPosition > endPosition { + continue + } + run := str.Runs[logicalIndex] + asFont, ok := run.Face.(font.Face) if !ok { continue } - // Move to glyph position. - pos := f32.Point{ - X: float32(x)/64 - float32(g.XOffset)/64, - Y: -float32(g.YOffset) / 64, - } - builder.Move(pos.Sub(lastPos)) - lastPos = pos - var lastArg f32.Point - - // Convert sfnt.Segments to relative segments. - for _, fseg := range outline.Segments { - nargs := 1 - switch fseg.Op { - case fonts.SegmentOpQuadTo: - nargs = 2 - case fonts.SegmentOpCubeTo: - nargs = 3 + firstGlyph := 0 + lastGlyph := len(run.Glyphs) + if logicalIndex == startRun { + firstGlyph = startGlyph + } + if logicalIndex == endRun { + lastGlyph = endGlyph + if lastGlyph < len(run.Glyphs) { + lastGlyph++ + } + } + for _, g := range run.Glyphs[firstGlyph:lastGlyph] { + scaleFactor := float32(ppemInt) / float32(asFont.Upem()) + advance := g.XAdvance + outline, ok := asFont.GlyphData(g.ID, ppem16, ppem16).(fonts.GlyphOutline) + if !ok { + continue + } + // Move to glyph position. + pos := f32.Point{ + X: float32(x)/64 - float32(g.XOffset)/64, + Y: -float32(g.YOffset) / 64, } - var args [3]f32.Point - for i := 0; i < nargs; i++ { - a := f32.Point{ - X: fseg.Args[i].X * scaleFactor, - Y: -fseg.Args[i].Y * scaleFactor, + builder.Move(pos.Sub(lastPos)) + lastPos = pos + var lastArg f32.Point + + // Convert fonts.Segments to relative segments. + for _, fseg := range outline.Segments { + nargs := 1 + switch fseg.Op { + case fonts.SegmentOpQuadTo: + nargs = 2 + case fonts.SegmentOpCubeTo: + nargs = 3 } - args[i] = a.Sub(lastArg) - if i == nargs-1 { - lastArg = a + var args [3]f32.Point + for i := 0; i < nargs; i++ { + a := f32.Point{ + X: fseg.Args[i].X * scaleFactor, + Y: -fseg.Args[i].Y * scaleFactor, + } + args[i] = a.Sub(lastArg) + if i == nargs-1 { + lastArg = a + } + } + switch fseg.Op { + case fonts.SegmentOpMoveTo: + builder.Move(args[0]) + case fonts.SegmentOpLineTo: + builder.Line(args[0]) + case fonts.SegmentOpQuadTo: + builder.Quad(args[0], args[1]) + case fonts.SegmentOpCubeTo: + builder.Cube(args[0], args[1], args[2]) + default: + panic("unsupported segment op") } } - switch fseg.Op { - case fonts.SegmentOpMoveTo: - builder.Move(args[0]) - case fonts.SegmentOpLineTo: - builder.Line(args[0]) - case fonts.SegmentOpQuadTo: - builder.Quad(args[0], args[1]) - case fonts.SegmentOpCubeTo: - builder.Cube(args[0], args[1], args[2]) - default: - panic("unsupported segment op") - } + lastPos = lastPos.Add(lastArg) + x += advance + rune++ } - lastPos = lastPos.Add(lastArg) - x += advance - rune++ } return builder.End() } + +// langConfig describes the language and writing system of a body of text. +type langConfig struct { + // Language the text is written in. + language.Language + // Writing system used to represent the text. + language.Script + // Direction of the text, usually driven by the writing system. + di.Direction +} + +// toInput converts its parameters into a shaping.Input. +func toInput(face font.Face, ppem fixed.Int26_6, lc langConfig, runes []rune) shaping.Input { + var input shaping.Input + input.Direction = lc.Direction + input.Text = runes + input.Size = ppem + input.Face = face + input.Language = lc.Language + input.Script = lc.Script + input.RunStart = 0 + input.RunEnd = len(runes) + return input +} + +func mapDirection(d system.TextDirection) di.Direction { + switch d { + case system.LTR: + return di.DirectionLTR + case system.RTL: + return di.DirectionRTL + } + return di.DirectionLTR +} + +func unmapDirection(d di.Direction) system.TextDirection { + switch d { + case di.DirectionLTR: + return system.LTR + case di.DirectionRTL: + return system.RTL + } + return system.LTR +} + +// toGioGlyphs converts text shaper glyphs into the minimal representation +// that Gio needs. +func toGioGlyphs(in []shaping.Glyph, face Face) []text.Glyph { + out := make([]text.Glyph, 0, len(in)) + for _, g := range in { + out = append(out, text.Glyph{ + ID: g.GlyphID, + ClusterIndex: g.ClusterIndex, + RuneCount: g.RuneCount, + GlyphCount: g.GlyphCount, + XAdvance: g.XAdvance, + YAdvance: g.YAdvance, + XOffset: g.XOffset, + YOffset: g.YOffset, + }) + } + return out +} + +// ToLine converts the output into a text.Line with the provided dominant text direction. +func ToLine(o shaping.Line, dir system.TextDirection) text.Line { + if len(o) < 1 { + return text.Line{}
Unexport. Not used outside of the package and exposes go-text API.
+ } + line := text.Line{ + Runs: make([]text.Layout, len(o)), + Direction: dir, + } + for i := range o { + run := o[i] + asTTF := run.Face.(font.Face) + line.Runs[i] = text.Layout{ + Glyphs: toGioGlyphs(run.Glyphs, asTTF), + Runes: text.Range{ + Count: run.Runes.Count, + Offset: line.RuneCount, + }, + Direction: unmapDirection(run.Direction), + Face: run.Face, + Advance: run.Advance, + } + line.RuneCount += run.Runes.Count + computeGlyphClusters(&line.Runs[i]) + if line.Bounds.Min.Y > -run.LineBounds.Ascent { + line.Bounds.Min.Y = -run.LineBounds.Ascent + } + if line.Bounds.Max.Y < -run.LineBounds.Ascent+run.LineBounds.LineHeight() { + line.Bounds.Max.Y = -run.LineBounds.Ascent + run.LineBounds.LineHeight() + } + line.Bounds.Max.X += run.Advance + line.Width += run.Advance + if line.Ascent < run.LineBounds.Ascent { + line.Ascent = run.LineBounds.Ascent + } + if line.Descent < -run.LineBounds.Descent+run.LineBounds.Gap { + line.Descent = -run.LineBounds.Descent + run.LineBounds.Gap + } + } + computeVisualOrder(&line) + return line +} + +// computeVisualOrder will populate the Line's VisualOrder field and the +// VisualPosition field of each element in Runs. +func computeVisualOrder(l *text.Line) { + l.VisualOrder = make([]int, len(l.Runs)) + const none = -1 + bidiRangeStart := none + + // visPos returns the visual position for an individual logically-indexed + // run in this line, taking only the line's overall text direction into + // account. + visPos := func(logicalIndex int) int { + if l.Direction.Progression() == system.TowardOrigin { + return len(l.Runs) - 1 - logicalIndex + } + return logicalIndex + } + + // resolveBidi populated the line's VisualOrder fields for the elements in the + // half-open range [bidiRangeStart:bidiRangeEnd) indicating that those elements + // should be displayed in reverse-visual order. + resolveBidi := func(bidiRangeStart, bidiRangeEnd int) { + firstVisual := bidiRangeEnd - 1 + // Just found the end of a bidi range. + for startIdx := bidiRangeStart; startIdx < bidiRangeEnd; startIdx++ { + pos := visPos(firstVisual) + l.Runs[startIdx].VisualPosition = pos + l.VisualOrder[pos] = startIdx + firstVisual-- + } + bidiRangeStart = none + } + for runIdx, run := range l.Runs { + if run.Direction.Progression() != l.Direction.Progression() { + if bidiRangeStart == none { + bidiRangeStart = runIdx + } + continue + } else if bidiRangeStart != none { + // Just found the end of a bidi range. + resolveBidi(bidiRangeStart, runIdx) + bidiRangeStart = none + } + pos := visPos(runIdx) + l.Runs[runIdx].VisualPosition = pos + l.VisualOrder[pos] = runIdx + } + if bidiRangeStart != none { + // We ended iteration within a bidi segment, resolve it. + resolveBidi(bidiRangeStart, len(l.Runs)) + } +} + +// computeGlyphClusters populates the Clusters field of a Layout. +// The order of the clusters is logical, meaning +// that the first cluster represents the first runes in the layout, +// not the first glyphs displayed. +func computeGlyphClusters(l *text.Layout) { + clusters := make([]text.GlyphCluster, 0, len(l.Glyphs)+1) + if len(l.Glyphs) < 1 { + if l.Runes.Count > 0 { + // Empty line corresponding to a newline character. + clusters = append(clusters, text.GlyphCluster{ + Runes: text.Range{ + Count: 1, + Offset: l.Runes.Offset, + }, + }) + } + l.Clusters = clusters + return + } + rtl := l.Direction == system.RTL + + var ( + i int = 0 + inc int = 1 + runesProcessed int = 0 + glyphsProcessed int = 0 + ) + + if rtl { + i = len(l.Glyphs) - 1 + inc = -inc + glyphsProcessed = len(l.Glyphs) - 1 + } + // Construct clusters from the line's glyphs. + for ; i < len(l.Glyphs) && i >= 0; i += inc { + g := l.Glyphs[i] + xAdv := g.XAdvance * fixed.Int26_6(inc) + for k := 0; k < g.GlyphCount-1 && k < len(l.Glyphs); k++ { + i += inc + xAdv += l.Glyphs[i].XAdvance * fixed.Int26_6(inc) + } + + startRune := runesProcessed + runeIncrement := g.RuneCount + startGlyph := glyphsProcessed + glyphIncrement := g.GlyphCount * inc + if rtl { + startGlyph = glyphsProcessed + glyphIncrement + 1 + } + clusters = append(clusters, text.GlyphCluster{ + Advance: xAdv, + Runes: text.Range{ + Count: g.RuneCount, + Offset: startRune + l.Runes.Offset, + }, + Glyphs: text.Range{ + Count: g.GlyphCount, + Offset: startGlyph, + }, + }) + runesProcessed += runeIncrement + glyphsProcessed += glyphIncrement + } + l.Clusters = clusters +} diff --git a/font/opentype/opentype_test.go b/font/opentype/opentype_test.go index 95502302..4e3a8da2 100644 --- a/font/opentype/opentype_test.go +++ b/font/opentype/opentype_test.go @@ -1,13 +1,17 @@ package opentype import ( - "strings" + "math" + "reflect" "testing" + nsareg "eliasnaur.com/font/noto/sans/arabic/regular" + "github.com/go-text/typesetting/shaping" "golang.org/x/image/font/gofont/goregular" "golang.org/x/image/math/fixed" "gioui.org/io/system" + "gioui.org/text" ) var english = system.Locale{ @@ -15,6 +19,11 @@ var english = system.Locale{ Direction: system.LTR, } +var arabic = system.Locale{ + Language: "AR", + Direction: system.RTL, +} + func TestEmptyString(t *testing.T) { face, err := Parse(goregular.TTF) if err != nil { @@ -22,11 +31,9 @@ func TestEmptyString(t *testing.T) { } ppem := fixed.I(200) + shaper := Shaper{} - lines, err := face.Layout(ppem, 2000, english, strings.NewReader("")) - if err != nil { - t.Fatal(err) - } + lines := shaper.Layout([]text.Face{face}, ppem, 2000, english, []rune{}) if len(lines) == 0 { t.Fatalf("Layout returned no lines for empty string; expected 1") } @@ -43,3 +50,839 @@ func TestEmptyString(t *testing.T) { t.Errorf("got bounds %+v for empty string; expected %+v", got, exp) } } + +// simpleGlyph returns a simple square glyph with the provided cluster +// value. +func simpleGlyph(cluster int) shaping.Glyph { + return complexGlyph(cluster, 1, 1) +} + +// ligatureGlyph returns a simple square glyph with the provided cluster +// value and number of runes. +func ligatureGlyph(cluster, runes int) shaping.Glyph { + return complexGlyph(cluster, runes, 1) +} + +// expansionGlyph returns a simple square glyph with the provided cluster +// value and number of glyphs. +func expansionGlyph(cluster, glyphs int) shaping.Glyph { + return complexGlyph(cluster, 1, glyphs) +} + +// complexGlyph returns a simple square glyph with the provided cluster +// value, number of associated runes, and number of glyphs in the cluster. +func complexGlyph(cluster, runes, glyphs int) shaping.Glyph { + return shaping.Glyph{ + Width: fixed.I(10), + Height: fixed.I(10), + XAdvance: fixed.I(10), + YAdvance: fixed.I(10), + YBearing: fixed.I(10), + ClusterIndex: cluster, + GlyphCount: glyphs, + RuneCount: runes, + } +} + +func simpleCluster(runeOffset, glyphOffset int, ltr bool) text.GlyphCluster { + g := text.GlyphCluster{ + Advance: fixed.I(10), + Runes: text.Range{ + Count: 1, + Offset: runeOffset, + }, + Glyphs: text.Range{ + Count: 1, + Offset: glyphOffset, + }, + } + if !ltr { + g.Advance = -g.Advance + } + return g +} + +func TestLayoutComputeClusters(t *testing.T) { + type testcase struct { + name string + line text.Layout + expected []text.GlyphCluster + } + for _, tc := range []testcase{ + { + name: "empty", + expected: []text.GlyphCluster{}, + }, + { + name: "just newline", + line: text.Layout{ + Direction: system.LTR, + Glyphs: toGioGlyphs([]shaping.Glyph{}, nil), + Runes: text.Range{ + Count: 1, + }, + }, + expected: []text.GlyphCluster{ + { + Runes: text.Range{ + Count: 1, + }, + }, + }, + }, + { + name: "simple", + line: text.Layout{ + Direction: system.LTR, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(0), + simpleGlyph(1), + simpleGlyph(2), + }, nil), + Runes: text.Range{ + Count: 3, + }, + }, + expected: []text.GlyphCluster{ + simpleCluster(0, 0, true), + simpleCluster(1, 1, true), + simpleCluster(2, 2, true), + }, + }, + { + name: "simple at nonzero offset", + line: text.Layout{ + Direction: system.LTR, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(5), + simpleGlyph(6), + simpleGlyph(7), + }, nil), + Runes: text.Range{ + Count: 3, + Offset: 5, + }, + }, + expected: []text.GlyphCluster{ + simpleCluster(5, 0, true), + simpleCluster(6, 1, true), + simpleCluster(7, 2, true), + }, + }, + { + name: "simple with newline", + line: text.Layout{ + Direction: system.LTR, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(0), + simpleGlyph(1), + simpleGlyph(2), + }, nil), + Runes: text.Range{ + Count: 4, + }, + }, + expected: []text.GlyphCluster{ + simpleCluster(0, 0, true), + simpleCluster(1, 1, true), + simpleCluster(2, 2, true), + { + Runes: text.Range{ + Count: 1, + Offset: 3, + }, + Glyphs: text.Range{ + Offset: 3, + }, + }, + }, + }, + { + name: "ligature", + line: text.Layout{ + Direction: system.LTR, + Glyphs: toGioGlyphs([]shaping.Glyph{ + ligatureGlyph(0, 2), + simpleGlyph(2), + simpleGlyph(3), + }, nil), + Runes: text.Range{ + Count: 4, + }, + }, + expected: []text.GlyphCluster{ + { + Advance: fixed.I(10), + Runes: text.Range{ + Count: 2, + }, + Glyphs: text.Range{ + Count: 1, + }, + }, + simpleCluster(2, 1, true), + simpleCluster(3, 2, true), + }, + }, + { + name: "ligature with newline", + line: text.Layout{ + Direction: system.LTR, + Glyphs: toGioGlyphs([]shaping.Glyph{ + ligatureGlyph(0, 2), + }, nil), + Runes: text.Range{ + Count: 3, + }, + }, + expected: []text.GlyphCluster{ + { + Advance: fixed.I(10), + Runes: text.Range{ + Count: 2, + }, + Glyphs: text.Range{ + Count: 1, + }, + }, + { + Runes: text.Range{ + Count: 1, + Offset: 2, + }, + Glyphs: text.Range{ + Offset: 1, + }, + }, + }, + }, + { + name: "expansion", + line: text.Layout{ + Direction: system.LTR, + Glyphs: toGioGlyphs([]shaping.Glyph{ + expansionGlyph(0, 2), + expansionGlyph(0, 2), + simpleGlyph(1), + simpleGlyph(2), + }, nil), + Runes: text.Range{ + Count: 3, + }, + }, + expected: []text.GlyphCluster{ + { + Advance: fixed.I(20), + Runes: text.Range{ + Count: 1, + }, + Glyphs: text.Range{ + Count: 2, + }, + }, + simpleCluster(1, 2, true), + simpleCluster(2, 3, true), + }, + }, + { + name: "deletion", + line: text.Layout{ + Direction: system.LTR, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(0), + ligatureGlyph(1, 2), + simpleGlyph(3), + simpleGlyph(4), + }, nil), + Runes: text.Range{ + Count: 5, + }, + }, + expected: []text.GlyphCluster{ + simpleCluster(0, 0, true), + { + Advance: fixed.I(10), + Runes: text.Range{ + Count: 2, + Offset: 1, + }, + Glyphs: text.Range{ + Count: 1, + Offset: 1, + }, + }, + simpleCluster(3, 2, true), + simpleCluster(4, 3, true), + }, + }, + { + name: "simple rtl", + line: text.Layout{ + Direction: system.RTL, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(2), + simpleGlyph(1), + simpleGlyph(0), + }, nil), + Runes: text.Range{ + Count: 3, + }, + }, + expected: []text.GlyphCluster{ + simpleCluster(0, 2, false), + simpleCluster(1, 1, false), + simpleCluster(2, 0, false), + }, + }, + { + name: "simple rtl at nonzero offset", + line: text.Layout{ + Direction: system.RTL, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(7), + simpleGlyph(6), + simpleGlyph(5), + }, nil), + Runes: text.Range{ + Count: 3, + Offset: 5, + }, + }, + expected: []text.GlyphCluster{ + simpleCluster(5, 2, false), + simpleCluster(6, 1, false), + simpleCluster(7, 0, false), + }, + }, + { + name: "simple rtl with newline", + line: text.Layout{ + Direction: system.RTL, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(2), + simpleGlyph(1), + simpleGlyph(0), + }, nil), + Runes: text.Range{ + Count: 4, + }, + }, + expected: []text.GlyphCluster{ + simpleCluster(0, 2, false), + simpleCluster(1, 1, false), + simpleCluster(2, 0, false), + { + Runes: text.Range{ + Count: 1, + Offset: 3, + }, + }, + }, + }, + { + name: "ligature rtl", + line: text.Layout{ + Direction: system.RTL, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(3), + simpleGlyph(2), + ligatureGlyph(0, 2), + }, nil), + Runes: text.Range{ + Count: 4, + }, + }, + expected: []text.GlyphCluster{ + { + Advance: fixed.I(-10), + Runes: text.Range{ + Count: 2, + }, + Glyphs: text.Range{ + Count: 1, + Offset: 2, + }, + }, + simpleCluster(2, 1, false), + simpleCluster(3, 0, false), + }, + }, + { + name: "ligature rtl with newline", + line: text.Layout{ + Direction: system.RTL, + Glyphs: toGioGlyphs([]shaping.Glyph{ + ligatureGlyph(0, 2), + }, nil), + Runes: text.Range{ + Count: 3, + }, + }, + expected: []text.GlyphCluster{ + { + Advance: fixed.I(-10), + Runes: text.Range{ + Count: 2, + }, + Glyphs: text.Range{ + Count: 1, + }, + }, + { + Runes: text.Range{ + Count: 1, + Offset: 2, + }, + }, + }, + }, + { + name: "expansion rtl", + line: text.Layout{ + Direction: system.RTL, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(2), + simpleGlyph(1), + expansionGlyph(0, 2), + expansionGlyph(0, 2), + }, nil), + Runes: text.Range{ + Count: 3, + }, + }, + expected: []text.GlyphCluster{ + { + Advance: fixed.I(-20), + Runes: text.Range{ + Count: 1, + }, + Glyphs: text.Range{ + Count: 2, + Offset: 2, + }, + }, + simpleCluster(1, 1, false), + simpleCluster(2, 0, false), + }, + }, + { + name: "deletion rtl", + line: text.Layout{ + Direction: system.RTL, + Glyphs: toGioGlyphs([]shaping.Glyph{ + simpleGlyph(4), + simpleGlyph(3), + ligatureGlyph(1, 2), + simpleGlyph(0), + }, nil), + Runes: text.Range{ + Count: 5, + }, + }, + expected: []text.GlyphCluster{ + simpleCluster(0, 3, false), + { + Advance: fixed.I(-10), + Runes: text.Range{ + Count: 2, + Offset: 1, + }, + Glyphs: text.Range{ + Count: 1, + Offset: 2, + }, + }, + simpleCluster(3, 1, false), + simpleCluster(4, 0, false), + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + computeGlyphClusters(&tc.line) + actual := tc.line.Clusters + for i, cluster := range actual { + expected := tc.expected[i] + if expected.Runes != cluster.Runes { + t.Errorf("cluster %d runes mismatch, expected %#+v, got %#+v", i, expected.Runes, cluster.Runes) + } + if expected.Glyphs != cluster.Glyphs { + t.Errorf("cluster %d glyphs mismatch, expected %#+v, got %#+v", i, expected.Glyphs, cluster.Glyphs) + } + if expected.Advance != cluster.Advance { + t.Errorf("cluster %d advance mismatch, expected %#+v, got %#+v", i, expected.Advance, cluster.Advance) + } + } + }) + } +} + +// makeTestText returns an ltr, rtl, and bidi sample of shaped text at the given +// font size and wrapped to the given line width. The runeLimit, if nonzero, +// truncates the sample text to ensure shorter output for expensive tests. +func makeTestText(fontSize, lineWidth, runeLimit int) (ltr, rtl, bidi, bidi2 []shaping.Line) { + ltrFace, _ := Parse(goregular.TTF) + rtlFace, _ := Parse(nsareg.TTF) + + shaper := &Shaper{} + ltrSource := "The quick brown fox jumps over the lazy dog." + rtlSource := "الحب سماء لا تمط غير الأحلام" + // bidiSource is crafted to contain multiple consecutive RTL runs (by + // changing scripts within the RTL). + bidiSource := "The quick سماء שלום لا fox تمط שלום غير the lazy dog." + // bidi2Source is crafted to contain multiple consecutive LTR runs (by + // changing scripts within the LTR). + bidi2Source := "الحب سماء brown привет fox تمط jumps привет over غير الأحلام" + if runeLimit != 0 { + ltrRunes := []rune(ltrSource) + rtlRunes := []rune(rtlSource) + bidiRunes := []rune(bidiSource) + bidi2Runes := []rune(bidi2Source) + if runeLimit < len(ltrRunes) { + ltrSource = string(ltrRunes[:runeLimit]) + } + if runeLimit < len(rtlRunes) { + rtlSource = string(rtlRunes[:runeLimit]) + } + if runeLimit < len(bidiRunes) { + bidiSource = string(bidiRunes[:runeLimit]) + } + if runeLimit < len(bidi2Runes) { + bidi2Source = string(bidi2Runes[:runeLimit]) + } + } + ltrText := shaper.shapeAndWrapText([]text.Face{ltrFace, rtlFace}, fixed.I(fontSize), lineWidth, english, []rune(ltrSource)) + rtlText := shaper.shapeAndWrapText([]text.Face{rtlFace, ltrFace}, fixed.I(fontSize), lineWidth, arabic, []rune(rtlSource)) + bidiText := shaper.shapeAndWrapText([]text.Face{ltrFace, rtlFace}, fixed.I(fontSize), lineWidth, english, []rune(bidiSource)) + bidi2Text := shaper.shapeAndWrapText([]text.Face{rtlFace, ltrFace}, fixed.I(fontSize), lineWidth, arabic, []rune(bidi2Source)) + return ltrText, rtlText, bidiText, bidi2Text +} + +func fixedAbs(a fixed.Int26_6) fixed.Int26_6 { + if a < 0 { + a = -a + } + return a +} + +func TestToLine(t *testing.T) { + ltr, rtl, bidi, bidi2 := makeTestText(16, 100, 0) + _, _, bidiWide, bidi2Wide := makeTestText(16, 200, 0) + type testcase struct { + name string + lines []shaping.Line + // Dominant text direction. + dir system.TextDirection + } + for _, tc := range []testcase{ + { + name: "ltr", + lines: ltr, + dir: system.LTR, + }, + { + name: "rtl", + lines: rtl, + dir: system.RTL, + }, + { + name: "bidi", + lines: bidi, + dir: system.LTR, + }, + { + name: "bidi2", + lines: bidi2, + dir: system.RTL, + }, + { + name: "bidi_wide", + lines: bidiWide, + dir: system.LTR, + }, + { + name: "bidi2_wide", + lines: bidi2Wide, + dir: system.RTL, + }, + } { + t.Run(tc.name, func(t *testing.T) { + // We expect: + // - Line dimensions to be populated. + // - Line direction to be populated. + // - Runs to be ordered from lowest runes first. + // - Runs to have widths matching the input. + // - Runs to have the same total number of glyphs/runes as the input. + runesSeen := text.Range{} + for i, input := range tc.lines { + seenRun := make([]bool, len(input)) + inputLowestRuneOffset := math.MaxInt + totalInputGlyphs := 0 + totalInputRunes := 0 + for _, run := range input { + if run.Runes.Offset < inputLowestRuneOffset { + inputLowestRuneOffset = run.Runes.Offset + } + totalInputGlyphs += len(run.Glyphs) + totalInputRunes += run.Runes.Count + } + output := ToLine(input, tc.dir) + if output.Bounds.Min == (fixed.Point26_6{}) { + t.Errorf("line %d: Bounds.Min not populated", i) + } + if output.Bounds.Max == (fixed.Point26_6{}) { + t.Errorf("line %d: Bounds.Max not populated", i) + } + if output.Direction != tc.dir { + t.Errorf("line %d: expected direction %v, got %v", i, tc.dir, output.Direction) + } + totalRunWidth := fixed.I(0) + totalLineGlyphs := 0 + totalLineRunes := 0 + for k, run := range output.Runs { + seenRun[run.VisualPosition] = true + if output.VisualOrder[run.VisualPosition] != k { + t.Errorf("line %d, run %d: run.VisualPosition=%d, but line.VisualOrder[%d]=%d(should be %d)", i, k, run.VisualPosition, run.VisualPosition, output.VisualOrder[run.VisualPosition], k) + } + if run.Runes.Offset != totalLineRunes { + t.Errorf("line %d, run %d: expected Runes.Offset to be %d, got %d", i, k, totalLineRunes, run.Runes.Offset) + } + runGlyphCount := 0 + glyphMul := 1 + if run.Direction.Progression() == system.TowardOrigin { + // Subtract one from this count so that we align with the offset + runGlyphCount = len(run.Glyphs) - 1 + glyphMul = -1 + } + runRuneCount := 0 + for m, cluster := range run.Clusters { + if cluster.Glyphs.Offset != runGlyphCount { + t.Errorf("line %d, run %d, cluster %d: expected Glyphs.Offset to be %d, got %d", i, k, m, runGlyphCount, cluster.Glyphs.Offset) + } + if cluster.Runes.Offset != runRuneCount+totalLineRunes { + t.Errorf("line %d, run %d, cluster %d: expected Runes.Offset to be %d, got %d", i, k, m, runRuneCount+totalLineRunes, cluster.Runes.Offset) + } + runGlyphCount += glyphMul * cluster.Glyphs.Count + runRuneCount += cluster.Runes.Count + } + if run.Direction.Progression() == system.TowardOrigin && runGlyphCount != -1 { + t.Errorf("line %d, run %d: expected -1 glyphs unaccounted in RTL run, got %d", i, k, runGlyphCount) + } else if run.Direction.Progression() == system.FromOrigin && runGlyphCount != len(run.Glyphs) { + t.Errorf("line %d, run %d: expected %d glyphs in LTR run, got %d", i, k, len(run.Glyphs), runGlyphCount) + } + if run.Runes.Count != runRuneCount { + t.Errorf("line %d, run %d: expected %d runes, counted %d", i, k, run.Runes.Count, runRuneCount) + } + runesSeen.Count += run.Runes.Count + totalRunWidth += fixedAbs(run.Advance) + totalLineGlyphs += len(run.Glyphs) + totalLineRunes += run.Runes.Count + } + if output.RuneCount != totalInputRunes { + t.Errorf("line %d: input had %d runes, only counted %d", i, totalInputRunes, output.RuneCount) + } + if totalLineGlyphs != totalInputGlyphs { + t.Errorf("line %d: input had %d glyphs, only counted %d", i, totalInputRunes, totalLineGlyphs) + } + if totalRunWidth != output.Width { + t.Errorf("line %d: expected width %d, got %d", i, totalRunWidth, output.Width) + } + for runIndex, seen := range seenRun { + if !seen { + t.Errorf("line %d, run %d missing from runs VisualPosition fields", i, runIndex) + } + } + } + lastLine := tc.lines[len(tc.lines)-1] + maxRunes := 0 + for _, run := range lastLine { + if run.Runes.Count+run.Runes.Offset > maxRunes { + maxRunes = run.Runes.Count + run.Runes.Offset + } + } + if runesSeen.Count != maxRunes { + t.Errorf("input covered %d runes, output only covers %d", maxRunes, runesSeen.Count) + } + }) + } +} + +func TestComputeVisualOrder(t *testing.T) { + type testcase struct { + name string + input text.Line + expectedVisualOrder []int + } + for _, tc := range []testcase{ + { + name: "ltr", + input: text.Line{ + Direction: system.LTR, + Runs: []text.Layout{ + {Direction: system.LTR}, + {Direction: system.LTR}, + {Direction: system.LTR}, + }, + }, + expectedVisualOrder: []int{0, 1, 2}, + }, + { + name: "rtl", + input: text.Line{ + Direction: system.RTL, + Runs: []text.Layout{ + {Direction: system.RTL}, + {Direction: system.RTL}, + {Direction: system.RTL}, + }, + }, + expectedVisualOrder: []int{2, 1, 0}, + }, + { + name: "bidi-ltr", + input: text.Line{ + Direction: system.LTR, + Runs: []text.Layout{ + {Direction: system.LTR}, + {Direction: system.RTL}, + {Direction: system.RTL}, + {Direction: system.RTL}, + {Direction: system.LTR}, + }, + }, + expectedVisualOrder: []int{0, 3, 2, 1, 4}, + }, + { + name: "bidi-ltr-complex", + input: text.Line{ + Direction: system.LTR, + Runs: []text.Layout{ + {Direction: system.RTL}, + {Direction: system.RTL}, + {Direction: system.LTR}, + {Direction: system.RTL}, + {Direction: system.RTL}, + {Direction: system.LTR}, + {Direction: system.RTL}, + {Direction: system.RTL}, + {Direction: system.LTR}, + {Direction: system.RTL}, + {Direction: system.RTL}, + }, + }, + expectedVisualOrder: []int{1, 0, 2, 4, 3, 5, 7, 6, 8, 10, 9}, + }, + { + name: "bidi-rtl", + input: text.Line{ + Direction: system.RTL, + Runs: []text.Layout{ + {Direction: system.RTL}, + {Direction: system.LTR}, + {Direction: system.LTR}, + {Direction: system.LTR}, + {Direction: system.RTL}, + }, + }, + expectedVisualOrder: []int{4, 1, 2, 3, 0}, + }, + { + name: "bidi-rtl-complex", + input: text.Line{ + Direction: system.RTL, + Runs: []text.Layout{ + {Direction: system.LTR}, + {Direction: system.LTR}, + {Direction: system.RTL}, + {Direction: system.LTR}, + {Direction: system.LTR}, + {Direction: system.RTL}, + {Direction: system.LTR}, + {Direction: system.LTR}, + {Direction: system.RTL}, + {Direction: system.LTR}, + {Direction: system.LTR}, + }, + }, + expectedVisualOrder: []int{9, 10, 8, 6, 7, 5, 3, 4, 2, 0, 1}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + computeVisualOrder(&tc.input) + if !reflect.DeepEqual(tc.input.VisualOrder, tc.expectedVisualOrder) { + t.Errorf("expected visual order %v, got %v", tc.expectedVisualOrder, tc.input.VisualOrder) + } + for i, visualIndex := range tc.input.VisualOrder { + if pos := tc.input.Runs[visualIndex].VisualPosition; pos != i { + t.Errorf("line.VisualOrder[%d]=%d, but line.Runs[%d].VisualPosition=%d", i, visualIndex, visualIndex, pos) + } + } + }) + } +} + +func FuzzLayout(f *testing.F) { + ltrFace, _ := Parse(goregular.TTF) + rtlFace, _ := Parse(nsareg.TTF) + f.Add("د عرمثال dstي met لم aqل جدmوpمg lرe dرd لو عل ميrةsdiduntut lab renنيتذدagلaaiua.ئPocttأior رادرsاي mيrbلmnonaيdتد ماةعcلخ.", true, uint8(10), uint16(200)) + + shaper := Shaper{} + f.Fuzz(func(t *testing.T, txt string, rtl bool, fontSize uint8, width uint16) { + locale := system.Locale{ + Direction: system.LTR, + } + if rtl { + locale.Direction = system.RTL + } + if fontSize < 1 { + fontSize = 1 + } + lines := shaper.Layout([]text.Face{ltrFace, rtlFace}, fixed.I(int(fontSize)), int(width), locale, []rune(txt)) + validateLines(t, lines, len([]rune(txt))) + }) +} + +func validateLines(t *testing.T, lines []text.Line, expectedRuneCount int) { + t.Helper() + runesSeen := 0 + for i, line := range lines { + if line.Bounds.Min == (fixed.Point26_6{}) { + t.Errorf("line %d: Bounds.Min not populated", i) + } + if line.Bounds.Max == (fixed.Point26_6{}) { + t.Errorf("line %d: Bounds.Max not populated", i) + } + totalRunWidth := fixed.I(0) + totalLineGlyphs := 0 + lineRunesSeen := 0 + for k, run := range line.Runs { + if line.VisualOrder[run.VisualPosition] != k { + t.Errorf("line %d, run %d: run.VisualPosition=%d, but line.VisualOrder[%d]=%d(should be %d)", i, k, run.VisualPosition, run.VisualPosition, line.VisualOrder[run.VisualPosition], k) + } + if run.Runes.Offset != lineRunesSeen { + t.Errorf("line %d, run %d: expected Runes.Offset to be %d, got %d", i, k, lineRunesSeen, run.Runes.Offset) + } + runGlyphCount := 0 + runRuneCount := 0 + for m, cluster := range run.Clusters { + if cluster.Runes.Offset != runRuneCount+lineRunesSeen { + t.Errorf("line %d, run %d, cluster %d: expected Runes.Offset to be %d, got %d", i, k, m, runRuneCount+lineRunesSeen, cluster.Runes.Offset) + } + runGlyphCount += cluster.Glyphs.Count + runRuneCount += cluster.Runes.Count + } + if run.Runes.Count != runRuneCount { + t.Errorf("line %d, run %d: expected %d runes, counted %d", i, k, run.Runes.Count, runRuneCount) + } + lineRunesSeen += run.Runes.Count + totalRunWidth += fixedAbs(run.Advance) + totalLineGlyphs += len(run.Glyphs) + } + if totalRunWidth != line.Width { + t.Errorf("line %d: expected width %d, got %d", i, line.Width, totalRunWidth) + } + runesSeen += lineRunesSeen + } + if runesSeen != expectedRuneCount { + t.Errorf("input covered %d runes, output only covers %d", expectedRuneCount, runesSeen) + } +} diff --git a/font/opentype/testdata/fuzz/FuzzLayout/2a7730fcbcc3550718b37415eb6104cf09159c7d756cc3530ccaa9007c0a2c06 b/font/opentype/testdata/fuzz/FuzzLayout/2a7730fcbcc3550718b37415eb6104cf09159c7d756cc3530ccaa9007c0a2c06 new file mode 100644 index 00000000..00ec62d7 --- /dev/null +++ b/font/opentype/testdata/fuzz/FuzzLayout/2a7730fcbcc3550718b37415eb6104cf09159c7d756cc3530ccaa9007c0a2c06 @@ -0,0 +1,5 @@ +go test fuzz v1 +string("\x1d") +bool(true) +byte('\x1c') +uint16(227) diff --git a/font/opentype/testdata/fuzz/FuzzLayout/3fc3ee939f0df44719bee36a67df3aa3a562e2f2f1cfb665a650f60e7e0ab4e6 b/font/opentype/testdata/fuzz/FuzzLayout/3fc3ee939f0df44719bee36a