Skip to content

Add Truncate methods #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
clipperhouse wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Add Truncate methods #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2b622a3
Implement TruncateString
clipperhouse Dec 14, 2025
df09b7b
Benchmarks & readme
clipperhouse Dec 14, 2025
a576bbd
Implement TruncateBytes
clipperhouse Dec 14, 2025
3bc3975
Fuzz Truncate
clipperhouse Dec 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 26 additions & 20 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,33 +133,39 @@ goarch: arm64
pkg: github.com/clipperhouse/displaywidth/comparison
cpu: Apple M2

BenchmarkString_Mixed/clipperhouse/displaywidth-8 10326 ns/op 163.37 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 14415 ns/op 117.03 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19343 ns/op 87.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/clipperhouse/displaywidth-8 10400 ns/op 162.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 14296 ns/op 118.00 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19770 ns/op 85.33 MB/s 0 B/op 0 allocs/op

BenchmarkString_EastAsian/clipperhouse/displaywidth-8 10561 ns/op 159.74 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 23790 ns/op 70.91 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19322 ns/op 87.31 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/clipperhouse/displaywidth-8 10593 ns/op 159.26 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 23980 ns/op 70.35 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19777 ns/op 85.30 MB/s 0 B/op 0 allocs/op

BenchmarkString_ASCII/clipperhouse/displaywidth-8 1033 ns/op 123.88 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 1168 ns/op 109.59 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1585 ns/op 80.74 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/clipperhouse/displaywidth-8 1032 ns/op 124.09 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 1162 ns/op 110.16 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1586 ns/op 80.69 MB/s 0 B/op 0 allocs/op

BenchmarkString_Emoji/clipperhouse/displaywidth-8 3034 ns/op 238.61 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 4797 ns/op 150.94 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6612 ns/op 109.50 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/clipperhouse/displaywidth-8 3017 ns/op 240.01 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 4745 ns/op 152.58 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6745 ns/op 107.34 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3343 ns/op 504.67 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 5414 ns/op 311.62 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3381 ns/op 498.90 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 5383 ns/op 313.41 MB/s 0 B/op 0 allocs/op

BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3393 ns/op 497.17 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15312 ns/op 110.17 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3395 ns/op 496.96 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15645 ns/op 107.83 MB/s 0 B/op 0 allocs/op

BenchmarkRune_ASCII/clipperhouse/displaywidth-8 256.9 ns/op 498.32 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 265.7 ns/op 481.75 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/clipperhouse/displaywidth-8 257.8 ns/op 496.57 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 267.3 ns/op 478.89 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1336 ns/op 541.96 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2304 ns/op 314.23 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1338 ns/op 541.24 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2287 ns/op 316.58 MB/s 0 B/op 0 allocs/op

BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 3689 ns/op 47.98 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 8069 ns/op 21.93 MB/s 192 B/op 14 allocs/op

BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 3457 ns/op 66.24 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 10441 ns/op 21.93 MB/s 0 B/op 0 allocs/op
```

Here are some notes on [how to make Unicode things fast](https://clipperhouse.com/go-unicode/).
46 changes: 26 additions & 20 deletions comparison/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,31 +20,37 @@ goarch: arm64
pkg: github.com/clipperhouse/displaywidth/comparison
cpu: Apple M2

BenchmarkString_Mixed/clipperhouse/displaywidth-8 10326 ns/op 163.37 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 14415 ns/op 117.03 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19343 ns/op 87.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/clipperhouse/displaywidth-8 10400 ns/op 162.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 14296 ns/op 118.00 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19770 ns/op 85.33 MB/s 0 B/op 0 allocs/op

BenchmarkString_EastAsian/clipperhouse/displaywidth-8 10561 ns/op 159.74 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 23790 ns/op 70.91 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19322 ns/op 87.31 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/clipperhouse/displaywidth-8 10593 ns/op 159.26 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 23980 ns/op 70.35 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19777 ns/op 85.30 MB/s 0 B/op 0 allocs/op

BenchmarkString_ASCII/clipperhouse/displaywidth-8 1033 ns/op 123.88 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 1168 ns/op 109.59 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1585 ns/op 80.74 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/clipperhouse/displaywidth-8 1032 ns/op 124.09 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 1162 ns/op 110.16 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1586 ns/op 80.69 MB/s 0 B/op 0 allocs/op

BenchmarkString_Emoji/clipperhouse/displaywidth-8 3034 ns/op 238.61 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 4797 ns/op 150.94 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6612 ns/op 109.50 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/clipperhouse/displaywidth-8 3017 ns/op 240.01 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 4745 ns/op 152.58 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6745 ns/op 107.34 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3343 ns/op 504.67 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 5414 ns/op 311.62 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3381 ns/op 498.90 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 5383 ns/op 313.41 MB/s 0 B/op 0 allocs/op

BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3393 ns/op 497.17 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15312 ns/op 110.17 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3395 ns/op 496.96 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15645 ns/op 107.83 MB/s 0 B/op 0 allocs/op

BenchmarkRune_ASCII/clipperhouse/displaywidth-8 256.9 ns/op 498.32 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 265.7 ns/op 481.75 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/clipperhouse/displaywidth-8 257.8 ns/op 496.57 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 267.3 ns/op 478.89 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1336 ns/op 541.96 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2304 ns/op 314.23 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1338 ns/op 541.24 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2287 ns/op 316.58 MB/s 0 B/op 0 allocs/op

BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 3689 ns/op 47.98 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 8069 ns/op 21.93 MB/s 192 B/op 14 allocs/op

BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 3457 ns/op 66.24 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 10441 ns/op 21.93 MB/s 0 B/op 0 allocs/op
```
96 changes: 96 additions & 0 deletions comparison/benchmark_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -474,3 +474,99 @@ func BenchmarkRune_Emoji(b *testing.B) {
}
})
}

func BenchmarkTruncateWithTail(b *testing.B) {
testStrings := []string{
"hello world",
"This is a very long string that will definitely be truncated",
"Hello 世界! 😀",
"👨‍💻 working on 🚀",
"中文字符串测试",
"😀😁😂🤣😃😄😅😆😉😊",
}

n := int64(0)
for _, s := range testStrings {
n += int64(len(s))
}

maxWidths := []int{5, 10, 20, 30}
tail := "..."

b.Run("clipperhouse/displaywidth", func(b *testing.B) {
options := displaywidth.Options{}
b.SetBytes(n)
b.ReportAllocs()
b.ResetTimer()

for i := 0; i < b.N; i++ {
for _, s := range testStrings {
for _, w := range maxWidths {
_ = options.TruncateString(s, w, tail)
}
}
}
})

b.Run("mattn/go-runewidth", func(b *testing.B) {
b.SetBytes(n)
b.ReportAllocs()
b.ResetTimer()

for i := 0; i < b.N; i++ {
for _, s := range testStrings {
for _, w := range maxWidths {
_ = runewidth.Truncate(s, w, tail)
}
}
}
})
}

func BenchmarkTruncateWithoutTail(b *testing.B) {
testStrings := []string{
"hello world",
"This is a very long string that will definitely be truncated",
"Hello 世界! 😀",
"👨‍💻 working on 🚀",
"a very long string that will definitely be truncated",
"中文字符串测试",
"😀😁😂🤣😃😄😅😆😉😊",
}

n := int64(0)
for _, s := range testStrings {
n += int64(len(s))
}

maxWidths := []int{5, 10, 20, 30}

b.Run("clipperhouse/displaywidth", func(b *testing.B) {
options := displaywidth.Options{}
b.SetBytes(n)
b.ReportAllocs()
b.ResetTimer()

for i := 0; i < b.N; i++ {
for _, s := range testStrings {
for _, w := range maxWidths {
_ = options.TruncateString(s, w, "")
}
}
}
})

b.Run("mattn/go-runewidth", func(b *testing.B) {
b.SetBytes(n)
b.ReportAllocs()
b.ResetTimer()

for i := 0; i < b.N; i++ {
for _, s := range testStrings {
for _, w := range maxWidths {
_ = runewidth.Truncate(s, w, "")
}
}
}
})
}
121 changes: 113 additions & 8 deletions fuzz_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,15 @@ package displaywidth

import (
"bytes"
"strings"
"testing"
"unicode/utf8"

"github.com/clipperhouse/displaywidth/testdata"
)

// FuzzBytes fuzzes the Bytes function with valid and invalid UTF-8.
func FuzzBytes(f *testing.F) {
// FuzzBytesAndString fuzzes the Bytes function with valid and invalid UTF-8.
func FuzzBytesAndString(f *testing.F) {
if testing.Short() {
f.Skip("skipping fuzz test in short mode")
}
Expand Down Expand Up @@ -114,12 +115,9 @@ func FuzzBytes(f *testing.F) {
t.Errorf("Bytes() with options %+v returned non-zero width %d for empty input", option, wb)
}

// Consistency check with String() for valid UTF-8
if utf8.Valid(text) {
ws := option.String(string(text))
if wb != ws {
t.Errorf("Bytes() returned %d but String() returned %d with options %+v for %q", wb, ws, option, text)
}
ws := option.String(string(text))
if wb != ws {
t.Errorf("Bytes() returned %d but String() returned %d with options %+v for %q", wb, ws, option, text)
}
}
})
Expand Down Expand Up @@ -222,3 +220,110 @@ func FuzzRune(f *testing.F) {
}
})
}

func FuzzTruncateStringAndBytes(f *testing.F) {
if testing.Short() {
f.Skip("skipping fuzz test in short mode")
}

// Seed with multi-lingual text (paragraph-sized chunks)
file, err := testdata.Sample()
if err != nil {
f.Fatal(err)
}
fs := string(file)
chunks := strings.Split(fs, "\n")
for _, chunk := range chunks {
f.Add(chunk)
}

// Seed with invalid UTF-8
invalid, err := testdata.InvalidUTF8()
if err != nil {
f.Fatal(err)
}
fs = string(invalid)
chunks = strings.Split(fs, "\n")
for _, chunk := range chunks {
f.Add(chunk)
}

// Seed with test cases
testCases, err := testdata.TestCases()
if err != nil {
f.Fatal(err)
}
fs = string(testCases)
chunks = strings.Split(fs, "\n")
for _, chunk := range chunks {
f.Add(chunk)
}

// Seed with random bytes
for i := 0; i < 10; i++ {
b, err := testdata.RandomBytes()
if err != nil {
f.Fatal(err)
}
f.Add(string(b))
}

// Seed with edge cases
f.Add("") // empty
f.Add("a") // single ASCII
f.Add("\t\n\r") // whitespace
f.Add("🌍") // emoji
f.Add("\u0301") // combining mark
f.Add("\xff\xfe\xfd") // invalid UTF-8

f.Fuzz(func(t *testing.T, text string) {
// Test with default options
ts := TruncateString(text, 10, "...")

// Invariant: truncated string should be less than or equal to maxWidth
if String(ts) > 10 {
t.Errorf("TruncateString() returned string longer than maxWidth for %q: %q", text, ts)
}

// Invariant: truncated string should be less than or equal to maxWidth
if len(ts) > len(text) {
t.Errorf("TruncateString() returned string longer than original for %q: %q", text, ts)
}

tb := TruncateBytes([]byte(text), 10, []byte("..."))

// Invariant: truncated bytes should be less than or equal to maxWidth
if Bytes(tb) > 10 {
t.Errorf("TruncateBytes() returned bytes longer than maxWidth for %q: %q", text, tb)
}

// Invariant: truncated bytes should be less than or equal to original
if len(tb) > len(text) {
t.Errorf("TruncateBytes() returned bytes longer than original for %q: %q", text, tb)
}

if !bytes.Equal(tb, []byte(ts)) {
t.Errorf("TruncateBytes() returned bytes different from TruncateString() for %q: %q != %q", text, tb, ts)
}

// Test with different options
options := []Options{
{EastAsianWidth: false}, // default
{EastAsianWidth: true},
}

for _, option := range options {
ts := option.TruncateString(text, 10, "...")

// Invariant: truncated string should be less than or equal to maxWidth
if option.String(ts) > 10 {
t.Errorf("TruncateString() returned string longer than maxWidth for %q: %q", text, ts)
}

tb := option.TruncateBytes([]byte(text), 10, []byte("..."))
if !bytes.Equal(tb, []byte(ts)) {
t.Errorf("TruncateBytes() returned bytes different from TruncateString() for %q: %q != %q", text, tb, ts)
}
}
})
}
Loading