Make it easy to crop image pixels or update images by character cells.

[amano-kenji]

Many terminal UI libraries would prefer to just display images in given character cells(for example, 30 columns 20 rows) and update images cell by cell.

Is it going to be inefficient to draw an image for the first time cell by cell? Updating an image cell by cell could be efficient.

To manipulate images cell by cell, programmers would need to know how many image pixels fit in a character cell horizontally and vertically. Physical pixels of a terminal emulator may not have the same size as image pixels.

Haskell's vty library only knows character cells. It doesn't know image pixels. If vty was able to draw an image cell by cell efficiently, it could support images easily. Because vty has to know what each character cell contains in order to know which character cells to actually update, it wouldn't use alpha channel in image pixels. For example, drawing a partially transparent image character cell over a text character cell would complicate the computation of whether a character cell needs to be updated.

[dankamongmen]

i'd disagree with this. Notcurses is, as far as i'm aware, the only terminal ui library with reasonable bitmap support. it absolutely does not want to restrict itself to character cells at a time, and does not require this from users, who can feed it an arbitrary bitmap and arbitrary pixel coordinates, and have this interact sanely with text.

it goes to fairly extreme lengths to make this work, but it works well (see https://www.youtube.com/watch?v=dcjkezf1ARY)

https://github.com/dankamongmen/notcurses/blob/8f972a3e0e638191e143845c5b980bb17c6936c6/src/lib/termdesc.h#L112 <-------------- some of the pixel information notcurses derives regarding the terminal

[AnonymouX47]

@amano-kenji I'm in support of and can testify to your first statement:

Many terminal UI libraries would prefer to just display images in given character cells

but not the rest.

Drawing graphics cell-by-cell might seem very attractive and useful to TUI developers but will be tedious for terminal emulator developers.

This will require encoding/compressing (on the application's end) and decoding/decompressing (on the terminal emulator's end) pixel data cell-by-cell, which is a really bad idea considering the encoding/compression time on the application's end, amount of memory used on the application's end (if storing the escape sequences), transmission time and decompression/decoding time on the terminal emulator's end.

At least, from my little experience, I've encoded images line-by-line and I can say that the difference between that and encoding images as a whole is quite clear. Some terminal emulators barely cope with the "line-by-line" images (though IMO that is mostly due to their bad design) e.g iTerm2.

[amano-kenji]

it absolutely does not want to restrict itself to character cells at a time

Okay. But, it's good to be able to fill character cells fully by knowing how many image pixels fit in a charater cell vertically and horizontally.

[amano-kenji]

For TUI libraries, cropping images by character cells is going to be important. To do that, they need to know how many image pixels fit in a character cell.

[AnonymouX47]

It may actually be the place of a graphics protocol to specify a means for this but there are already a couple known methods:

• TIOCGWINSZ ioctl command
  • The fastest but not always properly/fully implemented. So the size in pixels may not be correct on some terminal emulators.
  • The only problem with this is, it's not specified which dimensions are actually reported (i.e text area only or text area + chrome) and various terminal emulators mix this up.
  • From my experience, all terminal emulators that actually implement graphics protocols probably support this and give correct values.
• Xterm's XTWINOPS (Extension of ddterm's window manipulation controls; See CSI ... t here)
  • CSI 16 t (The most precise but least implemented)
  • CSI 14 t along with any means to determine the terminal size in cells and some division.
    • This is the most widely implemented but slower than TIOCGWINSZ since it requires querying the terminal.
    • Beware though, some terminal emulators mix this up with CSI 14 ; 2 t, which is very different.
    • Also, some terminal emulators wrongly swap the dimensions e.g Older VTE-based terminal emulators (but it seems to be fixed in recent versions).

It's possible for a graphics protocol to define an APC for this purpose but I'm not sure it's necessary.

EDIT: Added some points for completeness.

[amano-kenji]

The only problem with this is, it's not specified which dimensions are actually reported

Then, a graphics protocol may define TIOCGWINSZ precisely.

[amano-kenji]

There is a possibility that physical pixels of a terminal emulator may not be of the same size as image pixels.

[AnonymouX47]

Then, a graphics protocol may define TIOCGWINSZ precisely.

No, it can't because:

1. It's just not done. ioctl is a completely separate thing.
2. What if a terminal emulator implements multiple protocols?

The best that can be done IMO is to define an APC (Application Program Command; See ECMA-48 Section 8.3.2 Pg 33) or some other non-conflicting control sequence.

There is a possibility that physical pixels of a terminal emulator may not be of the same size as image pixels.

I don't understand 🤔. Care to explain?

[amano-kenji]

I don't understand. Care to explain?

https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/26#note_1770479

[AnonymouX47]

Oh, I see. You're talking about retina/HiDPI displays.

IMO, that's really for the terminal emulator to handle. As long as the terminal emulator uses the same dimensions (physical or logical) it reports to an application to process and render the pixel data sent by that application, it won't matter to the application which kind of dimensions the terminal emulator reports.

By the way, I had no idea that terminal-wg thread was still alive. Seems you woke things up :)

[amano-kenji]

The best that can be done IMO is to define an APC

Maybe, we should do this?

[AnonymouX47]

Why would you want to though?

Can you site one case where the existing methods do not suffice on a terminal emulator that actually implements a graphics protocol?

[amano-kenji]

You are the one who said

The best that can be done IMO is to define an APC

I want the best

[AnonymouX47]

I meant the best, if it had to be done.

Remember what I also said before that:

It's possible for a graphics protocol to define an APC for this purpose but I'm not sure it's necessary.

[amano-kenji]

haskell vty library records what each character cell contains in order to know which character cells to update.

For images, each rendered character cell would have to contain the hash of image pixels in the character cell or the entire image.

Should it be the hash for image pixels in a character cell or an entire image?