Find a mostly automated way of updating the templates

[egmontkob]

The themeable templates, in src/js/mc-templates.js, are maintained manually.

This looks quite painful to me, and might have something do with discrepancies like #10 and MidnightCommander/mc#4893.

Also, mc's UI slightly changes over time, e.g. the order of items in the Left menu has changed, the format of the character code in the editor is likely to change soon, these currently don't cause any problem but might pile up over time.

Let's try to find a more automated way, to make it easy and convenient to create these themeable templates.

[egmontkob]

One idea:

Create a theme that uses a different color and a different special Unicode character for every role. Launch mc with this theme.

In GNOME Terminal or the standalone VTE test app, highlight (shift+mouse) the entire area and then right click -> Copy as HTML.

Postprocess this data (could be as easy as a few seds) to convert the actual colors and fancy Unicode characters back to their role name.

(Plus, document the steps that lead to the "screenshots" to be taken, have the special files tar'ed up in the repo or have a script that creates them, include the files shown in the viewer / editor / diffviewer in the repo etc.)

---

I understand not every mc maintainer has convenient access to GNOME Terminal or VTE, so this might not be the ideal solution, but it's the one that occurred to me now and is reasonably easy to implement.

We might look into other possible ways of dumping a terminal window's contents, i.e. check if screen or tmux has such a feature, or if maybe ncurses or slang has such a hook, or look at libvterm (a headless terminal emulator library)...

[egmontkob]

Another possibility:

For every role, define a different truecolor background color.

Create a png screenshot in any terminal, manually crop to the exact terminal area only. Pass it, along with the terminal size in characters (stty size), to a simple image processing utility. It could, for each cell, scan let's say the edge pixels of that cell and return the most frequently found color. Then reverse lookup it to get the role.

As for the Unicode characters, we could parse a simple copy-paste of the terminal's contents (again: should work in almost every terminal).

[phplego]

Maybe Midnight Commander should have a dedicated hotkey that exports the current screen as an HTML template (with roles/classes for UI elements)?

[egmontkob]

I don't fully understand the widgeting system, but I believe the problem with this approach is that mc simply doesn't "know" how its entire screen looks like.

It knows how individual widgets look like and how they are stacked, plus there might be off-widget hacks (like the rotating dash painting over other widget's area). Also things like a horizontal line with some text in between might be implemented as drawing the entire horizontal line and then striking over the text's characters, in which case my previous sentence wasn't right either: mc doesn't really "know" how its individual widgets look like.

Composing the entire UI from these widgets is done by the screen drawing library (ncurses or slang), mc taking care not to ask ncurses/slang to refresh the screen (i.e. implement the contents of the buffers in the terminal) during these intermediate states. Sort of a double buffering.

mc knows the painting process, but not the result.

---

A minor issue compared to this: I don't think we have global shortcut keys, even Ctrl-L to refresh the screen doesn't work with the F9 menus opened. Each widget has its own keyboard handling. Now maybe this could be fixed to have a shortcut key that always works.

---

Maybe we should somehow catch mc's response output to a Ctrl-L, which is hopefully ncurses or slang painting the contents in regular writing order with no tricks whatsoever, and we could catch and postprocess that segment of the output flow?? And parse the stateful SGR transitions...

[egmontkob]

Maybe we could add a third, mock screen library in addition to ncurses and slang. Towards mc's widgets it would provide the same API.

It wouldn't know anything about SGR sequences and terminals, its backend would be dumping an HTML "screenshot".

For the sake of simplicity, I guess it wouldn't have to know anything about double wide characters or combining accents, at least for now I think we don't want to have any of them on the screenshots.

mc could run headless, a script injecting the keypresses required to get to the screens we want to screenshot. A problem might be that keyboard handling also goes through ncurses/slang to some extent.

Or mc could run with two backends in parallel (ncurses+mock or slang+mock), the real one handling the input. Then we could still freely choose whether we want to interactively get to the points we sceenshot, or script that.

Instead of a generic hotkey (if we don't want to implement or fix that), we could make it create a dump on SIGUSR1.

The feature would be hidden behind a nondefault configure option.

[zyv]

I'm afraid that the only principled approach would be to use mc itself as a skin parser and output generator. The downside is, obviously, you won't get live updates in the browser (unless you port mc to WebAssembly or provide an API, which runs mc on the input skins), which for me is the key advantage of having the skin browser in the first place.

A less principled approach would be to have mc export something semantic to what the browser can apply things to and do the final rendering of the skin, and so make mc-templates.js a bit less manual. This leads to a certain duplication of function, which might be deemed acceptable under the circumstances.

I would actually suggest maybe looking closer at this type of solution while trying to keep the needed changes to mc and the infrastructure around it to an absolute minimum. As appealing as it is to solve an interesting issue with lots of work and code, in the end, we have to consider how much practical problems the previously used approaches presented...

---

Also, you've probably seen the discussion in MidnightCommander/mc#2147, but I'll post these links to prior work nevertheless. This is actually the implementation of the first approach I mentioned (without live updates).

https://www.geek.co.il/~mooffie/mc-lua/docs/html/index.html

https://github.com/mooffie/mc/tree/skin_sampler/src/lua/misc/skin-sampler

[egmontkob]

I'm looking for a possible solution that is easy to implement, has a low fingerprint, and doesn't require to rewrite half of the existing stuff we have (I especially don't want to touch javascript, if possible).

I do see a great power in being able to modify the skin and see the result immediately, without having to run mc. I mean: if one can run mc then there's not much point in a separate skin viewer, just run mc with the skin you're working on :) So I'd like to keep the current web viewing architecture essentially unmodified. Just introduce a component that lets us conveniently update the templates, rather than manually.

That's why I had the initial idea of "Copy as HTML" from a terminal emulator, simply postprocessed using some sed magic.

But now I'd prefer the "mock screen library" approach. I think it should be easily doable. Let me give it a try, I'm curious...

[zyv]

I'm looking for a possible solution that is easy to implement, has a low fingerprint, and doesn't require to rewrite half of the existing stuff we have (I especially don't want to touch javascript, if possible).

I think we're on the same page here, just wanted to make sure though... and somehow I find the proxy screen library idea a bit scary, but clearly too much semantics are lost at the "Copy as HTML" stage. Also, it isn't obvious to me whether the screen library layer actually knows about the semantic colors and not only their values, but you've probably checked that already.

[egmontkob]

Also, it isn't obvious to me whether the screen library layer actually knows about the semantic colors and not only their values, but you've probably checked that already.

I figured it out as I was working on it... I have a version now that works quite well (colors only, no fancy Unicode chars).

The patch is a bit scarier than I anticipated; maybe we don't want to mainstream it, just leave it lying around as a patch that we update sometimes. We'll see.

So yup we do have a cache, identical colors receive the same technical color pair ID. I have to skip this cache so that they get different numbers. Plus I have to add a new reverse mapping from color pair ID to their skin file keywords.

[zyv]

So for your information, right now, whenever there is a change in the master branch of the mc repository, it triggers a workflow in this repository which fetches the latest code of mc and pushes the skin browser along with the skins out to the web.

Obviously, an extra patch can be stored in this repository and a step to apply it, build mc and generate the templates can be trivially added to the workflow here. My only worry is that this will keep breaking all too often due to changes in mc master.

But maybe some middle ground can be found as in upstreaming the minimal part of it which is most likely to drift, and keep the rest here or something like that.

[egmontkob]

Obviously, an extra patch can be stored in this repository and a step to apply it, build mc and generate the templates can be trivially added to the workflow here.

There'd still be a missing step: to automatically drive mc to the point where these "screenshots" can be taken. That would require to set up a part of the filesystem to contain the desired types of files (including devices which only root can create -- or somehow mock that), and then to start mc in a (perhaps headless) terminal emulator of the desired size and inject the necessary keypresses to get to each desired screen.

Not that it's hopeless to do it, but right now this is not a goal for me. I'm super happy if a maintainable (albeit somewhat ugly) patch plus a few easy manual steps can replace the manual editing of the template files.

Once it's done, we can further automate the workflow as we see desired.

[egmontkob]

and then to start mc in a (perhaps headless) terminal emulator

No need for an actual terminal emulator that can parse escapes sequences. Just a forkpty(), and a parent process that reads and throws away all the incoming data should do it.

And it would inject to mc's stdin the hardwired escape sequences of the keys we'd press.

Sounds pretty easy to automate this step.

[egmontkob]

Here's yesterday's early and rough draft. (As exciting as this issue is, I'm trying to return to more important stuff now, will get back to this later.)

mc-skins-11-draft1.patch

Goes on top of 3158 / 4813. Need to compile with slang.

When mc is idle, send it a SIGUSR1. In response, it creates a /tmp/mc-screenshot-<timestamp>.html. (If you send SIGUSR1 while mc is busy doing stuff, it may crash, the signal handler is absolutely not async signal safe :))

This file, apart from the <pre> </pre> tags, is a drop-in replacement for the corresponding lines in src/js/mc-templates.js.

Only the colors are templated, the Unicode characters are currently hardcoded, that'll be the next step to address (and I'm afraid will be an even uglier hack than the colors).