Keyboard woes

[shermp]

I've spend the last couple of days trying to see if I could figure out how to do some sort of text input. I'm afraid I haven't gotten very far.

Getting a QLineEdit, or even a TouchLineEdit (which inherits QLineEdit) onto confirmation dialog is easy. But then I've got to try and attach a keyboard to it. And this is where I've gotten all in a muddle and a tangle.

There looks to be so many different pieces to the puzzle. KeyboardController (and ExtendedKeyboardController and SearchKeyboardController). KeyboardFrame. KeyboardReceiver. VirtualKeyboard. And I'm not really sure how they all interact with each other.

This is partially what I THINK might happen:

• A KeyboardReceiver wraps a QLineEdit or QTextEdit (there's a constructor for both). I'm not really sure what this does, but I think it's the glue that connects a text edit widget with the keyboard
• A KeyboardController manages a keyboard? It seems to be the derived class SearchKeyboardController that is mostly used. They have a method to set a KeyboardReceiver
• There's a factory to create a SearchKeyboardController
• And then there's KeyboardFrame. It has a method called createKeyboard that invokes the localizedKeyboard method of the above factory.

And yet, after all this digging, I STILL haven't managed to show a *****y keyboard on screen!

Has anyone (cough @pgaskin ) looked into how the keyboard system works? And how one might use it?

[shermp]

Success!!! (Kinda, sorta, maybe)

There's still a lot of issues, but something actually works. As you can see, there's lots of rendering or positioning issues. The accept or close buttons didn't work, which makes me think the keyboard may be a modal. And worst of all, it sometimes segfaults when trying to open that editbox.

But I can type text into a TouchLineEdit 😀

[shermp]

I did it! I have an initial (seemingly) working implementation for allowing user text input.

So, it turns out that the ConfirmationDialog object already constructs a KeyboardFrame, so I could find and use that (thanks findChild). It was a matter of calling it's createKeyboard method, creating a TouchLineEdit, wrapping it in a KeyboardReceiver and hooking the KR with the keyboard.

Note to all: If creating a widget with calloc and manually calling its constructor, then adding it do another widget, then one MUST de-parent the child widget before Qt attempts to delete the parent, otherwise segfaults occur.

There's still work to be done. I'm sure I've made mistakes somewhere, and the code is going to need to be looked at closely by others before I'm comfortable merging it into master. But I've made a good start.

And here's the result so far:

[root@kobo ~]# qndb -s dlgConfirmTextInput -m dlgConfirmLineEdit "Input Example" "Input"

dlgConfirmTextInput Hello World!

[shermp]

Complete refactor of the confirmation dialog system, by spinning it off into it's own class? Check.

Multi-line text editing support? Check.

There's probably enough here now to create the most rudimentary of file editors...

[shermp]

Behold! What follows is the worlds crappiest script for the crappiest file editor you'll ever see. As an extra bonus feature, it even eats line endings on saving the document back!

Works with the current WIP branch https://github.com/shermp/NickelDBus/tree/text-edit

WARNING: THE FOLLOWING SHELL SCRIPT IS FOR PROFESSIONALS ONLY. PLEASE DO NOT TRY THIS AT HOME.

#!/bin/sh

strip_qndb_prefix() {
    printf "%s" ${1#dlgConfirmTextInput }
}

result=$(qndb -s dlgConfirmTextInput -m dlgConfirmLineEdit "Choose File" "Open")
filename=$(strip_qndb_prefix "$result")

if [ -f "$filename" ]; then
    filecontents=$(cat "$filename")
    edit_result=$(qndb -s dlgConfirmTextInput -m dlgConfirmTextEditSet "Editing" "Save" "$filecontents")
    edit_content=$(strip_qndb_prefix "$edit_result")
    printf "%s" "$edit_content" > "$filename"
fi

[shermp]

I'm not sure what to do about international keyboards, whether I should bother or not (I've currently hard-coded it to show the English KB). Two parameters are needed, a QLocale (easy) and something called a KeyboardScript, which is a lot stranger, and I haven't been able to figure out.

It seems to be treated like an integer, so I assume it may be an enum. What is weird is what happens to that parameter. In the decompilation for `SearchKeyboardControllerFactory::localizedKeyboard():

uVar2 = count_leading_zeroes(param_2 - 1);
uVar2 = uVar2 >> 5;

Which corresponds to the following assembly:

        00a68e6a a8 f1 01 08     sub.w      r8,r8,#0x1
        00a68e6e 3b 28           cmp        param_1,#0x3b
        00a68e70 b8 fa 88 f8     clz        r8,r8
        00a68e74 4f ea 58 18     lsr.w      r8,r8,#0x5

uVar2 is then passed as parameters to the respective SearchKeyboardController* constructors.

Ghidra seems to not have any further information on what a KeyboardScript type is, or where it comes from. I wonder if it comes from an external library perhaps?

(Note, I'm currently just passing 0 to the functions in the absence of any further information. Seems to work alright for the English keyboard at least.)

[shermp]

Well I figured out how to use ConfirmationDialogFactory::showTextEditDialog() in combination with a N3ConfirmationTextEditField. It means no messing around with KeyboardFrame and SearchKeyboardController and KeyboardReceiver.

The consequence is dropping support for multi-line text editing, as N3ConfirmationTextEditField essentially wraps a QLineEdit and there doesn't appear to be a QTextEdit equivalent that I can find so far.

I'm not too upset though, I was feeling that the previous code was probably a bit fragile, and was rather complicated, and relied on a lot of libnickel symbols. Multi-line editing was probably approaching feature-creep anyway.

Also, as a sidenote, I learned it's possible to call ::operator new() as a function, like one would call malloc(). Doing so allocates memory without calling the constructor. Crucially, it appears to allow Qt to delete said objects without segfaulting.

[pgaskin]

Has anyone (cough @pgaskin ) looked into how the keyboard system works? And how one might use it?

Not really. I looked into it a bit to update the keyboard patches and to look into how touch works, but I haven't done much with it.

I did it! I have an initial (seemingly) working implementation for allowing user text input.

Behold! What follows is the worlds crappiest script for the crappiest file editor you'll ever see. As an extra bonus feature, it even eats line endings on saving the document back!

Nice!

Ghidra seems to not have any further information on what a KeyboardScript type is, or where it comes from. I wonder if it comes from an external library perhaps?

About KeyboardScript, that's used in _ZN31SearchKeyboardControllerFactory17localizedKeyboardEP7QWidget14KeyboardScriptRK7QLocale, where it's initially in r1 since the function is static. As of 15875, it ends up casted as-is to a SearchKeyboardController::LocaleMode and passed to any SearchKeyboardController subclass's constructor which takes it as the second parameter. Ultimately, it always ends up being used in SearchKeyboardController itself only, where it is stored in the variable at offset 0x3c. From here, it's used by _ZN24SearchKeyboardController17updateImeReceiverEb, which tail-calls a subclass of ImeReceiver based on it.

Practically speaking, it's either an enum, function pointer, or function table offset (I've only skimmed the code and followed references, not actually done the math and/or hooking). I'll look into it more when I have a chance.

The consequence is dropping support for multi-line text editing, as N3ConfirmationTextEditField essentially wraps a QLineEdit and there doesn't appear to be a QTextEdit equivalent that I can find so far.

Yes, that's correct. Theoretically, it's possible to replace the edit field itself after creating it, then linking them together, but it's not worth the hassle or fragility. It's possible to make it more stable if we completely figured out GestureReceiver, but we haven't done that yet (and it'd still be more fragile than I'd be comfortable with).

Also, as a sidenote, I learned it's possible to call ::operator new() as a function, like one would call malloc(). Doing so allocates memory without calling the constructor. Crucially, it appears to allow Qt to delete said objects without segfaulting.

That's... odd. libstdc++ itself calls malloc directly for operator new. Some sanitizers will replace it with one which stores a count in the first byte and returns a pointer to the second, but that doesn't appear to be used in the Kobo libraries. Qt itself doesn't appear to make any changes for most objects.

Do you have the address where it segfaulted? Also, what address does a call to operator new and a call to malloc right before it return?

[shermp]

That's... odd. libstdc++ itself calls malloc directly for operator new. Some sanitizers will replace it with one which stores a count in the first byte and returns a pointer to the second, but that doesn't appear to be used in the Kobo libraries. Qt itself doesn't appear to make any changes for most objects.

Do you have the address where it segfaulted? Also, what address does a call to operator new and a call to malloc right before it return?

I think I may have been mistaken. The root cause of my issues I was having at the time was probably not making the size of the memory allocation large enough (I've since rectified that). Sure enough, just ran a test, and malloc works as well. I'll probably continue to use new though as to me semantically it makes sense for an object whose lifecycle is managed by Qt after its creation.

Practically speaking, it's either an enum, function pointer, or function table offset (I've only skimmed the code and followed references, not actually done the math and/or hooking). I'll look into it more when I have a chance.

It's probably an enum, that seems to be the sort of thing an enum would be used for, and would probably have the same concept as enum QLocale::Script. I've seen the scalar values 1 and 0 being passed to N3ConfirmationTextEditField for that parameter in Ghidra.