Creation of database from data backup fails

[gyps]

Dear all,

first of all I need to state that it's really great openMittsu exists an allows me to connect to my Threema groups without having to use propriatary smartphone OS'es! Thanks!

When I tried to import not only my old ID but also the data backup that goes with it openMittsu emits the following error:

Since I now created a new Identity and continue to use the old one on the smartphone, the issue is not really pressing but still I wanted to report it.

Cheers,
Alex

[blizzard4591]

Hi Alex,

please build the current master branch with debug flags (cmake -DCMAKE_BUILD_TYPE=Debug -DOPENMITTSU_DEBUG=On) and try importing the backup again.
The log should contain information on where this fails - my guess is reading a group or contact ID. If so, there should be logged message starting with "Trying to parse" before the exception happens. If you could send me the last couple of messages before the exception (redacted is fine), I should be able to understand what happens.

[srett]

Building from source on Debian 11, I cannot import my Android backup. I successfully built openmittsu and qt5-sqlcipher before using the instructions from the readme.

Import wizard error:
[...]ByteArrayConversions.cpp:18: Invalid size of input byte-array: 0 instead of 8 bytes.

Terminal:
QSqlDatabasePrivate::removeDatabase: connection 'openMittsuDatabaseConnection' is still in use, all queries will cease to work.

So, the Terminal message makes me assume this isn't a parsing issue, but let's try a debug build.

This time, it stops with a different error:
[...]ContactMessageType.cpp:54: Unhandled ContactMessageType string representation, this should never happen: VOICEMESSAGE

Terminal:

[2021-07-19 21:01:17.373] [main] [debug] [BackupReader.cpp:105] Will now read the contact message files...
[2021-07-19 21:01:17.378] [main] [info] Found 17 contacts message files.
QSqlDatabasePrivate::removeDatabase: connection 'openMittsuDatabaseConnection' is still in use, all queries will cease to work.
[2021-07-19 21:02:24.328] [main] [debug] [DatabasePointerAuthority.cpp:34] The database in this authority was NULL, but a weak reference was requested!
[2021-07-19 21:02:24.328] [main] [debug] [DatabaseWrapper.cpp:51] DatabaseWrapper was informed of new database, but we got nothing!
[2021-07-19 21:02:24.328] [main] [debug] [DatabasePointerAuthority.cpp:34] The database in this authority was NULL, but a weak reference was requested!
[2021-07-19 21:02:24.329] [main] [debug] [DatabaseWrapper.cpp:51] DatabaseWrapper was informed of new database, but we got nothing!

The timestamps show that there's no additional output, unfortunately, the weak reference stuff appeared when I closed the application. Any ideas?

[blizzard4591]

Okay, I am a bit mystified by this. Which Threema version are you using on Android?
I am using 4.55 Build 2000682.
I send myself a voice message, but in the related message_ID.csv in the backup, the voice message has type FILE, not "VOICEMESSAGE". I can see this string appear in recent Threema code, but I am not 100% on what this means.
So, either this is old, legacy stuff, or something recent I do not see a reason for doing...

I did some changes in 609e44e, please try this version with debug output enabled (cmake -DCMAKE_BUILD_TYPE=Debug -DOPENMITTSU_DEBUG=On) and check if this works. If not, the log should contain information on what exact file it was parsing when the error occurred.

[srett]

Now I'm getting the "0 instead of 8 bytes" error again.

Full (anonymized) log:

[info] Qt library load paths are: /usr/lib/x86_64-linux-gnu/qt5/plugins
[info] Starting OpenMittsu Version: 0.9.16 (+19 commits) build from revision g609e44e....
[debug] Creating QMainWindow instance.
[debug] [MessageCenterPointerAuthority.cpp:32] The MessageCenter in this authority was NULL, but a weak reference was requested!
[debug] [DatabasePointerAuthority.cpp:34] The database in this authority was NULL, but a weak reference was requested!
[debug] [DatabaseWrapper.cpp:51] DatabaseWrapper was informed of new database, but we got nothing!
[debug] [DatabasePointerAuthority.cpp:34] The database in this authority was NULL, but a weak reference was requested!
[debug] [DatabaseWrapper.cpp:51] DatabaseWrapper was informed of new database, but we got nothing!
[debug] [DatabasePointerAuthority.cpp:34] The database in this authority was NULL, but a weak reference was requested!
[debug] [DatabaseWrapper.cpp:51] DatabaseWrapper was informed of new database, but we got nothing!
[debug] [DatabasePointerAuthority.cpp:34] The database in this authority was NULL, but a weak reference was requested!
[debug] [DatabaseWrapper.cpp:51] DatabaseWrapper was informed of new database, but we got nothing!
[debug] [Client.cpp:216] OpenSSL context seems to work, could load certificate.
[debug] [DatabasePointerAuthority.cpp:34] The database in this authority was NULL, but a weak reference was requested!
[debug] [DatabaseWrapper.cpp:51] DatabaseWrapper was informed of new database, but we got nothing!
[debug] Starting event loop.
[debug] [Client.cpp:597] Updating contacts list on IdentitiesChanged() signal.
[debug] [Updater.cpp:61] Update query URL is https://update.openmittsu.de/update.php?major=0&minor=9&patch=16&commitsSinceTag=19&gitHash=g609e44e&os=Linux&bits=64&endianness=LITTLE_ENDIAN&channel=nightly.
[info] No new version available. The latest version online is: 0.9.16+9 @ g10ce35f (deb) in channel NIGHTLY.
[info] Using the crypto-database interface (QSQLCIPHER).
[info] SQLCipher Version reported as '3.4.1'
[debug] [BackupReader.cpp:59] Will now read the contacts.csv...
[debug] [ContactBackupObject.cpp:52] Trying to parse contact from data [....]
[info] Parsed 17 contacts from file.
[debug] [BackupReader.cpp:82] Will now read the groups.csv...
[debug] [GroupBackupObject.cpp:44] Trying to parse group from data [......]
[info] Parsed 3 groups from file.
[debug] [BackupReader.cpp:105] Will now read the contact message files...
[info] Found 17 contacts message files.
[info] Will now read contact message file message_xxx.csv.
[info] Will now read contact message file message_yyy.csv.
[info] Will now read contact message file message_zzz.csv.
[info] Will now read contact message file message_aaa.csv.
[info] Will now read contact message file message_*THREEMA.csv.
[info] Will now read contact message file message_bbb.csv.
[info] Will now read contact message file message_ccc.csv.
QSqlDatabasePrivate::removeDatabase: connection 'openMittsuDatabaseConnection' is still in use, all queries will cease to work.

message_ccc.csv is the last file being printed, but it doesn't even contain any VOICEMESSAGE. I grepped the whole backup and in fact, only message_yyy.csv contains one VOICEMESSAGE, and it seems that file was processed successfully. The timestamp of that entry is from the 14th of January this year. So maybe not even the root cause?
(EDIT: Slight correction, I only checked the message_* files, the group message files contain a bunch more VOICEMESSAGE entries, from both users with iPhone and Android.)

The backup was created on the Android Version from the Threema Shop, 4.54 Build 679.

Oh and as a side note, I was wondering (but didn't test): Is it necessary to use build type Debug when I want the additional logging from OPENMITTSU_DEBUG=On? It takes huge amounts of disk space, I'm a bit space constrained in my current setup...

[l1m3r]

I've got the same problem.

• openMittsu_0.9.16plus19-g609e44e-windows-x64-le-DEBUG.zip
• Windows 8.1
• Threema 2.56 Build 2000688 Google Play

Import Failed!
An Error occured while importing the data backup into database.
Problem: ...ByteArrayConversions.cpp: 18: Invalid size of input byte-array: 0 instead of 8 Bytes.

openMittsu.log

[...] [main] [debug] [BackupReader.cpp:59] Will now read the contacts.csv...
[...] [main] [debug] [ContactBackupObject.cpp:60] Trying to parse contact from data '...', '...
..........
[...] [main] [debug] [ContactBackupObject.cpp:60] Trying to parse contact from data '...', '...
[...] [main] [info] Parsed UVWXYZ contacts from file.
[...] [main] [debug] [BackupReader.cpp:82] Will now read the groups.csv...
[...] [main] [debug] [GroupBackupObject.cpp:51] Trying to parse group from data '...', '...
..........
[...] [main] [debug] [GroupBackupObject.cpp:51] Trying to parse group from data '...', '...
[...]

The log really ends with just one final time stamp ([...]) on the last line while openMittsu was in the middle of processing the groups.

But my backups are not 100% valid. For some reason the end of the zip file is corrupted/missing.
7zip-error: CRC-Error : Wrong Password? : ballot_vote.csv
(but I can import re-created/fixed backups in Threema)

[fionnb]

FYI: getting the same error from a fresh install of openmitsu r439.1988334-1 from AUR while importing a (full) backup from Threema 4.66 Build 3000720.