Invalid character reference cleanup fails on documents containing non-ASCII

[msouchon]

KDSoapMessageReader.cpp has a recovery path that strips invalid numeric character references and re-parses. It doesn't work if the response contains any non-ASCII characters before the bad reference.

xmlToMessage() passes reader.characterOffset() to handleNotWellFormedError(), which uses it to index a QByteArray:

qint64 i = offset - 1;
while (i >= 0 && data.at(i) != '&') {
    if (data.at(i) == '<') {
        return dataCleanedUp; // empty -> no retry
    }

characterOffset() counts decoded characters; data.at() indexes bytes. Multi-byte UTF-8 earlier in the document makes the byte position larger than the reported character position, so the backward scan starts too early, hits an earlier <, and returns empty. xmlToMessage() then skips the retry and produces a fault instead of recovering.

I'm happy to look at implementing a solution with some guidance as to what would be preferred.

1. Ignore the offset. Scan the buffer forward for &#…; and test each candidate with isInvalidCharRef(). (I did this in a fork and am currently using it now.)
2. Convert the offset. Walk the buffer to translate character offset to byte offset before scanning. Has to account for what characterOffset() counts (UTF-16 code units), CRLF normalization, and non-UTF-8 encodings.
3. Parse from a QString. QXmlStreamReader reader(QString::fromUtf8(data)) and do the repair in character space, so the offset lines up natively.
4. Sanitize before parsing. Filter illegal characters and references in a single pass up front, so the recovery path is never needed.

Separately, in isInvalidCharRef(): the test is val <= 0x20, which classifies 0x20 (space) as invalid, so   is replaced with ?. Should be val < 0x20. Doesn't really come up when using the offset, but I did need to correct it for the full scan in my fork.

Here is a unit test that demonstrates the issue: unittests/msexchange_wsdl/test_msexchange_wsdl.cpp (line ~500)

QTest::newRow("invalid 13 after non-ascii") << serverResponseXML.arg(QString::fromUtf8("\xc3\xa9\xc3\xa9\xc3\xa9&#x13;")) << request << QString::fromUtf8("subject \xc3\xa9\xc3\xa9\xc3\xa9?");

[dfaure-kdab]

Since this is a recovery path, I guess performance doesn't matter all that much, and parsing from a QString would be the most correct solution?

(Your point 2 says "non-utf8 encoding" and your point 3 says fromUtf8 -- don't we only care for UTF8 during xml parsing?)

[msouchon]

This is just an issue when receiving non-utf8 encoded text in a tag. When dealing with external servers fixing this isn't possible.

Here is an example of what the data could be and what could be done to strip the non utf8 characters:

#include <QByteArray>
#include <QDebug>
#include <QRegularExpression>
#include <QXmlStreamReader>

void parseXml(QByteArray ba) {
    QXmlStreamReader xml(ba);
    while (!xml.atEnd() && !xml.hasError()) {
        QXmlStreamReader::TokenType token = xml.readNext();

        if (token == QXmlStreamReader::StartElement) {
            if (xml.qualifiedName() == QStringLiteral("Value")) {
                QString content = xml.readElementText();
                qDebug() << "Content:" << content.trimmed();
            }
        }
    }
    if (xml.hasError()) {
        qDebug() << "XML Parse Error:" << xml.errorString();
    }
}

int main(int argc, char *argv[])
{
    QByteArray ba("<Value>                      € &#xE;ß$   38335     </Value>");
    parseXml(ba);

    QString xmlStr = QString::fromUtf8(ba);
    xmlStr.replace(QRegularExpression("&#x0?[0-8BCEF];", QRegularExpression::CaseInsensitiveOption), "?");
    QByteArray baCleaned = xmlStr.toUtf8();
    parseXml(baCleaned);
}

Output:

Content: ""
XML Parse Error: "Invalid character reference."
Content: "€ ?ß$   38335"