IIS log parser (iis.py) ignores valid IIS log entries containing characters outside the "URI Safe characters" list - potential fix included

[J-A-Sec]

Describe the problem:

I attempted to create a timeline from IIS logs containing web app exploitation attempts, including SQL injection. The log file was ~36,000 lines, but Plaso only parsed ~8,000 lines.

To Reproduce:

The version of Plaso you used:

20250918

The operating system you are running Plaso on:

Ubuntu 24.04

Steps to reproduce the behavior including command line and arguments and output:

Run the command line below against the IIS log file:
log2timeline --storage-file /path/to/storagefile.plaso /path/to/inputfile

Please provide the source data you used when you experienced the problem. For publicly available data please provide a URL or path of the source data.

I cannot provide the entire source file, but here are some sample lines that have been anonymised:

2026-01-17 04:35:51 10.0.0.1 GET /~login - 443 - 12.34.56.78 curl/7.54.0 - 302 0 0 41
2026-01-17 19:22:16 10.0.0.1 GET /Page%'+AND+2*3*8=6*8+AND+'oLjH'!='oLjH%/CookiePolicy - 443 - 12.34.56.78 User-Agent:+Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+360SE) https://webapp.example.com 302 0 0 109
2026-01-17 19:22:08 10.0.0.1 GET /(select+198766*667891)/plugins/ckeditor/ckeditor.js - 443 - 12.34.56.78 User-Agent:+Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+360SE) - 302 0 0 118
2026-01-17 19:22:15 10.0.0.1 GET /Page'+AND+2*3*8=6*8+AND+'kH4r'='kH4r/CookiePolicy - 443 - 12.34.56.78 User-Agent:+Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+360SE) https://webapp.example.com 302 0 0 109
2026-01-17 19:22:45 10.0.0.1 GET /bundles0'XOR(if(now()=sysdate(),sleep(15),0))XOR'Z/cookieConsent.js - 443 - 12.34.56.78 User-Agent:+Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+360SE) https://webapp.example.com 302 0 0 107

The method you used to install Plaso:

Installed from [GiFT PPA][https://launchpad.net/~gift] stable track

Expected behavior:

It is expected that all log entries are parsed, no matter which characters are in the URI fields.

Debug output/tracebacks:

Example 19 of 488 extraction warnings:
**************************** Extraction warning: 19 **************************** Message : unable to parse log line: 5409 "2026-01-17 19:22:16 10.0.0.1 GET /Page%'+AND+2*3*8=6*8+AND+'oLjH'!='oLjH%/..." Parser chain : text/winiis Path specification : type: OS, location: /path/to/inputfile.log

Additional context

Based on the local fix below, the issue appears to be that the "URI safe characters" defined in iis.py are not inclusive enough of the characters that IIS will accept (or at the very least, record in the log). Similar to the issue here: #1961

I was able to get Plaso to parse all ~36,000 IIS log lines by adding characters to the definitions of _URI, _URI_STEM and _QUERY in the iis.py parser:

{}|\^~[]`'"<>@$*!

The asterisk and exclamation mark were additions to the extra characters already defined in _QUERY.

How this looks in the code:

# Username can consist of: "domain.username", "domain\username",
  # "domain\user$", "domain/user", "user@domain" or "-" for an anonymous user.
  _USERNAME = pyparsing.Word(pyparsing.alphanums + '-.\\$@/') | _BLANK

  _URI_SAFE_CHARACTERS = '/.?&+;_=()-:,%'

  _URI = pyparsing.Word(pyparsing.alphanums + _URI_SAFE_CHARACTERS + '{}|\\^~[]`\'"<>@$*!') | _BLANK

  _URI_STEM = (pyparsing.Word(
      pyparsing.alphanums + _URI_SAFE_CHARACTERS + '{}|\\^~[]`\'"<>@$*!') | _BLANK)

  _UA = pyparsing.Word(
      pyparsing.alphanums + _URI_SAFE_CHARACTERS + '[]') | _BLANK

  _COOKIE = pyparsing.Word(
      pyparsing.alphanums + _URI_SAFE_CHARACTERS + '@{}"\\') | _BLANK

  # Per https://blogs.iis.net/nazim/use-of-special-characters-like-in-an-iis-url
  # IIS does not require that a query comply with RFC1738 restrictions on valid
  # URI characters
  _QUERY = (pyparsing.Word(
      pyparsing.alphanums + _URI_SAFE_CHARACTERS + '{}|\\^~[]`\'"<>@$*!') |
           _BLANK)

[joachimmetz]

Related issue #4921

[J-A-Sec]

I agree that the whack-a-mole approach is not ideal. I've found further instances of logs not fully parsing, I just haven't had chance to isolate the problem characters yet.

[joachimmetz]

if you can collect the different edge case that might be interesting

I put together https://github.com/libyal/dtformats/blob/main/documentation/Microsoft%20IIS%20W3C%20extended%20log%20file%20format.asciidoc a while a go to track observations (needs some updating with the findings in #4921)

[Revo1212]

Some edge cases from testing against main.

The rough numbers in the first post point somewhere other than the character
sets. 8,000 parsed plus 488 warnings accounts for about 8,500 of 36,000 lines,
and the remaining 27,500 produced neither an event nor a warning. That is what
_ParseLines does once _MAXIMUM_CONSECUTIVE_LINE_FAILURES is exceeded: it
breaks out of the loop and the rest of the file is never read. Injection attempts
arrive in bursts, so 21 consecutive rejected lines are enough to drop everything
behind them. Widening the character set does fix it, but only because the burst
stops forming.

Subtracting what _QUERY accepts from printable ASCII leaves exactly !, #
and *. The patch in the first post adds ! and * to _URI, _URI_STEM and
_QUERY but not #, and leaves _UA alone, so
sqlmap/1.8.3#stable+(https://sqlmap.org) still fails on the user agent field.
That is sqlmap's default UA, so every request from an unmodified run is a
rejected line, which is how the burst above forms in the first place.

On updating the doc with the #4921 findings, I would be careful with the
characters listed there. ç is UTF-8 ç (C3 A7) read as cp1252, and À¯ is
C0 AF read as cp1252, the overlong UTF-8 / from ..%c0%af... Those are not
characters a client sent, they are what a log looks like when it is decoded with
the wrong code page. IIS writes UTF-8 only when logInUTF8 is set and otherwise
the server's ANSI code page, which the file does not record anywhere. The format
section might want a note on that; it currently only covers percent-encoding.

Separately, _IIS_6_0_LOG_LINE binds _URI to cs_uri_stem and cs_uri_query
rather than _URI_STEM and _QUERY, so on that path a single ' fails a line.
Logs land there more easily than it looks: _ParseHeader requires its match at
offset 0, so a #Fields written further down the file, which IIS emits after a
configuration change, is never bound.

Re whack-a-mole, agreed. Is enumeration the right axis though? The character set
is whatever a client chooses to send, but the delimiter is one space and the
field count comes from #Fields. Would CharsNot(' \t\n') on the URI fields,
with validation moved off the grammar, be worth trying? I can put that together
and report what it breaks, or send the doc notes above as a PR to dtformats.

[joachimmetz]

Re whack-a-mole, agreed. Is enumeration the right axis though?

Sure a report of observations and sample lines/files of edge cases could be beneficial. This can help decide on next steps (to improve parsing strategy)

[Revo1212]

Attached, one behaviour per file, with a README of the counts and the scripts
that produced them. Numbers are from plaso 20250918. I diffed iis.py,
text_parser.py, text_plugins/interface.py and engine/extractors.py against main
first: main has been reformatted since, but every constant and path below is
unchanged.

Correction to my earlier post before anything else. I had logInUTF8 backwards.
It defaults to true from IIS 7 on
(https://learn.microsoft.com/en-us/iis/configuration/system.applicationhost/log/),
so an ANSI log means IIS 6, or a server where someone turned it off, and not the
general case. The comment above ENCODING in iis.py still describes the older
default.

The encoding case surprised me anyway, for where it fails rather than what it
rejects. ansi-late.log has a cp1252 ü in a URI at offset 80101 and plaso
parses the file: 428 of its 429 requests become events, with one warning for the
line it dropped. ansi-early.log is the same log, shortened so the byte falls at
offset 40200, and its 220 requests produce nothing at all. Selection decodes only
the first 64 KiB (EncodedTextReader.BUFFER_SIZE), the UnicodeDecodeError there
leaves result = False, and with nothing matching, WrongParser is raised. That
path produces no extraction warning of any kind, so the file is skipped in
silence.

ENCODING is utf-8, so --codepage does not reach this either. cp1252, cp932 and
utf-8 all fail the same way.

buffer_boundary.py walks one byte across that edge. 0xfc at 65535 loses the
file and at 65536 it parses. A byte in C2..F4 moves the flip one earlier, since
the stream reader holds it as a possibly incomplete sequence rather than raising.

One aside from measuring that. With only winiis enabled there is a second
warning, error decoding 0xfc at offset: 80101, which is absent from a normal
run. TextPlugin.init registers its codecs error handler process-wide under
the fixed name text_parser_handler, so the last plugin constructed owns it; with
everything enabled that is zsh_extended_history, whose _parser_mediator is None,
and the warning is dropped. The byte is still escaped, so event counts do not
change. That looks like the tracking half of #3301 not reaching the storage
file. Not IIS-specific, so I can open it separately if it is worth having.

utf8-uri.log is the same URI in a UTF-8 log. Selection succeeds and the grammar
rejects the line instead, since pyparsing.alphanums is ASCII-only. On the
character sets, subtracting what _QUERY accepts from printable ASCII leaves !,
# and *, which is query-bang.log, ua-hash.log and query-star.log.

abort-20.log and abort-21.log are one file either side of
_MAXIMUM_CONSECUTIVE_LINE_FAILURES. At 20 rejected lines the 5 requests that
follow still parse, at 21 they do not.

fields-midfile.log carries a second #Fields after the first data line, which is
what IIS writes when the logged fields change. _ParseHeader runs once, before the
line loop, so a later directive is never offered to it. That, and not the
offset-0 check I pointed at earlier, is why it is never bound.
iis6-fallback.log and header-quote.log are the same request with and without
a header: _IIS_6_0_LOG_LINE binds _URI to cs_uri_stem and cs_uri_query rather
than _URI_STEM and _QUERY, so one ' fails on the fallback path and parses with
a header.

record-stage.log has 40 lines with hour 24. They warn, produce no event and do
not increment the consecutive counter, so the 5 valid lines behind them are still
read. time-without-date.log declares time with no date field and no #Date.
There is no extraction warning there, but the timeliner does warn, unable to determine timestamp, and the event is stored with timestamp 0.

Happy to add cases for anything missing here, or to turn any of them into tests
if that is more use than a description.

[Revo1212]

iis-edge-cases.zip

[joachimmetz]

Thanks will take a closer look after the upcoming release