Encoding issue for "special" characters when applying "NormalizeNewLines" or "NormalizeSeparators" transformers

[szantotomi]

As an example here is a piece of dummy code i have:

*** Test Cases ***
Robotidy Normalizer test
   [Documentation]   Random piece of code that will break because of "special" characters:
  ...  Log "Хокей"
       Log   Хокей
       

When I apply either "NormalizeNewLines" or "NormalizeSeparators" transformers on this, python breaks with the following stacktrace:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Python311\Scripts\robotidy.exe\__main__.py", line 7, in <module>
  File "C:\Python311\Lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\rich_click\rich_command.py", line 19, in main
    return super().main(*args, standalone_mode=False, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\robotidy\decorators.py", line 30, in wrapper
    raise err
  File "C:\Python311\Lib\site-packages\robotidy\decorators.py", line 21, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\robotidy\cli.py", line 447, in cli
    status = tidy.transform_files()
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\robotidy\app.py", line 52, in transform_files
    self.output_diff(model_path, old_model, new_model)
  File "C:\Python311\Lib\site-packages\robotidy\app.py", line 126, in output_diff
    console.print(line, end="", highlight=False, soft_wrap=True)
  File "C:\Python311\Lib\site-packages\rich\console.py", line 1667, in print
    with self:
  File "C:\Python311\Lib\site-packages\rich\console.py", line 869, in __exit__
    self._exit_buffer()
  File "C:\Python311\Lib\site-packages\rich\console.py", line 827, in _exit_buffer
    self._check_buffer()
  File "C:\Python311\Lib\site-packages\rich\console.py", line 2023, in _check_buffer
    legacy_windows_render(buffer, LegacyWindowsTerm(self.file))
  File "C:\Python311\Lib\site-packages\rich\_windows_renderer.py", line 17, in legacy_windows_render
    term.write_styled(text, style)
  File "C:\Python311\Lib\site-packages\rich\_win32_console.py", line 442, in write_styled
    self.write_text(text)
  File "C:\Python311\Lib\site-packages\rich\_win32_console.py", line 403, in write_text
    self.write(text)
  File "C:\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 13-17: character maps to <undefined>

The output robotidy generates is as follows:

Loaded configuration from C:\repos\SportsRobotFrameworkTestAutomation\pyproject.toml
Transforming robotidy-bug.robot file
--- robotidy-bug.robot    before
+++ robotidy-bug.robot    after
@@ -1,7 +1,6 @@
-
 *** Test Cases ***
 Robotidy Normalizer test
-   [Documentation]   Random piece of code that will break because of "special" characters:

So it looks that processing the line with the "special" character fails and makes the script execution break completely

[bhirsz]

Thanks for the report, it indeed looks like a bug. We're processing the characters (I hope ;) I will add tests) but outputting the difference between original and formatted file fails. I will check it.

[bhirsz]

@szantotomi Can you share your platform and/or console you're using? On windows it does work but I see there could be potential problem - the default encoding for the printing output is taken based on the console type. It is possible it is using wrong encoding and instead we should enforce our encoding when printing.

I can do it easily but first I want to reproduce this issue in our tests to be sure it is fixed.

[szantotomi]

Hello! For me it's windows (10 enterprise), too.
Console I have used is the one that opens in VS Code by default. From the prompt I'd say it's a powershell.
About the file(s)m as i see they are in UTF-8 and using CRLF for EOL sequence
Currently have python 3.11.0 and RF 4.1.2 set for the repo

[bhirsz]

I've tested it on Powershell, VS Code terminal, with Python 3.11 and no luck. Also tested on min and max available versions of our dependencies (rich, rich_click). But I will keep investigating, it may be some other related settings that we don't share.