Skip to content

Special UTF-8 characters not compared correctly #3071

Closed
#3568
Closed
Special UTF-8 characters not compared correctly#3071
#3568
Labels
plugins
Milestone
v2.16.59

Description

@FoliniC
FoliniC
opened on Dec 11, 2025

I have two UTF-8 files with windows newline.
When the same special unicode character is present in the files it's viewed as different.
Example:
Unicode name: THERMOMETER.​
Full emoji code points: U+1F321 U+FE0F (the colored emoji form 🌡️).​
Text-only base variant: U+1F321 (🌡).​
In UTF‑8 the sequence is F0 9F 8C A1 EF B8 8F.

The following line is marked as different:
thermostat: { title: 'Thermostat', desc: 'Schedule hourly temperatures for heating/cooling', icon: '🌡️' },

removing the special unicode character remove the difference

Changing encoding of files (say windows 1252) wrongly translate the characters, but are displayed as equal (visual compare).
Winmerge shows them as different
Image
removing the last character give a correct comparison

Image

Activity

  1. sdottaka commented on Dec 11, 2025

    @sdottaka
    sdottaka
    on Dec 11, 2025
    Member

    Are you comparing a file containing U+1F321 + U+FE0F with a file containing only U+1F321?
    If so, WinMerge detects U+FE0F as a difference, and there is currently no option to ignore this character.

    If both files contain U+1F321 + U+FE0F, but WinMerge still shows a difference, please try File → Recompare As → Binary and check the hex view to see whether the bytes actually match.
    If possible, please attach the two files here so we can investigate further.

    Also, WinMerge can display colored emoji.
    Please set Options → Editor → General → Rendering mode to one of the DirectWrite modes.

  2. FoliniC commented on Dec 15, 2025

    @FoliniC
    FoliniC
    on Dec 15, 2025
    Author

    Hi Takashi,
    sorry for the late response... but it was tricky to get the repro.
    It didn't matter the enconding it seems a problem with rem lines.
    Repro:
    Compare two directories with the attached file present only in one.
    Compare this file, on the side with no content paste following 3 lines
    /** Comment line */
    another line
    last line

    Refresh compare. The first line should be marked as different, but it's not.

    Looking into prediffer I see that I have "ignore Comments" enabled..
    So it should be a prediffer's problem. Disabling it seems to work flawlessly

    Image

    Repro image

    FileWithComments.js

  3. sdottaka commented on Dec 15, 2025

    @sdottaka
    sdottaka
    on Dec 15, 2025
    Member

    Thank you for taking the time to isolate the issue and provide detailed repro steps.

    I couldn’t reproduce the problem using the attached FileWithComments.js. However, I’ll look into the IgnoreCommentsC prediffer plugin.
    If you find any additional conditions or a more specific repro case, please let me know

  4. sdottaka commented on Dec 16, 2025

    @sdottaka
    sdottaka
    on Dec 16, 2025
    Member

    To see which settings you are using, please attach the text file displayed when selecting Help → Configuration.

  5. sdottaka
    added
    stat:can't reproduce
    help wanted
    plugins
    on Dec 31, 2025

  6. KernelLeak commented on Aug 21, 2026

    @KernelLeak
    KernelLeak
    on Aug 21, 2026

    I've just now managed to hit this exact bug, here's a to reproduction recipe:

    • Open a new compare tab with "File > New > Text"
    • Enter/paste the same text containing umlaut characters on both sides, like 
      Bavaria ipsum dolor sit amet trihöleridi dijidiholleri auf gehds beim Schichtl Gaudi, noch da Giasinga Heiwog hogg ma uns zamm. Kaiwe Almrausch wia da Buachbinda Wanninger Steckerleis, Wurscht! Biawambn Obazda g’hupft wia gsprunga Weißwiaschd scheans, Brezn nois Griasnoggalsubbm Weibaleid.

Hea de Sonn auf der Oim da gibt’s koa Sünd, aau di hera, samma meara. Vasteh oans barfuaßat a bravs von Kneedl Watschnbaam fias. Naa jedza no a Maß Gamsbart wia Ohrwaschl, Biagadn obfrettn.

Ja mei naa Auffisteign Brezn Ramasuri, nimma da Gschicht Graudwiggal. Glacht damischa is ma Wuascht Schuabladdla. Pfenningguat des muas ma hoid kenna aau di hera, samma meara boarischer iabaroi und glei wirds no fui lustiga, unbandig.

— da Blindtext kimmt von bavaria-ipsum.de
      (nonsensical lorem-ipsum-style text generated using the generator mentioned in the last line)
    • prefix the last line (line 7) with "//" to make it a C-style comment, and press F5 to sync if necessary
    • notice only the last line being highlighted as different
    • activate "Plugins > Prediffer > Ignore Comments (C-Family Languages)"
    • notice that now also lines 1 and 3 (the only lines with umlauts in them) are highlighted as different, despite being exactly the same as before)
    • deactivate "Plugins > Prediffer > Ignore Comments (C-Family Languages)"
    • notice that we're back to only the last line being highlighted
  7. sdottaka
    closed this as completedin #3568on Aug 22, 2026
  8. sdottaka
    added this to the v2.16.59 milestone on Aug 22, 2026
  9. sdottaka
    removed
    stat:can't reproduce
    help wanted
    on Aug 22, 2026

  10. sdottaka commented on Aug 22, 2026

    @sdottaka
    sdottaka
    on Aug 22, 2026
    Member

    @KernelLeak

    Thank you for the detailed reproduction steps. This helped us identify the issue.

    The issue has been fixed in PR #3568.

    Since this requires a relatively large change, the fix is planned to be included in the 2.16.59 or 2.16.60 release, currently scheduled for October 2026.

    Thank you again for reporting this and providing such a clear reproduction recipe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    plugins

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions