Just FYI: Failure on upgrade from 0.7 to 1.0 - non-string content passed through collation

[tjchambers]

Elixir 1.19.5 - Erlang OTP 28

When I updated my app to use the new 1.0.0 version and ran tests I had one failure I wanted to report just as an FYI. It passed on 0.7 of ex_cldr_collation, a fact which is unclear to me why it passed.

Original code snippet:

  defp sort(participants) do
    Enum.sort_by(
      participants,
      fn %{
           visited: visited,
           saved: saved,
           person: person,
           a: a,
           b: b,
           u: u,
           g: g
         } ->
        [saved, visited, a, b, u, g, person.name]
      end,
      Cldr.Collation.Insensitive
    )
  end

In the above code saved and visited are booleans, a,b,u,g are integers and person.name (the true intended target of the "case-insensitive" is a string which may contain any unicode characters. As you can deduce that name is the only true element that is intended as a target for the collation, the preceding all needing to be equal for that sorting to apply. Under 0.7 this did not emit a warning or error and functioned I believe as expected (rarely are the first 6 elements identical, so it may be incorrectly sorting, but there was no failure reported).

in tests run using Cldr.Collation 1.0.0 I get the following failure:

 ** (EXIT) an exception was raised:
          ** (FunctionClauseError) no function clause matching in Cldr.Collation.Table.lookup/1

          The following arguments were given to Cldr.Collation.Table.lookup/1:

              # 1
              true

          Attempted function clauses (showing 2 out of 2):

              def lookup(codepoint) when is_integer(codepoint)
              def lookup(codepoints) when is_list(codepoints)

          stacktrace:
            (ex_cldr_collation 1.0.0) lib/cldr/collation/table.ex:77: Cldr.Collation.Table.lookup/1
            (ex_cldr_collation 1.0.0) lib/cldr/collation/table.ex:161: Cldr.Collation.Table.longest_match/1
            (ex_cldr_collation 1.0.0) lib/cldr/collation.ex:391: Cldr.Collation.do_produce_full/3
            (ex_cldr_collation 1.0.0) lib/cldr/collation.ex:266: Cldr.Collation.build_sort_key/3
            (ex_cldr_collation 1.0.0) lib/cldr/collation.ex:193: Cldr.Collation.compare/3
            (elixir 1.19.4) lib/list.ex:515: anonymous fn/4 in List.keysort_fun/2
            (stdlib 7.1) lists.erl:1886: :lists.sort/2
            (elixir 1.19.4) lib/enum.ex:3348: Enum.sort_by/3
            (sct 2.4.0) lib/sct/emailing/web/og_invitation_live.ex:52: SctWeb.OGInvitationLive.mount/3

My inference, which may be inaccurate is that the argument true is from saved which is the initial and most major value being passed to sort_by via the anonymous function. It is very appropriate that this is the value true in my tests. I do not know what was done re: Collation with non string values in the prior releases of ex_cldr_collation, whether they were ignored, etc.

I changed the code to drop the reference to Cldr.Collation.Insensitive to a less appropriate but acceptable value of just down casing the person.name.

  defp sort(participants) do
    Enum.sort_by(
      participants,
      fn %{
           visited: visited,
           saved: saved,
           person: person,
           a: a,
           b: b,
           u: u,
           g: g
         } ->
        [saved, visited, a, b, u, g, String.downcase(person.name)]
      end
    )
  end

I only report all of the above for awareness. I know this is a change of MAJOR version making this a very appropriate time to change public behavior. I also see no reason to "fix" anything here specifically, although ignoring non-string values in some manor might be what the prior code did (I have not attempted to see what it did). I would say the sorting on multiple values using a list like this is not out of the ordinary in my code, and the types of those values may be mixed. In other areas I have interpolated into a single string with padding for alignment before using sort in order to mix non-string and string values (a bit of a pain, but...). In those cases I am therefore passing a single value - a string. I did not do that in the above, and if I found it was required I could adjust the above accordingly.

Having declared all of this I see 4 options in increasing degree of effort :) :

• thank me for the reported issue and close it
• Add where appropriate something somewhere in docs to highlight this difference in behavior and close issue
• Identify the prior behavior and how it was handled and document that as well as new behavior
• Make an update to 1.0.0 to handle this behavior in some way similar to prior behavior after understanding that

Thanks for all the work on all of the CLDR modules!

PS My only use of ex_cldr_collation is the Cldr.Collation.Insensitive functionality (extensively) during sorting. I use nothing else from this package.

[tjchambers]

I wanted to add that in other code I pass an atom as the first list element followed by person.name and that atom is an enum. That also failed as well in tests:

    ** (FunctionClauseError) no function clause matching in Cldr.Collation.sort_key/2

     The following arguments were given to Cldr.Collation.sort_key/2:

         # 1
         {:raw_seed, "Mike Holme"}

         # 2
         %Cldr.Collation.Options{strength: :secondary, alternate: :non_ignorable, backwards: false, normalization: false, case_level: false, case_first: false, numeric: false, reorder: [], max_variable: :punct, type: :standard, tailoring: nil, backend: :elixir}

     Attempted function clauses (showing 3 out of 3):

         def sort_key(input, options) when is_list(options)
         def sort_key(string, %Cldr.Collation.Options{} = options) when is_binary(string)
         def sort_key(codepoints, %Cldr.Collation.Options{} = options) when is_list(codepoints)

     code: conn = get(conn, @test_domain <> ~p"/sc/#{@sc_test_permalink}/opinion-sharers")
     stacktrace:
       (ex_cldr_collation 1.0.0) lib/cldr/collation.ex:240: Cldr.Collation.sort_key/2
       (ex_cldr_collation 1.0.0) lib/cldr/collation.ex:193: Cldr.Collation.compare/3
       (elixir 1.19.4) lib/list.ex:515: anonymous fn/4 in List.keysort_fun/2
       (stdlib 7.1) lists.erl:1886: :lists.sort/2
       (elixir 1.19.4) lib/enum.ex:3348: Enum.sort_by/3
       (sct 2.4.0) lib/sct_web/controllers/sc/align/opinion_gather_assign_controller.ex:30: SctWeb.OpinionGatherAssignController.new/2

My inference is that this is for a similar "reason" as above, albeit in different code.

[kipcole9]

Lots and lots (and lots!!!) of testing when the arguments are all strings. Not good negative testing on invalid arguments (or maybe valid, but need to be cast to strings).

Exacerbated by now having a native Elixir implementation and a NIF wrapper.

Have you had a chance to try with the NIF (which would be equivalent to the previous behaviour)?

I'm travelling for the next few weeks - still working hard on all of this, just quite irregular hours. So I won't have a chance to dive into it for a few hours yet.

[tjchambers]

Have you had a chance to try with the NIF (which would be equivalent to the previous behaviour)?

I am game to do that if I knew how. As I indicated there were no failures on 0.7.5 of this module. And there is no urgency re: me. So if somehow you could wtp share with me how to reconfigure would be happy to do the legwork.

[tjchambers]

NB: With the adjustments to use String.downcase for now I am deploying in production with 1.0.0.

[kipcole9]

Since Localise.Collation.Insensitive is a compatibility module I think the right behaviour - for it - it to apply to_string/1 on the arguments. Will work on that, with tests, in my current morning (UTC+1)

As always, thanks for reporting this. Never, never hesitate to report and issue - whether its a bug, a usability issue or a feature request.

[tjchambers]

Thanks - will be interested in how the sort treats true/false and enums. Also of course integers like "123" vs. "99".