Shifted option for variable collation elements#19
Open
lucafavatella wants to merge 8 commits intojtauber:masterfrom
Open
Shifted option for variable collation elements#19lucafavatella wants to merge 8 commits intojtauber:masterfrom
lucafavatella wants to merge 8 commits intojtauber:masterfrom
Conversation
Extracted from [`settings` branch](jtauber@cf393b8) by James Tauber.
Extracted - with amendments and integrations - from `settings` branch by James Tauber: * Settings initialization: * Commit ["beginnings of settings capability"](jtauber@cf393b8); * Commit ["support for strength setting"](jtauber@fb87e92). * Documentation: * Commit ["initial settings documentation"](jtauber@8bd6a8c). * `normalization` setting: * Commit "support for normalization setting"](jtauber@6187dd5). Normalization test taken from UCA 5.2.0 conformance tests: ``` CollationTest/5.2.0/CollationTest_NON_IGNORABLE.txt:312:0332 0334; # ... ```
Extracted - with minor amendments in tests - from `settings` branch by James Tauber: * Code: * Commit ["support for strength setting"](jtauber@fb87e92). * Documentation: * Commit ["initial settings documentation"](jtauber@8bd6a8c). ---- Wrt the amended tests: According to [the algorithm for computing the sort key](https://www.unicode.org/reports/tr10/tr10-36.html#Step_3), the sort key does not have to end with a level separator. See also the following examples: * https://www.unicode.org/reports/tr10/tr10-36.html#Array_To_Sort_Key_Table * https://www.unicode.org/reports/tr10/tr10-36.html#Comparison_Of_Sort_Keys_Table However [the conformance tests](https://www.unicode.org/Public/UCA/10.0.0/CollationTest.html) include in (non-normative) comments "a representation of the sort key" always ending with "|]" where the vertical bar stands for "the ZERO separator". Do not append a final unnecessary level separator.
... and performing minor refactoring.
... i.e. algorithm step http://www.unicode.org/reports/tr10/tr10-36.html#S2.3 In order to achieve that, this commit extracts info on variable collation elements from allkeys data file as specified at https://www.unicode.org/reports/tr10/tr10-36.html#File_Format Description of conformance test is at https://www.unicode.org/Public/UCA/10.0.0/CollationTest.html Sample comparison of non-ignorable vs. shifted ordering is at http://www.unicode.org/reports/tr10/tr10-36.html#Variable_Weighting_Examples Intended meaning of combination of variable weighting (e.g. shifted) and strength is described in the following locations: * https://www.unicode.org/reports/tr10/tr10-36.html#Parametic_Tailoring * https://www.unicode.org/reports/tr35/tr35-collation.html#Common_Settings
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #18 (included in this PR).
Supersedes #15 #16 #17.
Notes:
I am a bit concerned that the conformance tests doc says
but tests succeed without having implemented the identical level at all... so the conformance tests currently do not really need the identical level.
What is more our tests calling the conformance tests keep strength to default, that is tertiary and not quaternary. And conformance tests pass - even the shifted ones. But based on common settings combinations doc
that suggests that the conformance tests for shifted do not test the punctuation... or the code actually handles the shifted ternary as quaternary (i.e. keeping L4).
I am pretty confident that some other language / language library already has Unicode tests further to the UCA conformance tests by Unicode. We should find and import such extra tests.