Adds more test cases for npm #638

prabhu · 2025-08-24T09:42:09Z

These additional tests were vibe-coded using cdx1-pro trained with the new enhanced purl schema definitions. The tests were manually reviewed, enhanced, and re-tested with the make check command.

Methodology

Technically, any frontier model or LLMs tuned with the new purl4ml repo can be used to generate decent-quality additional tests for PURL types.

Prompt

you are an AI tasked with improving the test coverage for PURL. For each purl type, I will upload an enhanced purl schema with detailed information and then the current tests in json format. Carefully understand the enhanced schema and share the updated test json by adding additional tests both positive and negative including edge cases. acknowledge that you understand this request so that I can start sharing the json schema and tests in this chat.

Screenshots

Signed-off-by: Prabhu Subramanian <[email protected]>

matt-phylum · 2025-08-25T13:07:46Z

tests/types/npm-test.json

+      "test_type": "parse",
+      "input": "pkg:npm/@angular/[email protected]",
+      "expected_output": null,
+      "expected_failure": true,


Is this true? Last I heard it was expected to parse because there was no instruction to specifically reject unencoded @ characters in positions where they could require encoding depending on the rest of the PURL.

purl-spec/types/npm-definition.json

Line 16 in 96d6f8c

"note": "The namespace is used for the scope of a scoped NPM package. The npm scope @ sign prefix is always percent encoded, as it was in the early days of npm scope."

That's what the note says, but:

I don't think it is possible for a type spec like NPM to alter the percent encoding behavior. Percent encoding is not type-specific.

Whether the @ sign is supposed to be encoded or not does not specify the behavior of parsing when it is not.

This is yet another example of the difference between the specification and the instructions for parser developers.
According to the proposed parsing algorithm, @ can be safely used as the test input. Of course the canonical form will need to encode the character as %40.

matt-phylum · 2025-08-25T13:08:32Z

tests/types/npm-test.json

+    {
+      "description": "Negative Test: An npm purl with an uppercase name should fail parsing",
+      "test_group": "advanced",
+      "test_type": "parse",
+      "input": "pkg:npm/[email protected]",
+      "expected_output": null,
+      "expected_failure": true,
+      "expected_failure_reason": "Should fail to parse due to uppercase characters in the name"
+    },


This is a valid PURL and must parse correctly as name "React" with an uppercase R.

These are tests based only on the contents in the type definitions. The PRs are drafted to encourage comments like these, so thanks a lot.

To me (and my bot), the test looks valid because there is a note that explicitly says the name must be lowercased.

purl-spec/types/npm-definition.json

Line 21 in 96d6f8c

"note": "Per the package.json spec, new package 'must not have uppercase letters in the name', therefore the name must be lowercased. The npm name used to be case sensitive in the early days for some old packages."

This has been a known issue with pkg:npm since 2021: #136.

Even following the incorrect spec, if the name is case insensitive, then parsing a name with mixed case should not result in an error.

matt-phylum · 2025-08-25T13:09:34Z

tests/types/npm-test.json

+    {
+      "description": "Normalization Test: An npm purl with an uppercase name should be lowercased on roundtrip",
+      "test_group": "advanced",
+      "test_type": "roundtrip",
+      "input": "pkg:npm/[email protected]",
+      "expected_output": "pkg:npm/[email protected]",
+      "expected_failure": false,
+      "expected_failure_reason": null
+    },


This test is wrong, and also conflicts with the previous test. The correct output is pkg:npm/LoDash.

Agreed. This normalization test must be rewritten in a different manner.

matt-phylum · 2025-08-25T13:11:08Z

tests/types/npm-test.json

+    {
+      "description": "Negative Test: An npm purl with a name starting with a dot should fail",
+      "test_group": "advanced",
+      "test_type": "parse",
+      "input": "pkg:npm/[email protected]",
+      "expected_output": null,
+      "expected_failure": true,
+      "expected_failure_reason": "Should fail to parse because the name starts with a dot"
+    },
+    {
+      "description": "Negative Test: An npm purl with a name starting with an underscore should fail",
+      "test_group": "advanced",
+      "test_type": "parse",
+      "input": "pkg:npm/[email protected]",
+      "expected_output": null,
+      "expected_failure": true,
+      "expected_failure_reason": "Should fail to parse because the name starts with an underscore"
+    },


Why should these fail? PURL does not (and IMO should not) define any rules to forbid this.

Good question. My bot includes information from additional sources.

https://github.com/npm/validate-npm-package-name/blob/main/test/index.js#L71
https://github.com/npm/validate-npm-package-name/blob/main/test/index.js#L91

Including name validation rules makes PURL implementations more complicated to write and more likely to be wrong, and PURL more likely to break as ecosystems evolve. I don't think this adds any benefit because whether somebody writes pkg:npm/_foo (invalid according to this test) or pkg:npm/a0103e9a-398d-4136-bced-c1cdbe95ebcd, neither PURL can be resolved to a package today.

matt-phylum · 2025-08-25T13:12:22Z

tests/types/npm-test.json

+    {
+      "description": "Negative Test: An npm purl with a unicode character (emoji) in the name should fail parsing",
+      "test_group": "advanced",
+      "test_type": "parse",
+      "input": "pkg:npm/foo%F0%9F%98%[email protected]",
+      "expected_output": null,
+      "expected_failure": true,
+      "expected_failure_reason": "Should fail to parse because the name contains characters not allowed by npm naming rules"
+    },
+    {
+      "description": "Negative Test: An npm purl with a unicode character (emoji) in the scope should fail parsing",
+      "test_group": "advanced",
+      "test_type": "parse",
+      "input": "pkg:npm/%40foo%F0%9F%98%83/[email protected]",
+      "expected_output": null,
+      "expected_failure": true,
+      "expected_failure_reason": "Should fail to parse because the scope contains characters not allowed by npm naming rules"
+    },


Again, PURL does not define validation rules to forbid this so the tests should not test for that validation.

npm does forbid, though. I agree that these shouldn't be in the purl-spec tests but must be present in SBOM tools at least.

matt-phylum · 2025-08-25T13:13:06Z

tests/types/npm-test.json

+    {
+      "description": "Normalization Test: Both scope and name with uppercase letters are lowercased",
+      "test_group": "advanced",
+      "test_type": "roundtrip",
+      "input": "pkg:npm/%40MyScope/[email protected]",
+      "expected_output": "pkg:npm/%40myscope/[email protected]",
+      "expected_failure": false,
+      "expected_failure_reason": null
+    },


This test makes the previous (incorrect) test about lowercasing the name redundant.

matt-phylum · 2025-08-25T13:14:49Z

tests/types/npm-test.json

+    {
+      "description": "Positive Test: An npm purl with a subpath containing unicode characters (emoji) should parse correctly",
+      "test_group": "advanced",
+      "test_type": "parse",
+      "input": "pkg:npm/[email protected]#src/%F0%9F%98%83/file.js",
+      "expected_output": {
+        "type": "npm",
+        "namespace": null,
+        "name": "foo",
+        "version": "1.0.0",
+        "qualifiers": null,
+        "subpath": "src/😃/file.js"
+      },
+      "expected_failure": false,
+      "expected_failure_reason": null
+    },
+    {
+      "description": "Positive Test: An npm purl with a qualifier value containing unicode characters (emoji) should roundtrip correctly",
+      "test_group": "advanced",
+      "test_type": "roundtrip",
+      "input": "pkg:npm/[email protected]?vcs_url=https://a.com/b/c%F0%9F%98%83d.git",
+      "expected_output": "pkg:npm/[email protected]?vcs_url=https://a.com/b/c%F0%9F%98%83d.git",
+      "expected_failure": false,
+      "expected_failure_reason": null
+    },
+    {
+      "description": "Positive Test: An npm purl with a qualifier value containing unicode characters (emoji) should parse correctly",
+      "test_group": "advanced",
+      "test_type": "parse",
+      "input": "pkg:npm/[email protected]?vcs_url=https://a.com/b/c%F0%9F%98%83d.git",
+      "expected_output": {
+        "type": "npm",
+        "namespace": null,
+        "name": "foo",
+        "version": "1.0.0",
+        "qualifiers": {
+          "vcs_url": "https://a.com/b/c😃d.git"
+        },
+        "subpath": null
+      },
+      "expected_failure": false,
+      "expected_failure_reason": null
+    },


Do these tests have anything to do with NPM? They seem like they should be basic tests for the core spec.

I noticed that there were no such tests. So, I pushed a commit to showcase some.

prabhu added 2 commits August 24, 2025 10:34

Adds more test cases for npm

00ecd91

Signed-off-by: Prabhu Subramanian <[email protected]>

test cases with chinese and japanese characters for npm

4a6f680

Signed-off-by: Prabhu Subramanian <[email protected]>

prabhu marked this pull request as draft August 24, 2025 20:14

matt-phylum reviewed Aug 25, 2025

View reviewed changes

Adds more test cases for npm #638

Are you sure you want to change the base?

Adds more test cases for npm #638

Uh oh!

Conversation

prabhu commented Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Methodology

Prompt

Screenshots

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prabhu Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

prabhu commented Aug 24, 2025 •

edited

Loading

prabhu Aug 26, 2025 •

edited

Loading