[ENG] en_core_web_trf inaccurately identifying noun chunks

[thkuo]

Summary

I was trying to identify noun chunks from my texts. When assessing pre-trained models, en_core_web_trf performed best in most cases, which was expected due to the data size. However, some results from the model was hard to understand. I wonder whether anyone could help explain it or suggest further processing for my original data.

Methods

Python version

3.12

Model versions:

• en_core_web_trf: 3.8.0
• en_core_web_lg: 3.8.0
• en_core_web_sm: 3.8.0

Data

Original text: 'Acute cardiovascular disease (CVD) events \< 3 months ago'

Workflow

import spacy
import pandas as pd

models = ["en_core_web_sm", "en_core_web_lg", "en_core_web_trf"]

sentence =  'Acute cardiovascular disease (CVD) events \\< 3 months ago'

for m in models:
    print(">>>{}".format(m))
    processor = spacy.load(m)
    doc = processor(sentence)
    decomp_df = pd.DataFrame([
        {"text": token.text, "lemma": token.lemma_, "dep_tag": token.dep_}
         for token in doc])
    print(decomp_df)
    print([chunk.text for chunk in doc.noun_chunks])

The output will be

>>>en_core_web_sm
              text           lemma   dep_tag
0            Acute           acute      nmod
1   cardiovascular  cardiovascular      amod
2          disease         disease     nsubj
3                (               (     punct
4              CVD             CVD     appos
5                )               )     punct
6           events           event      ROOT
7                \               \      nmod
8                <               <      dobj
9                3               3    nummod
10          months           month  npadvmod
11             ago             ago    advmod
['Acute cardiovascular disease', 'CVD']
>>>en_core_web_lg
              text           lemma   dep_tag
0            Acute           acute      amod
1   cardiovascular  cardiovascular      amod
2          disease         disease     nsubj
3                (               (     punct
4              CVD             CVD     appos
5                )               )     punct
6           events           event      ROOT
7                \               \      dobj
8                <               <     appos
9                3               3    nummod
10          months           month  npadvmod
11             ago             ago    advmod
['Acute cardiovascular disease', 'CVD', '\\']
>>>en_core_web_trf
              text           lemma   dep_tag
0            Acute           acute      amod
1   cardiovascular  cardiovascular      amod
2          disease         disease      nmod
3                (               (     punct
4              CVD             cvd     appos
5                )               )     punct
6           events           event      ROOT
7                \               \     punct
8                <               <     appos
9                3               3    nummod
10          months           month  npadvmod
11             ago             ago    advmod
['(CVD']

Could anyone explain why the most accurate model would suggest '(CVD' as the noun chunk, please?

[apoorva-01]

.noun_chunks isn't a separate noun-chunker, it's a fixed rule that reads off the dependency parse. So (CVD isn't the model "choosing" a bad chunk, it's a side effect of how trf parsed the fragment.

The rule (spaCy's English syntax iterator) walks every NOUN/PROPN/PRON, and for the ones whose dep_ is a head label (nsubj, dobj, appos, attr, pobj, ROOT, ...) it yields the span from that token's left_edge to the token itself:

yield word.left_edge.i, word.i + 1, np_label

In your trf parse, CVD is tagged appos (a head label), and the ( got attached as a left dependent inside CVD's subtree, so CVD.left_edge is (. Span runs ( → CVD = (CVD. Check doc[4].left_edge and you'll see it.

Then it gets worse: chunks can't nest (there's a prev_end guard), so once it emits (CVD it skips the real outer chunk headed by events (ROOT), whose left edge is Acute. That's why trf gives you only ['(CVD'] while sm/lg give the sensible ones. All three disagree purely because their parse trees disagree, trf isn't "more accurate" for noun chunks here, it just attached the paren differently.

What actually fixes it:

• Your input is a sentence fragment with a stray escaped \<. The parsers are trained on well-formed sentences, and that \ token (tagged punct) is exactly the kind of thing that skews attachment. Clean it up (drop the \<, or feed real sentences) and the parse settles down.
• If you just want clean spans regardless, strip leading punctuation: chunk[1:] if chunk[0].is_punct else chunk.
• For clinical text specifically, a biomedical model parses these fragments far better than the general web models. scispaCy (en_core_sci_lg) is worth a look.

[thkuo]

Thank you, @apoorva-01, for the helpful explanation and recommendations. Cleaning the poorly formatted tokens (i.e., \>) couldn't work well at my site unfortunately, but cleaning output spans will be part of my workflow. In addition, I am checking out scispaCy.

In my case, the dependency label of CVD is appos, which is a type of modifiers (appositional modifier), and ( and CVD are next to other modifiers (i.e., amod, amod, and nmod) in the trf result. I assume merging a chain of modifiers into a chunk would be more sensible, so I am also trying to edit the iterator to consider a chain of modifiers a chunk.

I am closing the discussion and will share updates if other solutions could be helpful generally.