[SPARK-54938][PYTHON][TEST][FOLLOW-UP] Fix test_pyarrow_array_type_inference for pandas >= 3

[zhengruifeng]

What changes were proposed in this pull request?

pandas 3.x changed default string dtype to use pyarrow-backed storage, causing pa.array() to infer large_string instead of string for string Series. Conditionally expect large_string on pandas >= 3.

Why are the changes needed?

to resolve failure in https://github.com/apache/spark/actions/runs/23819581811/job/69428367355

Does this PR introduce any user-facing change?

No, test-only

How was this patch tested?

manually check

pandas==3.0.1

In [3]: import pyarrow as pa

In [4]: import pandas as pd

In [5]: ser = pd.Series(["a", "b", "c"], dtype=pd.StringDtype())

In [6]: pa.array(ser)
Out[6]:
<pyarrow.lib.LargeStringArray object at 0x103455d80>
[
  "a",
  "b",
  "c"
]

In [7]: pa.array(ser, pa.string())
Out[7]:
<pyarrow.lib.StringArray object at 0x1095546a0>
[
  "a",
  "b",
  "c"
]

pandas==2.3.3

In [7]: ser = pd.Series(["a", "b", "c"], dtype=pd.StringDtype())

In [8]: pa.array(ser)
Out[8]:
<pyarrow.lib.StringArray object at 0x10ae16620>
[
  "a",
  "b",
  "c"
]

In [9]: pa.array(ser, pa.string())
Out[9]:
<pyarrow.lib.StringArray object at 0x10ae14a00>
[
  "a",
  "b",
  "c"
]

Was this patch authored or co-authored using generative AI tooling?

Co-authored-by: Claude code (Opus 4.6)

[zhengruifeng]

cc @Yicong-Huang

[zhengruifeng]

thanks, merged to master