fix(mappers): Support simpleeval 1.0.5+ by subclassing simpleeval.EvalWithCompoundTypes

[edgarrmondragon]

Related

• Closes chore: Support simpleeval 1.0.5+ by implementing ModuleWrapper for modules exposed to stream maps expressions #3561
• Re-implements upstream perf: check disallowed items at name-resolution boundary, not every AST node danthedeckie/simpleeval#181

Summary by Sourcery

Support newer simpleeval versions in mapper expression evaluation while preserving safety guarantees for stream map expressions.

Bug Fixes:

• Restore compatibility with simpleeval 1.0.5+ by customizing evaluation to bypass redundant disallowed-item checks that conflict with mapper usage.

Enhancements:

• Introduce a dedicated _MapperEval subclass to centralize mapper-specific simpleeval behavior and safety constraints.
• Wrap datetime and json modules with simpleeval.ModuleWrapper when exposing them as mapper functions to align with simpleeval's security model.

Build:

• Relax the simpleeval dependency to require version 1.0.5 or later in project configuration.

Summary by Sourcery

Restore compatibility of mapper expression evaluation with newer simpleeval versions while maintaining existing safety guarantees.

Bug Fixes:

• Fix incompatibility with simpleeval 1.0.5+ by using a custom evaluator that bypasses redundant disallowed-item checks conflicting with mapper use cases.

Enhancements:

• Introduce a dedicated _MapperEval subclass to centralize mapper-specific simpleeval behavior and security constraints.
• Wrap the datetime and json modules with simpleeval.ModuleWrapper when exposing them as mapper functions to align with simpleeval's security model.

Build:

• Update the simpleeval dependency constraint to pin to version 1.0.7.

[sourcery-ai]

Reviewer's Guide

Introduces a dedicated _MapperEval subclass of simpleeval.EvalWithCompoundTypes to restore compatibility with simpleeval 1.0.5+ while preserving mapper safety guarantees, replaces direct simpleeval usage in mapper evaluation paths with this subclass, wraps exposed datetime/json modules with simpleeval.ModuleWrapper, and updates the simpleeval dependency to a fixed 1.0.7 version.

Sequence diagram for mapper record transformation using _MapperEval

sequenceDiagram
    actor MapperUser
    participant CustomStreamMap
    participant _MapperEval

    MapperUser->>CustomStreamMap: transform(record)
    activate CustomStreamMap
    CustomStreamMap->>CustomStreamMap: _eval(expr, record)
    CustomStreamMap->>_MapperEval: eval(expr)
    activate _MapperEval
    _MapperEval->>_MapperEval: _eval(ast_root)
    _MapperEval->>_MapperEval: _eval_name(Name)
    _MapperEval-->>CustomStreamMap: evaluated_value
    deactivate _MapperEval
    CustomStreamMap-->>MapperUser: transformed_record
    deactivate CustomStreamMap

Loading

Sequence diagram for stream selection evaluation via _eval_stream

sequenceDiagram
    participant Caller
    participant _eval_stream
    participant _MapperEval

    Caller->>_eval_stream: _eval_stream(expr, stream_name)
    activate _eval_stream
    _eval_stream->>_eval_stream: build names dict
    _eval_stream->>_MapperEval: _MapperEval(names=names)
    activate _MapperEval
    _MapperEval-->>_eval_stream: expr_evaluator
    deactivate _MapperEval
    _eval_stream->>_MapperEval: eval(expr)
    activate _MapperEval
    _MapperEval->>_MapperEval: _eval(ast_root)
    _MapperEval-->>_eval_stream: result
    deactivate _MapperEval
    _eval_stream-->>Caller: result
    deactivate _eval_stream

Loading

Class diagram for mapper evaluation changes using _MapperEval

classDiagram
    direction LR

    class EvalWithCompoundTypes {
        <<external>>
        +dict names
        +dict functions
        +dict nodes
        +eval(expr)
        +_eval(node)
        +_eval_name(node)
        +_eval_attribute(node)
        +_eval_call(node)
        +_check_disallowed_items(item)
    }

    class _MapperEval {
        <<internal>>
        +dict names
        +dict functions
        +dict nodes
        +_MapperEval(names, functions)
        +_eval(node)
        +_eval_name(node)
        +_check_disallowed_items(item)
    }

    class CustomStreamMap {
        +dict functions
        +simpleeval.EvalWithCompoundTypes expr_evaluator
        +CustomStreamMap()
        +transform(record)
        +_init_functions_and_schema(stream_map)
        +_init_faker_instance()
        +_eval(expr, record)
    }

    class simpleeval_ModuleWrapper {
        <<external>>
        +simpleeval_ModuleWrapper(module)
    }

    class datetime_module {
        <<module>>
    }

    class json_module {
        <<module>>
    }

    EvalWithCompoundTypes <|-- _MapperEval
    CustomStreamMap --> _MapperEval : uses_for_expr_evaluator

    CustomStreamMap --> simpleeval_ModuleWrapper : wraps_modules
    simpleeval_ModuleWrapper --> datetime_module : wraps
    simpleeval_ModuleWrapper --> json_module : wraps

Loading

File-Level Changes

Change	Details	Files
Introduce _MapperEval to customize simpleeval behavior for mapper expressions and work around new disallowed-items checks.	Add a private _MapperEval class subclassing simpleeval.EvalWithCompoundTypes for stream map evaluations. Override _check_disallowed_items to no-op, explicitly documenting why safety remains intact in the mapper context. Reimplement _eval to map AST node types to handlers and raise FeatureNotAvailable for unsupported nodes. Reimplement _eval_name to preserve name resolution semantics while still using _check_disallowed_items where appropriate.	singer_sdk/mapper.py
Adopt _MapperEval in all mapper expression evaluation call sites and harden function exposure using ModuleWrapper.	Replace direct constructions of simpleeval.EvalWithCompoundTypes with _MapperEval in CustomStreamMap and _eval_stream. Wrap datetime and json modules in simpleeval.ModuleWrapper before exposing them via the mapper functions dict. Annotate expr_evaluator with a concrete simpleeval.EvalWithCompoundTypes type for clarity.	singer_sdk/mapper.py
Adjust simpleeval dependency to a supported, fixed version compatible with the new mapper evaluator.	Update pyproject.toml to pin simpleeval to version 1.0.7, removing the previous <1.0.5 constraint and incompatible version exclusion. Regenerate/update the uv.lock file to reflect the new simpleeval version and dependency resolution.	pyproject.toml uv.lock

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#3561	Ensure stream map expression evaluation is compatible with simpleeval 1.0.5+ by correctly handling the new security checks (e.g., via ModuleWrapper or subclassing SimpleEval) for modules exposed to expressions.	✅
#3561	Update project dependency constraints so that simpleeval versions 1.0.5 and above are allowed/used.	✅
#3561	Wrap modules exposed to stream map expressions (such as datetime and json) in simpleeval.ModuleWrapper to conform to simpleeval’s updated security model.	✅

Possibly linked issues

• chore: Support simpleeval 1.0.5+ by implementing ModuleWrapper for modules exposed to stream maps expressions #3561: PR introduces _MapperEval and ModuleWrapper for datetime/json to restore stream map compatibility with simpleeval 1.0.5+.
• #N/A: PR implements ModuleWrapper for datetime/json in mapper expressions, directly addressing the issue’s simpleeval 1.0.5+ support request.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.73%. Comparing base (0503852) to head (ed8fd14).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3595   +/-   ##
=======================================
  Coverage   93.73%   93.73%           
=======================================
  Files          73       73           
  Lines        5890     5890           
  Branches      723      723           
=======================================
  Hits         5521     5521           
  Misses        274      274           
  Partials       95       95           

Flag	Coverage Δ
core	82.13% <ø> (ø)
end-to-end	75.50% <ø> (ø)
optional-components	42.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed-hq]

Merging this PR will not alter performance

✅ 8 untouched benchmarks

---

_{Comparing fix/simpleeval-EvalWithCompoundTypes-subclass (ed8fd14) with main (907a460)¹}

Footnotes

1. No successful run was found on main (0503852) during the generation of this report, so 907a460 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The _MapperEval subclass currently reimplements _eval and _eval_name very closely to EvalWithCompoundTypes; consider whether overriding only _check_disallowed_items (and relying on the base class for the rest) is sufficient to reduce maintenance risk when simpleeval internals change.
• The dependency constraint for simpleeval is pinned to ==1.0.7 while the PR description mentions supporting 1.0.5+; consider loosening this to a compatible range (e.g. >=1.0.5,<2.0) or updating the description to reflect the tighter pin.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The `_MapperEval` subclass currently reimplements `_eval` and `_eval_name` very closely to `EvalWithCompoundTypes`; consider whether overriding only `_check_disallowed_items` (and relying on the base class for the rest) is sufficient to reduce maintenance risk when simpleeval internals change.
- The dependency constraint for `simpleeval` is pinned to `==1.0.7` while the PR description mentions supporting 1.0.5+; consider loosening this to a compatible range (e.g. `>=1.0.5,<2.0`) or updating the description to reflect the tighter pin.

## Individual Comments

### Comment 1
<location path="singer_sdk/mapper.py" line_range="292-294" />
<code_context>
+        else:
+            return val
+
+        if callable(self.names):
+            try:
+                val = self.names(node)  # ty:ignore[call-top-callable]
+                self._check_disallowed_items(val)
+            except simpleeval.NameNotDefined:
</code_context>
<issue_to_address>
**issue (bug_risk):** Passing the full `ast.Name` node into `self.names` deviates from simpleeval’s expected callable contract and may break existing callables.

In simpleeval, a callable `names` is invoked as `self.names(node.id)` (a `str`), not with the AST node. Switching to `self.names(node)` changes the callable’s public contract and can break existing implementations that expect a string per the library’s documentation. Unless you explicitly need the full node, please continue passing `node.id` instead.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}