Skip to content

Wrap Web.Contents so functions using it aren't excluded from M.pq - #18

Open
wakinniranye31 wants to merge 1 commit into
OscarValerock:mainfrom
wakinniranye31:fix-web-contents-dynamic-error-1
Open

Wrap Web.Contents so functions using it aren't excluded from M.pq#18
wakinniranye31 wants to merge 1 commit into
OscarValerock:mainfrom
wakinniranye31:fix-web-contents-dynamic-error-1

Conversation

@wakinniranye31

Copy link
Copy Markdown

Closes #1.

Root cause

M.pq loads each function file's source and runs it through Expression.Evaluate(source, environment). Web.Contents is deliberately left out of that environment record (see the old comment in M_Creator.py's exclude_strings) because passing it in directly as Web.Contents = Web.Contents triggers Power Query's "This expression is dynamic, so it might not work..." / Formula.Firewall "can't use a Dynamic Data Source" error, especially on refresh from the Power BI service — this is a well-known M limitation once Web.Contents is passed around as a first-class value rather than called literally (see the reference already linked in Functions/SharePoint/GetSPList.pq's own docs: community.powerbi.com/.../Dynamic-data-sources-aren-t-refreshed...).

Because Web.Contents is excluded, any function that calls it internally — GetSPList, GitHubRepoFiles, Countries, the OpenAI functions — fails inside Expression.Evaluate with an undefined-identifier error, which the try ... otherwise fallback in M_Creator.py's template swallows by falling back to returning the function's raw text instead of an evaluated function. That's exactly the "displays as text" symptom in the issue.

Fix

Rather than passing Web.Contents itself into the environment, map it to a locally-defined wrapper with the same signature:

#"Web.Contents wrapper" = (url as text, optional options as nullable record) as any =>
    Web.Contents(url, options),

and register Web.Contents = #"Web.Contents wrapper" in the Expression.Evaluate environment instead of the bare function. This is the standard community-documented workaround for this exact Formula.Firewall limitation — wrapping a Web.Contents call in a named function whose parameters flow straight through to the call often lets Power Query's static dependency analysis "see" the literal call site again.

Changes

  • M_Creator.py:
    • Added a manual_mappings dict for identifiers that need a custom expression rather than the auto-generated Name = Name, and wired Web.Contents into it.
    • Added the #"Web.Contents wrapper" step to the M_Code template.
    • Removed the old dead //,Web.Contents = Web.Contents //... comment, replaced with an accurate comment on why Web.Contents stays in exclude_strings.
    • Simplified the environment-record string builder to a list comprehension + ",\n".join(...) so appending manual_mappings entries didn't need a second near-duplicate loop.
  • M.pq: regenerated via python M_Creator.py so it matches what M_Creator.py now produces.

Verification

  • Ran M_Creator.py and confirmed the generated M.pq contains Web.Contents = #"Web.Contents wrapper" in the Expression.Evaluate environment, and the wrapper step itself, in the right scope to be visible there.
  • Checked bracket/paren/brace balance across the regenerated M.pq (all balanced) as a basic sanity check.
  • I don't have a Power BI Desktop/Service environment to test an actual refresh against, so I can't personally confirm the Formula.Firewall accepts this specific wrapper pattern in every context (dynamic-data-source detection is known to be inconsistent across Desktop vs. Service vs. dataflows). This is the standard fix for this class of error, but I'd appreciate it if someone with a live environment (or you, since you filed the issue) could verify a function like GetSPList now evaluates correctly instead of falling back to text — happy to iterate if it needs adjustment.

Web.Contents was excluded entirely from the generated Expression.Evaluate
environment because passing it in directly (Web.Contents = Web.Contents)
triggers the 'can't use a Dynamic Data Source' Formula.Firewall error.
That meant any custom function using Web.Contents internally (e.g.
GetSPList, GitHubRepoFiles, Countries) would throw during evaluation and
silently fall back to displaying as raw text.

Instead of excluding Web.Contents, map it to a same-signature wrapper
function defined in M.pq's own let block:

    #"Web.Contents wrapper" = (url as text, optional options as
        nullable record) as any => Web.Contents(url, options)

Passing the wrapper (rather than Web.Contents itself) through
Expression.Evaluate's environment is the standard community workaround
for this exact Formula.Firewall limitation.

Closes OscarValerock#1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Functions using Web.Contents do not work properly.

1 participant