feat(scan): generic Apify actor provider (closes #325)

[ageem23]

Note: Opened against upstream-sync (mirror of santifer/career-ops:main) on my fork to get CodeRabbit feedback before sending upstream. The body below is drafted for the eventual upstream PR.

---

Follow-up to the Plugin Architecture RFC (#521) — implements the third item in the RFC's split plan ("Follow-up RFC + PR (Apify, closes santifer#325)"). Phase A (plugin contract + Greenhouse/Ashby/Lever ports) is already on main; this PR builds on that.

Closes santifer#325.

Summary

• providers/apify.mjs — generic provider that runs any pre-built Apify actor and maps its dataset items to the scanner's {title, url, company, location} shape. All variation (actor, input, field mappings) lives in portals.yml, not in code, so a single implementation serves any site with an Apify scraper available.
• providers/_apify.mjs — shared Apify transport (auth, run/poll/fetch, retries with end-to-end deadline, SSRF guard on actorId). Follows the _-prefix helper convention established by _http.mjs / _types.js.
• Optional field_map.description mapping persists the JD body to jds/{slug}.md and rewrites the entry's url to local:jds/... so downstream tools read it from disk instead of re-fetching the remote URL. Avoids paying for the same Apify run twice and dodges HTTP failures on stale job-board links. Falls through to the remote URL when the description field is missing or shorter than 50 chars.
• .env.example — documents APIFY_TOKEN.
• templates/portals.example.yml — worked Indeed example showing both the basic field_map shape and the optional description mapping.

Conformance to RFC santifer#521

• Provider contract: default export with id, detect(), fetch(entry, ctx). detect() returns null (explicit provider: apify required — no URL scheme to auto-detect from).
• Error contract: missing token / missing required entry config / invalid field_map shape all throw so scan.mjs records the error per-entry and continues with the next.
• Helper convention: _apify.mjs uses the _ prefix and is skipped by the loader.
• scan-history.tsv: source label uses the slugified actor id, so misceres/indeed-scraper produces a stable per-actor dedup key.

Round-2 review fixes already applied

From the prior PR santifer#454 review thread, these are folded in (one CodeRabbit pass already absorbed on my fork as PR #8):

• APIFY_TOKEN sent via Authorization: Bearer header, never as a query string (prevents token leakage into access logs).
• SSRF guard on actorId — strict ^[A-Za-z0-9][A-Za-z0-9_.-]*[~/][A-Za-z0-9][A-Za-z0-9_.-]*$ validation so a malformed config can't divert the bearer-token request off the intended /acts/<actor>/runs path.
• End-to-end timeout via a single shared deadline threaded through startRun → waitForRun → fetchDatasetItems. runActor's timeoutMs is now an actual wall-clock ceiling, not just a wait-loop ceiling.
• waitForRun() re-throws non-retriable 4xx (401/403/404) instead of looping to the deadline and masking the real cause as a generic timeout.
• runActor() validates timeoutMs is a positive finite number up front (a NaN previously produced Date.now() + NaN → confusing "did not finish within 0s" errors).
• apify.mjs saveJd() filename includes sha1(url)[:10] so two distinct postings sharing the same company+title slug don't collide on a single cache file (mirrors the same fix applied to the linkedin provider).
• field_map spec validation accepts only a string or non-empty array of strings; rejects malformed values like { title: 42 } at load time with a clear error instead of crashing mid-scan.

Deliberate scope limits

• Only one example entry in portals.example.yml — the Indeed actor. Workday-via-third-party-rental actors are also tested on my fork but recommending a specific paid third-party rental in upstream docs warrants its own discussion; happy to add it in a follow-up.
• No browser transport, no scraper provider — those are the separate follow-up PR per RFC RFC: Plugin-based Provider Architecture for `scan.mjs` santifer/career-ops#521. This PR is Apify-only.
• No proxyConfiguration handling — only relevant for the deferred Workday entry; can land with it.

Test plan

• node scan.mjs --dry-run with the Indeed example enabled and APIFY_TOKEN set — verify items show up under the misceres-indeed-scraper source label
• Same with APIFY_TOKEN unset — verify the Apify entry errors cleanly and the rest of the scan continues
• Verify a malformed field_map (e.g. title: 42) is rejected at fetch with the documented error
• Verify the optional description: mapping writes jds/{slug}.md and the pipeline entry uses the local:jds/... URL
• Verify a posting with a short/missing description falls through to the remote URL (no broken local:jds/... link)

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Integrated Apify for automated job discovery from external sources (e.g., Indeed scraping)
  • Job listings can be cached locally as formatted markdown files with job descriptions
  • Added environment variable documentation for Apify token authentication setup
  • Included example configuration showing how to set up Apify-backed job sources with automatic field mapping

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5c7c6e96-3c28-4e24-ba9c-71557c03d9b9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/apify-provider-pr

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@providers/apify.mjs`:
- Around line 137-172: The fallback to normalized.applicationUrl in saveJd is
dead/confusing because normalizeItem never sets applicationUrl; either remove
the unused reference in saveJd and simplify the hash input to use normalized.url
|| `${normalized.company}-${normalized.title}`, or alternatively populate
applicationUrl in normalizeItem; update the code in saveJd (function saveJd) to
use only normalized.url (with the company-title fallback) or add an explicit
applicationUrl field in normalizeItem so the reference is valid—pick one
consistent approach and apply it across saveJd and normalizeItem.

In `@templates/portals.example.yml`:
- Around line 1039-1043: Update the example comment that describes where the job
description files are written for the "description" mapping: change the
documented filename from "jds/{slug}.md" to the actual hash-suffixed pattern
used at runtime (e.g., "jds/{slug}-{hash}.md" or "jds/{slug}-{sha256}.md") so
users can locate the cached files correctly when using the description:
[description, descriptionText, descriptionHTML, jobDescription] mapping.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0aca62dc-07c7-4b97-8cc5-21c006aef44b

📥 Commits

Reviewing files that changed from the base of the PR and between 5d1f3a3 and f0d49c2.

📒 Files selected for processing (4)

• .env.example
• providers/_apify.mjs
• providers/apify.mjs
• templates/portals.example.yml

[ageem23]

Round-1 review feedback addressed in 2b5561b:

1. Dead applicationUrl fallback (saveJd) — removed; hash input simplified to normalized.url || ${company}-${title}.
2. jds/{slug}.md → jds/{slug}-{hash}.md — fixed in both apify.mjs module docstring and the portals.example.yml inline comment.
3. URL validation (pattern from feat(scan): add Workable, SmartRecruiters, Recruitee ATS parsers santifer/career-ops#653's review of parseRecruiteeResponse / parseWorkableMarkdown) — actor-supplied URLs are now rejected unless protocol is https:. Blocks javascript:, data:, file:, and protocol-downgraded http: URLs from a buggy or malicious actor before they become clickable in pipeline.md or feed the JD-cache filename hash.

Also scrubbed a stale mirrors providers/linkedin.mjs format section comment — that file isn't upstream.

Test gap noted but not addressed in this commit: PR santifer#653 added §11/§12/§13 in test-all.mjs (~27 assertions per provider). Happy to add an equivalent §11 for the apify provider here or in a follow-up — let me know which fits better for the upstream PR scope.

@coderabbitai review

[coderabbitai]

@ageem23 Thanks for the detailed round-up — all three fixes look well-scoped:

• The simplified hash input (normalized.url || \${company}-${title}``) is cleaner and the dead fallback removal reduces confusion.
• The {slug}-{hash}.md docstring/YAML alignment is good housekeeping.
• The https:-only guard is the right call — especially since actor output URLs flow into pipeline.md links and the JD cache filename hash; blocking downgrade/exotic protocols at the boundary is much safer than trusting the actor author.

On the test gap: My recommendation is to add §11 for the Apify provider in this PR. The new provider introduces meaningful branching surface (field_map validation, description fallback, URL rejection, JD file write/reuse, hasToken short-circuit) that benefits from co-located tests. Shipping the provider without any assertions makes future refactors riskier. If the upstream PR scope is strict, a narrow §11 covering just the security-relevant paths (URL rejection, malformed field_map, missing token) would be a reasonable minimum — the happy-path and description tests could follow in the same style as #653's §12/§13.

Triggering a fresh review now.

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[ageem23]

Added §11 to test-all.mjs in c3e3b2f — 34 assertions covering the security-relevant paths CodeRabbit asked for. Full suite: 100 passed, 0 failed.

Coverage (kept narrow per the round-2 guidance):

• URL rejection — isHttpsUrl() rejects http:, javascript:, data:, file:, ftp:, malformed/empty/null/undefined
• Malformed field_map — isFieldSpec() accepts string + non-empty string array; rejects number, empty array, mixed-type array, null, object
• Missing token / actor / field_map — apify.fetch() throws clear errors before any network call (env var save/restore so the test works whether or not APIFY_TOKEN is set locally)
• Actor-id SSRF guard — normalizeActorId() rejects path traversal, query, fragment, extra path segments, missing separator, non-string
• Defaults allowlist — normalizeItem() blocks portals.yml defaults outside title/url/company/location

To make the helpers testable, isFieldSpec / isHttpsUrl / normalizeItem (apify.mjs) and normalizeActorId (_apify.mjs) are now named exports alongside the existing default export. Behavior unchanged.

Skipped per CodeRabbit's "narrow §11" guidance: happy-path fetch() through a mocked runActor, JD-cache write path, and htmlToText sanitization. Worth a follow-up if the maintainer wants the broader surface.