Skip to content

refactor(sam): converge 4 SAM integrations — v1 scripts / v2 layers / v3 sam3.py / EVF-SAM #16

@yha9806

Description

@yha9806

Problem

Vulca currently has 4 parallel SAM integrations accreted over 5 days (2026-04-15 to 2026-04-20), each with its own dependency tree, with no convergence:

Path SAM gen Module State
`scripts/claude_orchestrated_pipeline.py` v1 (2023) `segment_anything` + `sam_vit_l` checkpoint Hardcoded `/tmp/sam_vit_l.pth`, no env override
`src/vulca/layers/sam.py` v2 (2024) `sam2.build_sam` + `SAM2ImagePredictor` HF Hub `facebook/sam2.1-hiera-small`
`src/vulca/layers/sam3.py` v3 (2025) `transformers.Sam3Model` Requires `transformers>=4.50` — conflicts with EVF-SAM `transformers==4.49` pin
`scripts/_evfsam_common.py` EVF-SAM text-prompted, vendored fork Forces transformers pin to 4.49

Why this matters

Surfaced 2026-04-24 γ Scottish showcase: `/decompose` mode="orchestrated" hit cascading issues:

  1. `No module named 'segment_anything'` (MCP Python 3.11 env didn't have v1 SAM dep — pyproject [sam] extra was for v2)
  2. `cannot import name 'ImageNetInfo' from 'timm.data'` (timm 0.6.13 too old for transformers 4.49 timm_wrapper; but torchscale pins timm=0.6.13)
  3. After install, MCP Python import cache held failed-import state — required full session restart

End user wasted ~30 minutes diagnosing dep chain that should have been a single `pip install`. Plus 1.2GB SAM v1 checkpoint download because the system can't probe / reuse the EVF-SAM weights or FastSAM-s.pt the user already has on disk.

Proposed convergence

v0.18.0 target (not v0.17.12 — too large for rollup release):

  1. Retire `scripts/claude_orchestrated_pipeline.py` — merge into `src/vulca/layers/orchestrated.py` so the orchestrated mode lives inside the package, gets pyproject `[sam]` extras correctly, and is unit-testable.
  2. Migrate orchestrated mode to SAM v2 — already-imported `src/vulca/layers/sam.py` works, no need for parallel v1 path. Eliminates segment-anything dep + /tmp checkpoint.
  3. Add `VULCA_SAM_CHECKPOINT` env var (transitional, while v1 path exists) — let user point at any pre-deployed checkpoint instead of hardcoded `/tmp/sam_vit_l.pth`.
  4. Startup capability probe — MCP server logs which SAM/loader paths are functional at boot. Agent can introspect `get_capabilities()` to choose mode intelligently.
  5. Unpin transformers OR sandbox EVF-SAM — current pin to 4.49 blocks SAM3 adoption. Either upgrade EVF-SAM fork to 4.50+ (upstream support TBD) or run EVF-SAM in subprocess venv to isolate.

Blockers

  • EVF-SAM fork is not upstream-tracked; transformers upgrade may require maintaining our own fork
  • SAM3 is GPU-only in current Hugging Face bindings; MPS support unclear

Scope

v0.18.0 target. Requires independent brainstorm session — large enough that this issue is a placeholder to track the design discussion. Do not attempt within v0.17.12 rollup.

Surfaced from

2026-04-24 γ Scottish showcase Part 1 session.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions