Conversation
…us Compute Introduce a unified execution module with an abstract ExecutionBackend interface and TaskSpec model, supporting four backends: local (ProcessPoolExecutor), Parsl, EnsembleLauncher, and Globus Compute. Includes config factory with resolution order (args > env > config.toml), HPC configs loader, comprehensive tests, and pytest --run-globus-compute option for live endpoint tests.
Remove dead num_nodes=1 after raise in aurora_parsl.py and fix misleading error message. Set _initialized=False at start of EnsembleLauncherBackend.shutdown() to prevent submitting to a partially torn-down backend.
When backend=globus_compute, MCP tools now return immediately after submitting jobs to the remote HPC endpoint instead of blocking until completion. A new JobTracker tracks submitted futures across tool calls, and new MCP tools (check_job_status, get_job_results, list_jobs, cancel_job) let the LLM agent poll for progress and retrieve results. Non-Globus backends (local, Parsl, EnsembleLauncher) are unchanged and continue to block until results are ready. Key changes: - Add is_async_remote property to ExecutionBackend (True for Globus) - Add check_endpoint_status() health check to GlobusComputeBackend - Add JobTracker with batch registration, status, results, cleanup - Add submit_or_gather() utility that branches on backend type - Add optional timeout parameter to gather_futures() - Add register_job_tools() to wire job tools into any MCP server - Integrate tracker into MACE, XANES, and gRASPA MCP servers
- Add CGFastMCP: FastMCP subclass with integrated execution backend, lazy init, built-in job tools, @tool() and @ensemble_tool() decorators - Refactor EnsembleLauncherBackend with client-only mode (shared orchestrator via checkpoint_dir) and managed mode - Update get_backend() to route client_only vs managed EL initialization - Rewrite mace_mcp_hpc.py to use CGFastMCP decorators - Clean up parsl_tools.py: remove dead code, use stdlib logging - Fix __main__ pickle issue via _fix_module_for_pickle + sys.modules alias - Add client-only mode demo cell to notebook 3 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Modified the EL backend implemenations, and added a EL backend test
…mport - parsl_tools.run_mace_core: stop swallowing exceptions and returning None. run_ase_core already returns a structured failure dict on simulation errors, and programmer errors should propagate. - cg_fastmcp.ensemble_tool: raise TypeError with a clear message when the decorated function does not have exactly one parameter, instead of crashing with IndexError at decoration time. - ensemble_launcher_backend: soft-import ensemble_launcher and defer the failure to construction / call time. SYSTEM_CONFIG_REGISTRY is now a lazy view backed by builder functions so the module loads cleanly without EL installed, restoring the deferred-error behaviour callers of chemgraph.execution.config expected.
- persist_file parameter: when set, batch metadata and Globus Compute task UUIDs are written to JSON after registration and after results are cached, and loaded on init. Allows MCP servers to recover job state across restarts. - TrackedTask.globus_task_id and TrackedTask.future are both optional; loaded-from-disk batches have no in-memory Future and are queried via the Globus Compute Client directly in get_status. - Lazy Globus Compute Client with a separate gc_lock for thread safety. - _wait_for_globus_task_ids polls each ComputeFuture briefly after submission to capture the Globus task_id assigned asynchronously by the Executor background thread. - cancel_batch / cleanup_old_batches handle the no-future case.
- init_backend now accepts tracker_kwargs= and forwards it to JobTracker(...) in _ensure_backend. Callers can pass persist_file= so MCP servers recover job state across restarts. - set_pre_submit_hook(hook): hook receives each TaskSpec before backend.submit() and returns a (possibly mutated) one. Lets a server centralise transport concerns -- inline-structure embedding for local-submit-to-remote-worker, remote-path rewriting -- instead of repeating that logic in every tool body. Wired into the @tool, @ensemble_tool, and @schema_fanout_tool submit paths. - @schema_fanout_tool(worker=...): the decorated function is an expander (ensemble schema -> list of per-item args). The framework calls worker(item) on the backend for each item and gathers results. Preserves the ensemble schema as the agent-facing API (one tool call, server-side fanout), complementing @ensemble_tool which exposes list[Schema] for callers that want client-side enumeration.
…ersistence - mace_input_schema_ensemble / graspa_input_schema_ensemble: new remote_structure_directory field for pre-staged HPC files (paired with the upcoming transfer_files tool). input_structure_directory now defaults to empty string so callers can pass either. - mace_input_schema/_ensemble model description spells out that 'mace_mp' is the calculator type, not a model name -- LLMs were confusing the two. - Nullable schema fields (driver, model, wall_time) typed as str|None / float|None for correct OpenAPI schema generation. - GlobusComputeBackend._ensure_executor re-creates the Executor when it has been shut down (e.g. after a remote task failure). Uses getattr() so we don't depend on the SDK's private _stopped attr existing. - check_endpoint_status logs exc_info on failure for easier debugging. - xanes_mcp_hpc: JobTracker(persist_file=~/.chemgraph/xanes_jobs.json) so XANES job state survives MCP server restarts. Instructions updated to tell the LLM to surface batch_ids to the user.
- execution/globus_transfer.py: GlobusTransferManager wraps the globus_sdk TransferClient with token caching, batched transfer_files / wait_for_transfer / check_transfer_status / list_remote_directory. Lazy globus_sdk import, lazy auth. - execution/config.get_transfer_manager(): builds a manager from [execution.globus_transfer] in config.toml with env var overrides (GLOBUS_TRANSFER_SOURCE_ENDPOINT_ID, _DESTINATION_ENDPOINT_ID, _DESTINATION_BASE_PATH). Returns None when not configured so MCP servers can skip registration silently. - mcp/transfer_tools.register_transfer_tools(): registers transfer_files, check_transfer_status, list_remote_files on a FastMCP/CGFastMCP server. Uses mcp.add_tool() (not the backend-submitting @tool() decorator) because these are orchestration tools, not compute tasks -- they call the Globus Transfer API directly from the MCP server process. - get_backend() globus_compute endpoint_id fallback now treats empty-string endpoint_id as unset, matching the GLOBUS_COMPUTE_ ENDPOINT_ID env-var override behaviour.
…GFastMCP run_mace_single and run_mace_ensemble were collapsed to bare run_mace_core(params) calls in PR #127, dropping inline-structure embedding, remote-path support, JobTracker persistence, and the Globus Transfer registration that 51ba171 had built. This restores all of that on top of the new CGFastMCP framework. - Worker is now a separate function _mace_worker(job: dict) that handles two transport keys on the worker FS: remote_structure_file (use the path directly) and inline_structure (materialise an AtomsData dict to a temp XYZ). Embeds full_output back into the result for inline calls so callers do not need remote FS access. - Pre-submit hook _mace_transport_hook centralises the schema -> job-dict conversion, mace_mp -> medium-mpa-0 model normalisation, and inline embedding (when the input file exists on the submitting host). Hook rewrites task.callable from run_mace_single to _mace_worker so the LLM still sees a clean schema-shaped tool. - run_mace_ensemble switches to @schema_fanout_tool with a server-side expander, preserving the directory-driven UX (single LLM call instead of N). Local mode enumerates files via resolve_structure_files; remote mode submits a backend probe to ls remote_structure_directory and builds remote_structure_file per item. - extract_output_json registered via mcp.add_tool() (orchestration, no backend wrap). transfer_files/check_transfer_status/ list_remote_files registered conditionally when get_transfer_manager() finds [execution.globus_transfer] config. - __main__ now wires tracker_kwargs={persist_file: ~/.chemgraph/ mace_jobs.json} so MACE batches survive MCP server restarts. - Drop `from __future__ import annotations`: forward refs break FastMCP's signature introspection because the wrapper's __globals__ is cg_fastmcp's, not the tool module's.
Wraps the Academy distributed agent framework with ChemGraph LLM agents for federated HPC screening workflows. Decoupled from the existing pipeline -- no chemgraph.cli / chemgraph.agent / chemgraph.eval references; only the lazily imported chemgraph.agent.llm_agent.ChemGraph. - ChemGraphAgent: Academy Agent wrapping a single ChemGraph instance, exposes run_query / get_info actions. - ScreeningAgent: iterates a molecule batch, writes per-result JSONs for fault-tolerant aggregation. Failed-molecule records now store str(exc) so the actual exception message survives. - CoordinatorAgent: polls a results dir, optionally analyses results via an LLM, suggests follow-up molecules. - AcademyConfig + build_manager: bridge config.toml to Academy Manager / Exchange / Launcher (local, Redis, Parsl, Globus Compute). - RateLimiter: stdlib async token-bucket for shared per-provider LLM quotas across agents. Lazy imports in __init__.py let the package load without the optional academy-py dependency; ChemGraphAgent / ScreeningAgent / CoordinatorAgent raise ModuleNotFoundError on access if academy-py is missing, while AcademyConfig and RateLimiter remain usable. pyproject's academy optional-dep + pytest marker are already in HEAD (commit 04bcc8a). tests/test_academy.py and scripts/academy_example/ remain untracked and will land in follow-ups.
- globus_transfer.py: disambiguate same-basename inputs with a numeric suffix so two files that share a name (e.g. /a/in.cif and /b/in.cif) don't silently overwrite each other on the remote collection. - job_tracker.py: promote the "no Globus task_id within timeout" message to a warning at submit time, and emit a per-task warning at reload time for batches restored without a task_id (those tasks cannot be queried via the Globus Compute API and would otherwise be silently orphaned across server restarts). - globus_compute_backend.py: catch "executor stopped" exceptions in submit(), rebuild the Executor, and retry once. The previous _ensure_executor relied on the SDK's private _stopped attribute, which fails silently if the SDK exposes the shutdown state differently. - cg_fastmcp.py: wrap _apply_pre_submit_hook in try/except and re-raise hook failures as a ValueError naming the hook and task_id so they surface as a structured tool error instead of an opaque traceback.
Both servers now mirror the mace_mcp_hpc.py pattern:
- CGFastMCP with lazy backend initialisation via init_backend(); the
worker subprocesses re-importing the module no longer instantiate a
backend at import time.
- Job-management tools (check_job_status, get_job_results, list_jobs,
cancel_job, check_endpoint_status) are auto-registered by
CGFastMCP._register_job_tools; the external register_job_tools call
is dropped.
- __main__ wires init_backend(tracker_kwargs={"persist_file": ...})
and pairs run_mcp_server with shutdown_backend in finally. This also
closes a real bug in graspa_mcp_hpc.py, which was instantiating
JobTracker() with no persist_file and silently losing job state
across restarts despite the server's instructions promising
persistence.
- Globus Transfer tools (transfer_files, check_transfer_status,
list_remote_files) are registered on both servers when the transfer
manager is configured, matching the existing MACE behaviour.
- gRASPA expander now supports remote_structure_directory the same way
MACE does: a one-shot probe task lists CIFs on the remote endpoint
and the worker reads them directly from the staged path.
- Ensemble flows use the schema_fanout_tool decorator; per-job structure
metadata is propagated through the worker output (since the framework
meta is only the index).
Legacy *_mcp_parsl.py modules now raise a DeprecationWarning at import
pointing to the *_hpc.py replacement; they remain functional because
scripts/mcp_xanes_example/ still imports xanes_mcp_parsl.
…us Compute Introduce a unified execution module with an abstract ExecutionBackend interface and TaskSpec model, supporting four backends: local (ProcessPoolExecutor), Parsl, EnsembleLauncher, and Globus Compute. Includes config factory with resolution order (args > env > config.toml), HPC configs loader, comprehensive tests, and pytest --run-globus-compute option for live endpoint tests.
Remove dead num_nodes=1 after raise in aurora_parsl.py and fix misleading error message. Set _initialized=False at start of EnsembleLauncherBackend.shutdown() to prevent submitting to a partially torn-down backend.
When backend=globus_compute, MCP tools now return immediately after submitting jobs to the remote HPC endpoint instead of blocking until completion. A new JobTracker tracks submitted futures across tool calls, and new MCP tools (check_job_status, get_job_results, list_jobs, cancel_job) let the LLM agent poll for progress and retrieve results. Non-Globus backends (local, Parsl, EnsembleLauncher) are unchanged and continue to block until results are ready. Key changes: - Add is_async_remote property to ExecutionBackend (True for Globus) - Add check_endpoint_status() health check to GlobusComputeBackend - Add JobTracker with batch registration, status, results, cleanup - Add submit_or_gather() utility that branches on backend type - Add optional timeout parameter to gather_futures() - Add register_job_tools() to wire job tools into any MCP server - Integrate tracker into MACE, XANES, and gRASPA MCP servers
- Add CGFastMCP: FastMCP subclass with integrated execution backend, lazy init, built-in job tools, @tool() and @ensemble_tool() decorators - Refactor EnsembleLauncherBackend with client-only mode (shared orchestrator via checkpoint_dir) and managed mode - Update get_backend() to route client_only vs managed EL initialization - Rewrite mace_mcp_hpc.py to use CGFastMCP decorators - Clean up parsl_tools.py: remove dead code, use stdlib logging - Fix __main__ pickle issue via _fix_module_for_pickle + sys.modules alias - Add client-only mode demo cell to notebook 3 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mport - parsl_tools.run_mace_core: stop swallowing exceptions and returning None. run_ase_core already returns a structured failure dict on simulation errors, and programmer errors should propagate. - cg_fastmcp.ensemble_tool: raise TypeError with a clear message when the decorated function does not have exactly one parameter, instead of crashing with IndexError at decoration time. - ensemble_launcher_backend: soft-import ensemble_launcher and defer the failure to construction / call time. SYSTEM_CONFIG_REGISTRY is now a lazy view backed by builder functions so the module loads cleanly without EL installed, restoring the deferred-error behaviour callers of chemgraph.execution.config expected.
- persist_file parameter: when set, batch metadata and Globus Compute task UUIDs are written to JSON after registration and after results are cached, and loaded on init. Allows MCP servers to recover job state across restarts. - TrackedTask.globus_task_id and TrackedTask.future are both optional; loaded-from-disk batches have no in-memory Future and are queried via the Globus Compute Client directly in get_status. - Lazy Globus Compute Client with a separate gc_lock for thread safety. - _wait_for_globus_task_ids polls each ComputeFuture briefly after submission to capture the Globus task_id assigned asynchronously by the Executor background thread. - cancel_batch / cleanup_old_batches handle the no-future case.
- init_backend now accepts tracker_kwargs= and forwards it to JobTracker(...) in _ensure_backend. Callers can pass persist_file= so MCP servers recover job state across restarts. - set_pre_submit_hook(hook): hook receives each TaskSpec before backend.submit() and returns a (possibly mutated) one. Lets a server centralise transport concerns -- inline-structure embedding for local-submit-to-remote-worker, remote-path rewriting -- instead of repeating that logic in every tool body. Wired into the @tool, @ensemble_tool, and @schema_fanout_tool submit paths. - @schema_fanout_tool(worker=...): the decorated function is an expander (ensemble schema -> list of per-item args). The framework calls worker(item) on the backend for each item and gathers results. Preserves the ensemble schema as the agent-facing API (one tool call, server-side fanout), complementing @ensemble_tool which exposes list[Schema] for callers that want client-side enumeration.
…ersistence - mace_input_schema_ensemble / graspa_input_schema_ensemble: new remote_structure_directory field for pre-staged HPC files (paired with the upcoming transfer_files tool). input_structure_directory now defaults to empty string so callers can pass either. - mace_input_schema/_ensemble model description spells out that 'mace_mp' is the calculator type, not a model name -- LLMs were confusing the two. - Nullable schema fields (driver, model, wall_time) typed as str|None / float|None for correct OpenAPI schema generation. - GlobusComputeBackend._ensure_executor re-creates the Executor when it has been shut down (e.g. after a remote task failure). Uses getattr() so we don't depend on the SDK's private _stopped attr existing. - check_endpoint_status logs exc_info on failure for easier debugging. - xanes_mcp_hpc: JobTracker(persist_file=~/.chemgraph/xanes_jobs.json) so XANES job state survives MCP server restarts. Instructions updated to tell the LLM to surface batch_ids to the user.
- execution/globus_transfer.py: GlobusTransferManager wraps the globus_sdk TransferClient with token caching, batched transfer_files / wait_for_transfer / check_transfer_status / list_remote_directory. Lazy globus_sdk import, lazy auth. - execution/config.get_transfer_manager(): builds a manager from [execution.globus_transfer] in config.toml with env var overrides (GLOBUS_TRANSFER_SOURCE_ENDPOINT_ID, _DESTINATION_ENDPOINT_ID, _DESTINATION_BASE_PATH). Returns None when not configured so MCP servers can skip registration silently. - mcp/transfer_tools.register_transfer_tools(): registers transfer_files, check_transfer_status, list_remote_files on a FastMCP/CGFastMCP server. Uses mcp.add_tool() (not the backend-submitting @tool() decorator) because these are orchestration tools, not compute tasks -- they call the Globus Transfer API directly from the MCP server process. - get_backend() globus_compute endpoint_id fallback now treats empty-string endpoint_id as unset, matching the GLOBUS_COMPUTE_ ENDPOINT_ID env-var override behaviour.
…GFastMCP run_mace_single and run_mace_ensemble were collapsed to bare run_mace_core(params) calls in PR #127, dropping inline-structure embedding, remote-path support, JobTracker persistence, and the Globus Transfer registration that 51ba171 had built. This restores all of that on top of the new CGFastMCP framework. - Worker is now a separate function _mace_worker(job: dict) that handles two transport keys on the worker FS: remote_structure_file (use the path directly) and inline_structure (materialise an AtomsData dict to a temp XYZ). Embeds full_output back into the result for inline calls so callers do not need remote FS access. - Pre-submit hook _mace_transport_hook centralises the schema -> job-dict conversion, mace_mp -> medium-mpa-0 model normalisation, and inline embedding (when the input file exists on the submitting host). Hook rewrites task.callable from run_mace_single to _mace_worker so the LLM still sees a clean schema-shaped tool. - run_mace_ensemble switches to @schema_fanout_tool with a server-side expander, preserving the directory-driven UX (single LLM call instead of N). Local mode enumerates files via resolve_structure_files; remote mode submits a backend probe to ls remote_structure_directory and builds remote_structure_file per item. - extract_output_json registered via mcp.add_tool() (orchestration, no backend wrap). transfer_files/check_transfer_status/ list_remote_files registered conditionally when get_transfer_manager() finds [execution.globus_transfer] config. - __main__ now wires tracker_kwargs={persist_file: ~/.chemgraph/ mace_jobs.json} so MACE batches survive MCP server restarts. - Drop `from __future__ import annotations`: forward refs break FastMCP's signature introspection because the wrapper's __globals__ is cg_fastmcp's, not the tool module's.
# Conflicts: # src/chemgraph/hpc_configs/aurora_parsl.py # src/chemgraph/tools/ase_core.py
demo_parsl_in_job_agent.py forced model_name="argo:gpt-5.4" and a local argoapi base_url, mirroring the temp config that had also landed in demo_ensemble_launcher_in_job_agent.py. Restore model selection from the --model flag (the amain(model=...) parameter) and drop the hardcoded base_url so the demo honours the user's chosen model.
TestELSystemConfigCrux asserted the exact Crux SystemConfig shape (ncpus==128, CPU-only) and the registry membership of "crux". These are tied to one specific machine's hardware layout and don't belong in the portable unit-test suite. The polaris references elsewhere are left as-is since they only pass "polaris" as an arbitrary system string to exercise generic GlobusCompute behaviour.
EnsembleLauncher is an optional, HPC-only dependency (not on PyPI for
Python 3.12), so instantiating EnsembleLauncherBackend() raised
ImportError and hard-failed all nine TestELBackend cases in any env
without it. Add pytest.importorskip("ensemble_launcher") in setup_class
so the class skips cleanly, matching the guard already used by the
GlobusCompute tests.
Update existing backends (Parsl, EL, Globus Compute, Globus Transfer) to ChemGraph
The chemgraph.academy package eagerly imported ChemGraphLogicalAgent,
which in turn imports academy.agent. Any consumer that touched the
package -- including chemgraph.cli.trace's --trace-dir code path and
pytest collection -- crashed when the optional [academy] extra was
not installed, despite the rest of the package (campaign spec,
prompt profile, event log) being pure stdlib + pydantic.
Fix in two layers:
* src/chemgraph/academy/__init__.py and
src/chemgraph/academy/core/__init__.py both grow a __getattr__-
based lazy resolver for ChemGraphLogicalAgent. The pure modules
stay eager so their public surface (~11 symbols) is reachable on
a CPU-only checkout. Accessing the lazy symbol without the extra
installed raises ImportError with an actionable
`pip install 'chemgraph[academy]'` hint.
* tests/test_academy_*.py and tests/test_tool_adapter_validation.py
(9 modules total) gain `pytest.importorskip("academy")` at the
top, so they skip cleanly when the extra is absent instead of
erroring at collection time.
Validated:
* Without academy-py: chemgraph.academy imports, all eager symbols
resolve, cli.trace loads, lazy ChemGraphLogicalAgent access
raises with hint, all 9 academy test modules skip cleanly.
* With academy-py: 48/48 academy tests pass.
Two-tier __init__ was necessary because pulling
chemgraph.academy.core.campaign transitively runs core/__init__.py,
which previously eager-imported core.agent. Both __init__s now
follow the same lazy pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
f1593ab ("revert: restore llm_agent.py to pre-academy shape") was meant to drop temporary event-callback wiring and turn-primitive imports, but its rewrite also dropped the calculator-availability injection that was on the file at the dev-globus fork point. The deletion was a merge-conflict casualty, not an intentional removal: get_calculator_selection_context() in schemas/ase_input.py:145 was left as dead code, and the two regression tests test_single_agent_initialization_injects_calculator_availability (in tests/test_graph_constructors.py and tests/test_graphs.py) have been failing on dev-globus ever since. Restore the 28-line wiring block verbatim from origin/dev:src/chemgraph/agent/llm_agent.py:464-490, plus the matching `from chemgraph.schemas.ase_input import get_available_calculator_names, get_calculator_selection_context, get_default_calculator_name` import at the top. Position in __init__: after the human-supervised system_prompt adjustment (line 324) and before the structured-output gate (line 326). All three instance attributes the block references (self.workflow_type, self.planner_prompt, self.executor_prompt) are set earlier in __init__ (lines 299, 308, 309), so the wiring order is safe. Both calculator-injection regression tests now pass; the 48-test academy suite is unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(academy,agent): PR-120 follow-up — optional-dep contract + calculator wiring
Prepare PR #120 for merge by dropping a non-functional workflow and fixing event/HPC-pickle regressions. The academy optional-dependency and calculator-context fixes originally bundled here are already on dev-globus (b5cac8b, d293cea), so this commit no longer carries them. - Remove single_agent_architector workflow: it was not registered in workflow_map and required a tools module that does not exist. Drop the graph, the CLI entry, and the related test parameters (including a stale graspa_agent case). - events: only emit llm_decision when the model requested tool calls; a plain text answer has no decision to report. - mcp: add explicit pickle-by-reference fix for the gRASPA _ls_remote_files TaskSpec callable; document that decorator-registered worker callables are already fixed by schema_fanout_tool. - tests: update the MACE worker test for the dropped full_output field; note in conftest that academy-only modules self-skip via importorskip.
The ground-truth generator's extract_output_json branch read a nested final_result['result'] key for the thermo/vib/ir drivers, but extract_output_json returns the flat ASEOutputSchema where those fields live at the top level. As a result vibrational frequencies and Gibbs free energy were silently dropped (e.g. q29 stored a single-point energy in place of frequencies; q17/q20/q30 stored single-point energy labelled as Gibbs free energy). Read these fields from the top level instead, and also accept the schema's dipole_value key. The structured-output judge's _compare_scalar ignored the property field, so a single-point energy within 5% of the Gibbs free energy passed silently. Add a normalized property-category check (gibbs/enthalpy/entropy/ dipole/energy) that tolerates phrasing differences but catches mismatched quantities. Also flag all-null expected ground truth as a failure with a parse_error instead of a trivial pass. Remove stale mcp_example scripts under new_eval.
The default dataset shipped under chemgraph/eval/data/ was stale and produced by the buggy ground-truth generator. Remove it; callers should pass an explicitly regenerated dataset via --dataset.
Extend the demo layer so run_ase (general ASE, selectable calculator) and run_graspa (GCMC) run across all execution backends, alongside the existing MACE thermo screen. No changes to the execution abstraction -- it was already backend-agnostic. - New backend-agnostic ASE MCP server (mcp/ase_mcp_hpc.py) mirroring mace_mcp_hpc.py: run_ase_single / run_ase_ensemble with a selectable calculator, inline/remote transport hook, and pickle-by-reference guards for Globus Compute. - New ase_input_schema_ensemble; calculator coercion factored into a shared helper used by both the single and ensemble schemas. - Demo harness (scripts/demo/_demo_chemistry.py) generalized into a workload registry (thermo / ase / graspa): build_ase_job, build_graspa_job, workload-aware submit_and_collect, inline full-output wrapper, gRASPA property extractor and summary table, ASE/gRASPA prompts, shared CLI helpers. - All demo_*_direct.py / demo_*_agent.py gain --workload (+ --calculator, --graspa-cifs) and per-workload MCP server selection. gRASPA is gated to HPC backends (SYCL binary path is baked into graspa_core); it exits non-zero on the local backend.
Add pluggable execution backends, HPC MCP servers, and Academy multi-agent campaigns
config.toml was added on main (e9e83bc "Restore root config.toml") but never existed on dev, which still gitignored it. Restore the file from main's tracked copy and drop the config.toml entry from .gitignore so both branches agree and the default config stays version-controlled.
Test fixes: - llm_agent: select the RAG/XANES default prompt via a prompt_is_default flag captured before the unsupervised-mode prompt mutation, so the default prompts are no longer masked by the rewritten system_prompt. - test_job_tracker: use asyncio.run instead of asyncio.get_event_loop().run_until_complete, which raises on Python 3.12. Lint fixes (ruff check .): - Move deprecation warnings.warn below imports in the deprecated parsl MCP modules (E402). - Hoist chemgraph.agent.events import to the top of turn.py (E402). - Add typing.Any import in academy daemon (F821). - Drop unused locals/imports and collapse multi-import lines (F841/F401/F541/E401).
llm_agent.py kept an old naive serialize_state that recursed through __dict__ with no depth or cycle guard, while turn.py has a hardened version with cycle detection and max_depth. On a cyclic LangGraph state this overflowed the (smaller) Windows stack in write_state. Remove the duplicate and import the hardened serialize_state from turn.py so write_state, the state return path, and cli/commands.py all share one implementation.
Update ChemGraph evaluation
Writers (smiles_to_coordinate_file, run_ase output) route relative paths through _resolve_path into CHEMGRAPH_LOG_DIR, but readers opened them from cwd -> FileNotFoundError on bare names like "water.xyz". Add _resolve_existing_path() and use it in run_ase, extract_output_json, file_to_atomsdata, generate_html. Prefers an existing path, else the log-dir location, else raw (so missing-file errors still surface). Add regression tests (EMT, no network).
Extends the CHEMGRAPH_LOG_DIR path resolution from the LangChain tool
readers to the MCP servers and the graspa/xanes cores, so a small model
that echoes back a bare filename ("water.xyz") for a file a sibling tool
wrote into the log dir still resolves it instead of failing with
FileNotFoundError.
- execution/utils.resolve_structure_files: resolve each listed filename
against the log dir (shared by the ase/mace/graspa/xanes _hpc ensemble
tools). Directory inputs are left unchanged.
- mcp/ase_mcp_hpc.py, mcp/mace_mcp_hpc.py: _embed_inline_if_local resolves
the bare name on the submitting host before its local-vs-remote check and
stores the resolved absolute path back into the job. Worker code unchanged.
- tools/graspa_core.py, tools/xanes_core.py: resolve input_structure_file via
_resolve_existing_path instead of Path(...).resolve(), so the single-
structure _hpc path (which delegates here) also resolves bare names. The
ase/mace cores already did this via run_ase_core.
- mcp/data_analysis_mcp.py: aggregate_simulation_results resolves each input.
- mcp/hpc_misc_mcp.py: inspect_json checks the log dir before its nearby-files
fallback.
- mcp/graspa_mcp_parsl.py, mcp/xanes_mcp_parsl.py: same fallback in their
copied list-resolution logic (deprecated servers; included for consistency).
The general server (mcp/mcp_tools.py) already delegates to the fixed core and
returns absolute paths; added tests to pin that behavior.
The _hpc worker functions are deliberately left untouched: when the backend
shares the filesystem the worker delegates to the (now-fixed) cores, and when
it does not the bare name is resolved on the submitting host before submission.
The multi-agent human-in-the-loop default changed to False, so unified_planner_router and construct_multi_agent_graph no longer route to / include the human_review path unless human_supervised=True is passed. Update the three multi-agent human-interrupt tests to opt in explicitly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correctness/robustness fixes surfaced during review of the dev integration:
- agent/turn.py: _tool_message_name no longer counts a named non-tool
message (e.g. a named AIMessage/HumanMessage in the Academy multi-agent
flow) as an executed tool, which had polluted executed_tool_names.
- execution/job_tracker.py:
* _save() tolerates non-JSON-serializable task results (default=str +
guard) so get_status/get_results can't crash when persist_file is set
(used by all *_mcp_hpc.py servers).
* register_batch() no longer blocks the full 3s globus wait for plain
concurrent.futures.Future objects (only ComputeFutures carry task_id).
* get_status() mutates task state under the (now re-entrant) lock,
matching the documented thread-safety contract.
- mcp/server_utils.py: run streamable-HTTP MCP servers with uvicorn
ws="none"; the HTTP transport doesn't use WebSockets, and loading
uvicorn's ws protocol failed against the websockets<11 pinned by
pyppeteer (missing ServerProtocol) -- broke every *_mcp_hpc.py server.
Testing:
- New regression tests: test_turn_executed_tools.py, test_job_tracker_hardening.py.
- Fixed stale test_tool_adapter_validation: drain the queued background
delivery before asserting peer receipt.
- New CI job 'test-extras' installs .[academy,parsl,globus_compute] and runs
the Academy + execution-backend tests (mocks/local subprocesses; live
globus/LLM stay gated). Core: 270 passed; with extras: 318 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflects the dev->main integration (execution backends, Academy multi-agent, CGFastMCP, HPC/Globus) landing via #139. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix relative path resolution in tool file readers
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrates 130 commits from
devintomain. This is a large integration bringing distributed execution backends, the Academy multi-agent module, the CGFastMCP framework, and HPC/Globus infrastructure intomain.Scope: ~137 files, +23.5k / −725 lines. Merge is clean and
config.tomlremains tracked on both branches (see Notes).Changes by area (with the PR that introduced each)
1. Pluggable execution backend layer (
src/chemgraph/execution/, ~10 files)Backend abstraction for Parsl, EnsembleLauncher (EL), and Globus Compute, with backend selection via
config.toml [execution]/ env vars.→ PRs #127, #131, #132, #133
2. Academy distributed multi-agent module (
src/chemgraph/academy/, ~39 files — new module)Distributed multi-agent screening, federated cross-HPC campaigns, HTTP exchange registration, operator-console dashboard, per-agent
allowed_toolswhitelist, and wakeups routed through the chemgraph turn primitive. Includes a largetests/test_academy_*.pysuite.→ Landed directly on
dev3. CGFastMCP framework + MCP migration (
src/chemgraph/mcp/, ~12 files)CGFastMCPbackend framework (tracker kwargs, pre-submit hook,schema_fanout_tool); migrated the XANES and gRASPA MCP servers; MACE MCP transport and persistence.→ PRs #134, #120
4. Globus Transfer + job persistence
GlobusTransfermanager and MCP file-staging tools, inline-structure transport gated to no-shared-filesystem backends,JobTrackerpersistence with Globus task-UUID round-trip, and Globus Compute executor recovery.→ PRs #120, #127
5. HPC config & Parsl hardening (
src/chemgraph/hpc_configs/,parsl_tools.py)Multi-node-safe MPI flavour per HPC system, configurable Parsl
worker_init, Aurora worker parameterization, clean DFK shutdown, and serialized MACE model loads.→ PRs #132, #133
6. Agent / model refactors
Single-agent workflows routed through
run_turn(agent/turn.py), dashboard event callbacks extracted toagent/events.py, shared LLM endpoint settings (models/settings.py,models/loader.py,models/openai.py), and version read from thechemgraphagentdistribution.7. Repo hygiene
Removed the dead
single_agent_architectorworkflow, dropped machine-specific Crux tests, and gitignoredruns/and**/*.model.config.tomlstays tracked.8. Remove evaluation dataset
Move evaluation dataset from the current repo to HF: https://huggingface.co/datasets/Autonomous-Scientific-Agents/groundtruth
Notes on config.toml
config.tomlwas added onmainand was missing fromdev(still gitignored there). A prep commit ondevrestoresconfig.tomlfrom main's copy and removes theconfig.tomlgitignore entry, so the file stays version-controlled on both branches and after merge.Rolled-up PRs
#120, #127, #131, #132, #133, #134