Release v1.3.0

.github/workflows/link-issues-by-milestone.yml

@@ -0,0 +1,16 @@
+name: Link Issue to Milestone Parent
+
+on:
+  issues:
+    types: [milestoned]
+
+jobs:
+  link-issue:
+    if: github.event.issue.milestone != null
+    uses: AllenNeuralDynamics/.github/.github/workflows/util-link-issues-by-milestone.yml@main
+    with:
+      issue-number: ${{ github.event.issue.number }}
+      issue-id: ${{ github.event.issue.id }}
+      milestone-description: ${{ github.event.issue.milestone.description }}
+    secrets:
+      service-token: ${{ secrets.SERVICE_TOKEN }}

README.md

@@ -11,7 +11,7 @@ Repository to contain code that will parse source files into aind-data-schema mo
 
 ## Usage
 
-The `GatherMetadataJob` is used to create the `data_description.json` and pull the `subject.json` and `procedures.json` from `aind-metadata-service`. Users are expected to provide the `instrument.json` and the `acquisition.json` as well as optional `processing.json`, `quality_control.json` and `model.json`. The job will attempt to validate all of the metadata files, displaying errors, and then will save all metadata fields into the selected folder.
+The `GatherMetadataJob` is used to create the `data_description.json` and pull the `subject.json` and `procedures.json` from `aind-metadata-service`. Users are expected to provide the `instrument.json` and the `acquisition.json` as well as optional `processing.json`, `quality_control.json` and `model.json`. If a user provides `procedures.json`, it will be merged with the procedures fetched from the service (subject and specimen procedures are deduplicated). The job will attempt to validate all of the metadata files, displaying errors, and then will save all metadata fields into the selected folder.
 
 ### Using the GatherMetadataJob
 
@@ -51,8 +51,8 @@ If no exact match exists, it will construct, fetch, merge or run mappers to gene
 | File | Method 1 | Method 2 | Method 3 |
 |------|----------|----------|----------|
 | data_description.json | Exact match in input directory | Construct from settings / fetch from metadata-service |  |
-| subject.json | Exact match in input directory | Fetch from metadata-service (requires subject_id) |  |
-| procedures.json | Exact match in input directory | Fetch from metadata-service (requires subject_id) |  |
+| subject.json | Exact match in input directory | Fetch from metadata-service (requires subject_id) | Constructed locally when subject_id is "calibration" |
+| procedures.json | Fetch from metadata-service (requires subject_id) | Merge with user-provided procedures but default to user-provided if there are any duplicates | Constructed locally (empty) when subject_id is "calibration" |
 | acquisition.json | Exact match in input directory | Run mappers on `<mapper>.json` files (and merge) | Merge all `acquisition*.json` files |
 | instrument.json | Exact match in input directory | Fetch from metadata-service (requires instrument_id) | Merge all `instrument*.json` files |
 | processing.json | Exact match in input directory |  |  |
@@ -71,6 +71,9 @@ When mappers are developed from the `BaseMapper` class and registered in `mapper
 
 - **`acquisition_start_time`** (datetime, optional): Acquisition start time in ISO 8601 format. This setting should only be used when an `acquisition.json` is not available.
 
+- **`subject_settings`** (optional): Settings for subject metadata. Only used when `subject_id` is `"calibration"`.
+  - **`calibration_object`** ([CalibrationObject](https://aind-data-schema.readthedocs.io/en/latest/), optional): A `CalibrationObject` from `aind_data_schema.components.subjects`. When `subject_id` is `"calibration"`, the metadata service is not contacted — instead a `Subject` is constructed locally using this object and an empty `Procedures` (no subject or specimen procedures). If omitted, a default empty `CalibrationObject` is used.
+
 - **`instrument_settings`**:
   - **`instrument_id`** (str): ID for the instrument used in data collection. When set, the instrument.json will attempt to be fetched from the metadata-service and saved as `instrument_<modality-abbreviation(s)>.json`. If multiple `instrument*.json` files exist after fetching they will be merged.
 
@@ -113,6 +116,35 @@ job = GatherMetadataJob(job_settings=job_settings)
 job.run_job()
 ```
 
+#### Calibration sessions
+
+When collecting data with a calibration object rather than a live subject, set `subject_id` to `"calibration"`. The job will skip the metadata service entirely and construct `subject.json` and `procedures.json` locally.
+
+```python
+from aind_data_schema.components.subjects import CalibrationObject
+from aind_data_schema_models.modalities import Modality
+from aind_metadata_mapper.gather_metadata import GatherMetadataJob
+from aind_metadata_mapper.models import JobSettings, DataDescriptionSettings, SubjectSettings
+
+job_settings = JobSettings(
+    output_dir="/path/to/output",
+    subject_id="calibration",
+    data_description_settings=DataDescriptionSettings(
+        project_name="<project-name>",
+        modalities=[Modality.ECEPHYS],
+    ),
+    subject_settings=SubjectSettings(
+        calibration_object=CalibrationObject(
+            description="Neuropixels dummy probe",
+            empty=False,
+        )
+    ),
+)
+
+job = GatherMetadataJob(job_settings=job_settings)
+job.run_job()
+```
+
 #### Validation settings
 
 - **`raise_if_invalid`** (bool, default=False): Controls validation behavior:
@@ -134,6 +166,30 @@ You probably shouldn't be modifying these.
   - `metadata_service_procedures_endpoint` (default="/api/v2/procedures/")
   - `metadata_service_instrument_endpoint` (default="/api/v2/instrument/")
 
+### Instrument CLI
+
+The `aind-instrument` command lets you upload and retrieve instruments from the metadata service without writing code.
+
+```bash
+# Upload an instrument
+aind-instrument upload instrument.json
+
+# Upload and overwrite an existing record
+aind-instrument upload instrument.json --replace
+
+# Keep the modification date from the file instead of updating to today
+aind-instrument upload instrument.json --no-update-modification-date
+
+# Get the latest record for an instrument
+aind-instrument get 422_MESO2_20241017
+
+# Get a specific version by modification date
+aind-instrument get 422_MESO2_20241017 --modification-date 2024-10-28
+
+# Save to a file instead of printing to stdout
+aind-instrument get 422_MESO2_20241017 --output-directory ./output
+```
+
 ### Developing Mappers
 
 Each MapperJob class should inherit from `BaseMapper` in `base.py`. The only parameter should be the `MapperJobSettings` from `base.py`. You cannot add additional parameters to your job or it will not be possible for it to be run automatically on the data-transfer-service. GatherMetadataJob will then run your mappers automatically when it detects the extracted metadata output.

pyproject.toml

@@ -21,8 +21,8 @@ dependencies = [
     "pyyaml",
     "pydantic-settings>=2.8.0",
     "jsonschema",
-    "aind-data-schema>=2.6.0,<3",
-    "aind-metadata-extractor==0.3.9",
+    "aind-data-schema>=2.7.1,<3",
+    "aind-metadata-extractor==0.3.11",
 ]
 
 [project.optional-dependencies]
@@ -40,6 +40,9 @@ doc = [
     "furo"
 ]
 
+[project.scripts]
+aind-instrument = "aind_metadata_mapper.cli.instrument:main"
+
 [tool.setuptools.packages.find]
 where = ["src"]
 

src/aind_metadata_mapper/__init__.py

@@ -1,3 +1,3 @@
 """Init package"""
 
-__version__ = "1.2.0"
+__version__ = "1.3.0"

src/aind_metadata_mapper/cli/__init__.py

@@ -0,0 +1 @@
+"""CLI tools for AIND metadata mapper."""

src/aind_metadata_mapper/cli/instrument.py

@@ -0,0 +1,115 @@
+"""CLI for instrument database operations.
+
+Provides ``upload`` and ``get`` subcommands for working with the AIND
+instrument metadata service.
+
+Usage::
+
+    aind-instrument upload instrument.json
+    aind-instrument upload instrument.json --replace
+    aind-instrument upload instrument.json --no-update-modification-date  # keep date from file
+    aind-instrument get 422_MESO2_20241017
+    aind-instrument get 422_MESO2_20241017 --modification-date 2024-10-28
+    aind-instrument get 422_MESO2_20241017 --output-directory ./output
+"""
+
+import logging
+import sys
+from pathlib import Path
+from typing import Optional
+
+from pydantic import BaseModel, Field
+from pydantic_settings import CliApp, CliPositionalArg, CliSubCommand
+
+from aind_metadata_mapper.utils import INSTRUMENT_BASE_URL, get_instrument, save_instrument
+
+logger = logging.getLogger(__name__)
+
+BASE_URL_FIELD = Field(
+    default=INSTRUMENT_BASE_URL,
+    description="Base URL for the instrument metadata service.",
+)
+
+
+class Upload(BaseModel):
+    """Upload an instrument JSON file to the metadata service database."""
+
+    file: CliPositionalArg[Path] = Field(
+        description="Path to an instrument JSON file to upload.",
+    )
+    replace: bool = Field(
+        default=False,
+        description=("If set, overwrite an existing record with the same " "instrument_id and modification_date."),
+    )
+    update_modification_date: bool = Field(
+        default=True,
+        description=(
+            "Update modification_date to today's date."
+            " Use --no-update-modification-date to keep the date from the file."
+        ),
+    )
+    base_url: str = BASE_URL_FIELD
+
+    def cli_cmd(self) -> None:
+        """Upload an instrument JSON to the database."""
+        try:
+            save_instrument(
+                self.file,
+                replace=self.replace,
+                update_modification_date=self.update_modification_date,
+                base_url=self.base_url,
+            )
+        except Exception as e:
+            logger.error(f"Upload failed: {e}")
+            sys.exit(1)
+
+
+class Get(BaseModel):
+    """Retrieve an instrument record from the metadata service database."""
+
+    instrument_id: CliPositionalArg[str] = Field(
+        description="Instrument identifier to look up.",
+    )
+    modification_date: Optional[str] = Field(
+        default=None,
+        description=("Specific modification date (YYYY-MM-DD) to retrieve. " "If omitted, returns the latest record."),
+    )
+    output_directory: Optional[Path] = Field(
+        default=None,
+        description=("Directory to save the instrument JSON file. " "If omitted, prints the JSON to stdout."),
+    )
+    base_url: str = BASE_URL_FIELD
+
+    def cli_cmd(self) -> None:
+        """Get an instrument record and display or save it."""
+        result = get_instrument(
+            self.instrument_id,
+            modification_date=self.modification_date,
+            output_directory=self.output_directory,
+            base_url=self.base_url,
+        )
+        if result is None:
+            logger.error(f"Instrument '{self.instrument_id}' not found.")
+            sys.exit(1)
+        if self.output_directory is None:
+            print(result.model_dump_json(indent=3))
+
+
+class InstrumentCLI(BaseModel):
+    """CLI for instrument database operations."""
+
+    upload: CliSubCommand[Upload]
+    get: CliSubCommand[Get]
+
+    def cli_cmd(self) -> None:
+        """Dispatch to the active subcommand."""
+        CliApp.run_subcommand(self)
+
+
+def main() -> None:
+    """Entry point for the aind-instrument CLI."""
+    CliApp.run(InstrumentCLI)
+
+
+if __name__ == "__main__":
+    main()