coldfront-initializer is a Django plugin (app) for ColdFront, an HPC resource allocation system. It loads initial data from YAML files into the ColdFront database. This is useful for:
- Importing data into a fresh ColdFront 2.x instance from YAML definitions
- Migrating from ColdFront 1.1.x to 2.0.x by exporting data to YAML and re-importing
- Testing with realistic fixture data
src/coldfront_initializer/
├── __init__.py # PluginConfig (Django app registration)
├── version.py # Version string
├── initializers/ # One initializer per model type
│ ├── base.py # BaseInitializer class, INITIALIZER_ORDER, registry
│ └── <model>_initializer.py # One file per model (e.g., users.py, projects.py)
├── management/commands/
│ ├── load_initializer_data.py # Loads YAML files into ColdFront DB
│ ├── copy_initializer_examples.py # Copies example YAMLs to a path
│ └── load_test_data.py # Wrapper that loads examples/ as test data
├── examples/ # Example YAML files (used as test data)
│ └── *.yml # One file per initializer, named <initializer_name>.yml
└── library/ # Community-maintained library of initial data
└── custom_fields/
└── custom_fields.yml
tests/
├── conftest.py # examples_dir + load_all fixtures
└── test_*.py # pytest + pytest-django tests
- Initializers live in
src/coldfront_initializer/initializers/, one file per model - Example YAML files live in
src/coldfront_initializer/examples/, one per initializer - Library YAML files live in
src/coldfront_initializer/library/<name>/— these are community-maintained data sets (e.g., standard custom fields for HPC centers) that can be loaded with--library
Each ColdFront model type has a dedicated initializer class in initializers/. All extend BaseInitializer (defined in base.py). The load order is defined by INITIALIZER_ORDER in base.py — this matters because some models depend on others.
Each initializer is registered in INITIALIZER_REGISTRY via register_initializer(). The initializers/__init__.py imports all modules to trigger registration.
Most initializers need no custom load_data method. Instead, they declare class variables that configure a generic pipeline in BaseInitializer:
| Class Variable | Purpose | Default |
|---|---|---|
MODEL |
The Django model class (required) | None |
REQUIRED_ASSOCS |
Required FK lookups {param: (Model, field)} |
{} |
OPTIONAL_ASSOCS |
Optional FK lookups (null-safe) {param: (Model, field)} |
{} |
OPTIONAL_M2M_ASSOCS |
Optional M2M relations {param: (Model, field)} |
{} |
RECORDS_ARE_DICT |
YAML records are dict-keyed (key = identifier) | False |
PRIMARY_KEY |
Field name for dict-keyed record keys | "name" |
MATCH_PARAMS |
Unique fields for get_or_create |
None → ["name", "slug"] |
The generic load_data pipeline:
- Normalize records — converts dict-keyed YAML into list-of-params format
_pre_process(params)hook — color mapping, schema validation, resource_object parsing- Pop well-known params —
custom_field_data,tags,created(only if model supports them) - Resolve required FKs — looks up model instances by field value
- Resolve optional FKs — same, but null-safe (handles
None/"null", catchesDoesNotExist) - Collect M2M values — pops M2M params before
split_params split_params+get_or_create— creates or retrieves the model instance- Set M2M values — applies collected M2M relations after creation
_post_process(obj, created, params)hook — project users, permissions, choices- Set custom fields and tags
Model capabilities (custom fields, tags, created date, slug field) are inferred at runtime via hasattr() and _meta.get_field(), not declared as flags.
class TenantInitializer(BaseInitializer):
MODEL = Tenant
OPTIONAL_ASSOCS = {"group": (TenantGroup, "name")}This single class does everything — no load_data method needed.
class ProjectInitializer(BaseInitializer):
MODEL = Project
REQUIRED_ASSOCS = {"owner": (User, "username")}
OPTIONAL_ASSOCS = {"tenant": (Tenant, "name"), "group": (Group, "name")}
def _pre_process(self, params):
self._usernames = params.pop("users", [])
def _post_process(self, obj, created, params):
for username in self._usernames:
user = User.objects.get(username=username)
ProjectUser.objects.get_or_create(user=user, project=obj)When YAML records use identifiers as keys (e.g., username: {details...}), set RECORDS_ARE_DICT = True and PRIMARY_KEY to the identifier field. The generic pipeline normalizes these into list-of-params format automatically.
load_initializer_data— Main command. Loads YAML files from--pathor--library. Reads files inINITIALIZER_ORDER, each named<initializer_name>.yml.copy_initializer_examples— Copies example YAML files to a user-specified directory.load_test_data— Convenience wrapper that callsload_initializer_data --path <examples_dir>.
- Each YAML file corresponds to one initializer type (e.g.,
users.yml,projects.yml). - Two formats are supported:
- List format (most initializers):
- name: foo\n slug: bar\n ... - Dict format (users, groups, roles, etc.):
username:\n password_hash: ...
- List format (most initializers):
- If the first line is
# cf_template, the file is treated as a Jinja2 template. - Custom fields are specified under
custom_field_data(legacycustom_fieldsis supported with a warning). - Tags are specified under
tags. - Foreign key references use string names that are resolved by
resolve_optional/resolve_required.
- Create a file in
initializers/(e.g.,my_model.py). - Define a class extending
BaseInitializerwithMODEL = MyModel. - Add
REQUIRED_ASSOCS,OPTIONAL_ASSOCS,OPTIONAL_M2M_ASSOCSas needed. - If records are dict-keyed, set
RECORDS_ARE_DICT = TrueandPRIMARY_KEY. - If special pre/post processing is needed, override
_pre_process/_post_process. - If the pipeline doesn't fit (rare), override
load_dataentirely. - Call
register_initializer("my_model", MyModelInitializer)at module level. - Add the import to
initializers/__init__.py. - Add
"my_model"toINITIALIZER_ORDERinbase.pyat the correct position. - Create an example YAML file in
examples/my_model.yml.
- FK resolution: Add entries to
REQUIRED_ASSOCSorOPTIONAL_ASSOCS. The model class can be a string (e.g.,"users.Group") for lazy imports. - M2M resolution: Add entries to
OPTIONAL_M2M_ASSOCS. Values are collected beforesplit_paramsand applied after creation. - Dict-keyed records: Set
RECORDS_ARE_DICT = True. The key becomesPRIMARY_KEYin the normalized params. - Skip a record on error: Set
params["_skip"] = Truein_pre_processand overrideload_datato check for it. - Null handling: Optional FKs automatically handle
None/"null"strings andDoesNotExistgracefully. - String model references: Use
"app.Model"format (e.g.,"users.Group","coldfront.ras.models.Project"). The resolver usesdjango.apps.apps.get_model().
- Formatter:
ruff format - Linter:
ruff check --fix - Run:
uv run ruff format && uv run ruff check --fix - Python: 3.12+
- Build system:
uvwithuv_build - Logging: Use
logging.getLogger(__name__)— neverprint(). - SPDX headers: Every
.pyfile starts with an SPDX copyright header.
COLDFRONT_ENV=.env.testing uv run -m pytest- Tests live in
tests/(configured via[tool.pytest]inpyproject.toml). DJANGO_SETTINGS_MODULEis set tocoldfront.config.settings; the.env.testingfile provides the required ColdFront env vars (e.g.ALLOWED_HOSTS).- pytest-django runs migrations once per session (not per test); each test
marked
@pytest.mark.django_dbruns inside a transaction that is rolled back afterward, so every test sees a clean, migrated DB.
- Name test files
test_*.pyand mark any DB-accessing test with@pytest.mark.django_db. - Fixtures (
tests/conftest.py):examples_dir— path tosrc/coldfront_initializer/examples/.load_all— loads every initializer from the example data inINITIALIZER_ORDERand returns{name: LoadResult}(None when no YAML). Use this to seed the DB before testing a specific initializer.
- Avoid temp files: pass YAML inline to
initializer.load_yaml_string(...)(construct the initializer with""as the path) instead of writing temp files. - Assert on
LoadResult(found,loaded,skipped,errors,warnings):loaded == foundanderrors == []mean the data loaded cleanly.loaded == MODEL.objects.count()verifies one DB row per loaded record (except create-always initializers like comments, which add one row per comment).
- Per-record behavior (skips, multiple comments, FKs) is verified by writing a
small inline YAML and loading it directly, then asserting on both the
LoadResultand the resulting DB rows.
# Run the test suite
COLDFRONT_ENV=.env.testing uv run -m pytest
# Load test data
uv run coldfront load_test_data -f
# Load custom YAML from a directory
uv run coldfront load_initializer_data --path /path/to/yaml/files
# Copy example files for editing
uv run coldfront copy_initializer_examples --path /path/for/example/files
# Format/lint code
uv run ruff format
uv run ruff check --fix
# Build and publish
uv build
uv publish- All initializers are registered via import in
initializers/__init__.py. Don't forget to add new imports there. - Load order in
INITIALIZER_ORDERmust be maintained — some models depend on others. - Model capabilities are inferred — you don't need to declare
supports_custom_fieldsorsupports_tags. The base class checkshasattr()at runtime. _pre_processand_post_processare no-ops by default. Only override them when needed.- For M2M fields, pop the values in
_pre_processand store them as instance variables. They would otherwise end up indefaultsand causeDirectAssignmenterrors. - For models without a
slugfield, the generic pipeline automatically popsslugfrom params if it's present in the YAML. split_paramsusesMATCH_PARAMS(or["name", "slug"]by default). For dict-keyed records,PRIMARY_KEYis automatically included.