chore: apply release 0.4 delta to main

[Yunnglin]

Summary

Sync the release/0.4.0 runtime fixes onto current main while preserving main-only package metadata and optional dependencies.

• upgrade the Tinker optional dependency to tinker==0.16.1
• add ms-swift==4.1.3 to the Megatron extra
• align the Docker runtime with the release image, CUDA library path, pinned Redis/LGTM images, and tilelang runtime fix
• harden the Megatron server launch script around Redis shutdown/startup, LGTM readiness, and twinkle.server launch
• advertise the Megatron supported model in the server config
• sync the shared Tinker compat ragged-output handling for SFT/GRPO/DPO, including the empty pipeline-stage case

Validation

• conda run -n twinkle python -m pytest tests/server/model/test_tinker_compat_output.py -q
• conda run -n twinkle python -m pytest tests/server/model/test_mock_model.py -q
• conda run -n twinkle python -m pre_commit run --all-files

[gemini-code-assist]

Code Review

This pull request prepares the project for the 0.4.0 release by updating version numbers, pinning base image tags (Redis and Otel-LGTM) in the Dockerfile, and upgrading the tinker dependency to version 0.16.1. Additionally, the Dockerfile is updated to use python -m pip instead of pip. Feedback was provided to include the --no-cache-dir flag when installing the local package in the Dockerfile to minimize the final image size.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Copilot]

Pull request overview

This PR applies the remaining release/0.4 deltas onto main, focusing on Tinker compatibility (ragged logps handling), RL feature extraction for DPO, and deployment/runtime pinning updates (Docker, scripts, dependency pins).

Changes:

• Update TwinkleCompatModelBase output handling to correctly expand ragged microbatch logits/logps into per-datum rows, and add a dedicated regression test suite.
• Adjust extract_rl_features_for_loss to pad/stack ragged DPO ref_logps.
• Pin/adjust release-facing runtime artifacts (tinker pin, Docker base + Redis/LGTM images, Megatron cookbook config + run script tweaks).

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/server/model/test_tinker_compat_output.py	New tests covering tensor-row vs ragged-microbatch logps/logits output normalization and RL feature extraction behavior.
src/twinkle/server/model/backends/common.py	Switch output normalization from “cat to tensor” to “expand into per-sample rows” to support ragged microbatches and reference-forward retention.
src/twinkle/server/common/datum.py	Pad + stack ragged ref_logps into a batch tensor for DPO-style losses/metrics.
src/twinkle/model/megatron/megatron.py	Make grad norm tracking more robust to non-tensor scalar-like values.
pyproject.toml	Update optional deps: pin tinker==0.16.1; add ms-swift==4.1.3 to megatron extras.
Dockerfile	Pin Redis/LGTM/base image tags; adjust CUDA library path and pip installs for runtime deps.
cookbook/client/server/megatron/server_config.yaml	Add supported_models list for gateway validation/capabilities.
cookbook/client/server/megatron/run.sh	Improve Redis/LGTM lifecycle handling and switch server startup to python -m twinkle.server launch.