feat: add video-to-video models (Kling O3 reference, 4K, v3 motion control)

[SecurityQQ]

Summary

• Add 6 new Kling video models with video input support to the SDK
• Make GenerateVideoPrompt accept both singular and array for images/audio/video fields
• Add video input to the video action with automatic routing (video+image → motion control, video-only → v2v reference)

New Models

SDK Model ID	fal Endpoint	Category	Price
kling-v3-4k	fal-ai/kling-video/o3/4k/image-to-video	image-to-video (4K)	$0.42/sec
kling-v3-ref	fal-ai/kling-video/o3/pro/reference-to-video	reference-to-video	$0.112-0.14/sec
kling-v3-4k-ref	fal-ai/kling-video/o3/4k/reference-to-video	reference-to-video (4K)	$0.42/sec
kling-v3-v2v-ref	fal-ai/kling-video/o3/standard/video-to-video/reference	v2v reference	$0.126/sec
kling-v3-motion	fal-ai/kling-video/v3/pro/motion-control	motion control (up to 30s)	$0.168/sec
kling-v3-motion-standard	fal-ai/kling-video/v3/standard/motion-control	motion control (up to 30s)	$0.126/sec

Changes by file

AI SDK Provider Layer

• src/ai-sdk/providers/fal.ts — Added REFERENCE_VIDEO_MODELS and V2V_REFERENCE_MODELS maps, kling-v3-4k to VIDEO_MODELS, v3 motion models to MOTION_CONTROL_MODELS. New dispatch branches for reference-to-video and v2v-reference with proper input mapping (start/end images, video_url, image_urls, generate_audio, keep_audio). New resolver methods.
• src/ai-sdk/providers/model-rules.ts — Duration rules for all 4 new duration-based models (stringIntDuration(3, 15, 5))
• src/ai-sdk/generate-video.ts — GenerateVideoPrompt type now accepts DataContent | Array<DataContent> for images/audio/video (backwards compatible). normalizePrompt() uses toArray() helper for uniform handling.

Definition Layer

• src/definitions/models/kling.ts — 6 new ModelDefinition exports with Zod schemas and pricing. Motion control models default to character_orientation: "video" (supports up to 30s reference video).
• src/definitions/models/index.ts — Registered all 6 new definitions in exports and allModels[]
• src/definitions/actions/video.ts — Added video input field. Routes: video+image → motionControl(), video-only → videoToVideoReference()

Legacy Provider

• src/providers/fal.ts — Added motionControl() and videoToVideoReference() convenience methods with ensureUrl() for local file upload support

UI/DX

• src/react/renderers/progress.ts — Time estimates for all 6 new models
• src/core/registry/resolver.ts — Added v2v and vid2vid aliases

Usage

# Motion control (video + image → transfer motion to character)
varg run kling-v3-motion --image_url https://... --video_url https://... --prompt "dancing"

# V2V reference (video → new video preserving motion/camera)
varg run kling-v3-v2v-ref --video_url https://... --prompt "replace characters"

# Video action with automatic routing
varg run video --video input.mp4 --image character.png --prompt "dancing"

// Programmatic API — singular or array both work
const { video } = await generateVideo({
  model: fal.videoModel("kling-v3-motion"),
  prompt: {
    text: "A woman dancing",
    images: imageData,        // singular works
    video: [vid1, vid2],      // array also works
  },
});

Verification

• bunx tsc --noEmit — zero new errors (all errors are pre-existing Uint8Array/heygen/captions issues)
• bun test src/ai-sdk/providers/fal.test.ts — 13/13 pass
• Model registration verified: all 6 models resolve to correct fal endpoints

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

walkthrough

adds expanded kling video model variants and new video transformation capabilities. generatevideoprompt now accepts singular or array values for media inputs. new fal provider methods enable motion-control and reference-guided video-to-video generation, with corresponding model definitions, duration rules, and progress estimates.

changes

cohort / file(s)	summary
type definitions src/ai-sdk/generate-video.ts	modified generatevideoprompt to accept images, audio, and video as either singular datacontent or arrays; added toarray helper for normalization.
model definitions src/definitions/models/kling.ts, src/definitions/models/index.ts	introduced 6 new kling variants: kling4k, klingref, kling4kref, klingv2vref, klingv3motion, klingv3motionstd with zod schemas for input validation, pricing, and fal provider mappings.
duration & routing rules src/ai-sdk/providers/model-rules.ts, src/core/registry/resolver.ts	extended modeldurations with 4 new kling model identifiers (kling-v3-4k, kling-v3-ref, kling-v3-4k-ref, kling-v3-v2v-ref); added alias mappings for v2v → "video-to-video".
fal provider expansion src/ai-sdk/providers/fal.ts, src/providers/fal.ts	added video-to-video reference and motion-control model routing in doGenerate; implemented new motioncontrol and videotovideoreference methods with file uploads and fal subscribe calls.
action & ui integration src/definitions/actions/video.ts, src/react/renderers/progress.ts	updated video action to handle new video input and route to motion-control or v2v-reference flows; added time estimates for all new kling variants in progress tracking.

estimated code review effort

🎯 3 (moderate) | ⏱️ ~25 minutes

possibly related prs

• feat: add 11 enhance/upscale models (7 image + 4 video) #185: modifies falvideomodel.dogenerate to add routing for new video model variants and endpoints
• fix: normalize video duration per model rules via Zod schemas #194: extends model-duration normalization logic and updates fal provider for new kling variants
• feat: add Kling 3.0 (O3) video model support #121: foundational pr extending kling/o3 video model work with model definitions, provider mappings, and progress estimates

poem

lights, camera, motion! 🎬
new kling variants dance across the fal,
videos become characters, characters find their groove,
reference frames guide the transformation—meow, what a move ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	the title clearly and specifically summarizes the main change: adding six new video-to-video models (kling o3 reference, 4k, v3 motion control) to the sdk.
Description check	✅ Passed	the description is well-structured, detailed, and directly related to the changeset. it covers new models, file changes, usage examples, and verification steps.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/video-to-video-models

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}