feat: Extend Lottie specification with VideoFrame asset support by stepancar · Pull Request #149 · lottie/lottie-spec

stepancar · 2025-08-21T09:28:29Z

Motivation

Currently, raster animations in Lottie are supported as sequences of frames where images are embedded directly. Since no interframe coding happens under the hood, raster animations are very large. Lottie is not just a vector format, it is also used as a universal animation format. Adding support for frames extracted from video resources makes animation size much smaller. Modern systems support a variety of codecs, so video processing and frame extraction will not be a problem.

Changes:
Add VideoAsset with id, resourceUrl, duration, width, height.
Add VideoFrame asset, similar to ImageAsset, but referencing a VideoAsset by id and specifying a timestamp.

This is a painless schema change. Players can keep their existing image-sequence engines: when encountering VideoFrame, simply extract the frame from the video, convert it to a base64 PNG, and treat it as a normal ImageAsset.

Effectively:

lottieWithVideoFrames -> lottieWithImageSequence

Before

 "assets": [
        {
            "id": "imgSeq_0",
            "w": 420,
            "h": 420,
            "t": "seq",
            "u": "",
            "p": "base64......loong string"
            "e": 1,
       },
       {
            "id": "imgSeq_0",
            "w": 420,
            "h": 420,
            "t": "seq",
            "u": "",
            "p": "base64...... .loong string"
            "e": 1,
       }, ....
       {
            "id": "sequence_0",
            "layers": [
                   {
                    "ty": 2,
                    "sc": "#00ffff",
                    "refId": "imgSeq_0",
                    "ks": {
                       ...
                    },
                    "ip": 0,
                    "st": 0,
                    "op": 1,
                    "sr": 1,
                    "bm": 0
                },
                 {
                    "ty": 2,
                    "sc": "#00ffff",
                    "refId": "imgSeq_0",
                    "ks": {
                       ...
                    },
                    "ip": 0,
                    "st": 0,
                    "op": 1,
                    "sr": 1,
                    "bm": 0
                }
            ]
       }
]

After

"assets": [
        {
            "id": "video_0",
            "ty": 2,
            "w": 420,
            "h": 420,
            "u": "",
            "p": "base64......loong string with compressed frames"
            "e": 1,
       },
       {
            "id": "imgSeq_0",
            "ty": 3,
            "w": 420,
            "h": 420,
            "t": 1000, // 1 second
            "u": "",
           "vsid": "video_0",
            "e": 1,
       },
       {
            "id": "imgSeq_1",
            "ty": 3,
            "w": 420,
            "h": 420,
            "t": 1040, // 1 second 40ms (next frame)
            "u": "",
           "vsid": "video_0",
            "e": 1,
       },
       {
            "id": "sequence_0",
            "layers": [
                   {
                    "ty": 2,
                    "sc": "#00ffff",
                    "refId": "imgSeq_0",
                    "ks": {
                       ...
                    },
                    "ip": 0,
                    "st": 0,
                    "op": 1,
                    "sr": 1,
                    "bm": 0
                },
                 {
                    "ty": 2,
                    "sc": "#00ffff",
                    "refId": "imgSeq_0",
                    "ks": {
                       ...
                    },
                    "ip": 0,
                    "st": 0,
                    "op": 1,
                    "sr": 1,
                    "bm": 0
                }
            ]
       }
]

CLA PR

cla-bot · 2025-08-21T09:28:31Z

We require contributors to sign our Contributor License Agreement, and we don't have @stepancar on file. In order for us to review and merge your code, please contact @heathmill or @jcgregorio to get yourself added.

mbasaglia · 2025-08-21T09:37:04Z

+                "p": {
+                    "title": "File Name",
+                    "description": "Name of the video file or a data url",
+                    "type": "string"
+                },
+                "u": {
+                    "title": "File Path",
+                    "description": "Path to the video file",
+                    "type": "string"
+                },
+                "e": {
+                    "title": "Embedded",
+                    "description": "If '1', 'p' is a Data URL",
+                    "$ref": "#/$defs/values/int-boolean"
+                }
+            },
+            "allOf": [
+                {
+                    "if": {
+                        "properties": {
+                            "e": {"const": 1}
+                        },
+                        "required": ["e"]
+                    },
+                    "then": {
+                        "properties": {
+                            "p": {"$ref": "#/$defs/values/data-url"}
+                        }
+                    }
+                }


I'd extract this into an abstract file-asset schema so we can share the definition between image and video assets (and other future assets

@mbasaglia Added file-asset

mbasaglia · 2025-08-21T09:39:17Z

+                "w": {
+                    "title": "Width",
+                    "description": "Width of the frame",
+                    "type": "number"
+                },
+                "h": {
+                    "title": "Height",
+                    "description": "Height of the frame",
+                    "type": "number"
+                }


The video asset schema also has w and h, are they needed here as well? if so, what's their semantic?

@mbasaglia I was thinking about extracting a frame from VideoResource, specifying width, height, x, y from the top left corner. But you are right, as a first iteration we can drop w/h from frame

mbasaglia · 2025-08-21T09:40:05Z

+        },
+        {
+            "type": "object",
+            "properties": {


I think we should introduce a ty for assets (not strictly required for this proposal)

@mbasaglia , I agree. Added ty prop

mbasaglia · 2025-08-21T09:40:38Z

+
+{schema_object:assets/video-frame}
+
+Video frame assets reference a video asset by ID and specify a timestamp to extract a specific frame. Players MUST extract the frame at the exact timestamp specified, or the closest available frame if the exact timestamp is not available. The extracted frame should be treated as a static image for rendering purposes.


should

Is this a SHOULD or a MUST?

This point is up for discussion.

The advantage of using MUST is that Lottie players could extract frames on the fly, which speeds up animation initialization and reduces memory usage before playback starts. At the same time, if players don’t treat the extracted frame as a static image but instead process it in real time, it could impose constraints on the video itself - for example, requiring more keyframes to ensure fast seeking, which would increase file size.

That’s why I would vote for MUST.

In the future, if we want to support optimization for initialization we could extend videoFrame with initialization: static | dynamic param to achieve that.

cla-bot · 2025-08-31T12:54:21Z

We require contributors to sign our Contributor License Agreement, and we don't have @stepancar on file. In order for us to review and merge your code, please contact @heathmill or @jcgregorio to get yourself added.

cla-bot · 2025-08-31T12:56:37Z

We require contributors to sign our Contributor License Agreement, and we don't have @stepancar on file. In order for us to review and merge your code, please contact @heathmill or @jcgregorio to get yourself added.

stepancar · 2025-08-31T13:11:45Z

@heathmill or @jcgregorio Hello. I have signed CLA, but bot is still complaining in this PR. Could you check? Thank you!

cla-bot · 2025-08-31T13:34:20Z

We require contributors to sign our Contributor License Agreement, and we don't have @stepancar on file. In order for us to review and merge your code, please contact @heathmill or @jcgregorio to get yourself added.

cla-bot · 2025-08-31T14:04:35Z

We require contributors to sign our Contributor License Agreement, and we don't have @stepancar on file. In order for us to review and merge your code, please contact @heathmill or @jcgregorio to get yourself added.

cla-bot · 2025-08-31T14:31:32Z

We require contributors to sign our Contributor License Agreement, and we don't have @stepancar on file. In order for us to review and merge your code, please contact @heathmill or @jcgregorio to get yourself added.

cla-bot · 2025-08-31T14:33:48Z

We require contributors to sign our Contributor License Agreement, and we don't have @stepancar on file. In order for us to review and merge your code, please contact @heathmill or @jcgregorio to get yourself added.

cla-bot · 2025-08-31T14:51:35Z

We require contributors to sign our Contributor License Agreement, and we don't have @stepancar on file. In order for us to review and merge your code, please contact @heathmill or @jcgregorio to get yourself added.

mbasaglia · 2025-09-01T08:47:45Z

@cla-bot check

cla-bot · 2025-09-01T08:47:49Z

The cla-bot has been summoned, and re-checked this pull request!

mbasaglia · 2025-09-01T08:48:17Z

ok I fixed the bot

stepancar · 2025-09-01T11:49:38Z

ok I fixed the bot

Thank you!

fmalita

What is the authoring story for these asset types, are you planning to add support in existing or new tools?

fmalita · 2025-09-02T16:28:56Z

+                    "type": "integer",
+                    "const": 3
+                },
+                "vsid": {


Naming bike-shed: normally sid refers for "slot ID", which I guess could be adopted here if we squint. But since we're not dealing with real slots, that may be confusing.

How about refId, which is the normal way to reference assets?

Agree, refId makes more sense, updated

fmalita · 2025-09-02T16:34:36Z

+                },
+                "t": {
+                    "title": "Timestamp",
+                    "description": "Timestamp in seconds where to extract the frame from the video",


How does this interact with start-time/time-stretch/general time remapping in referencing layers?

Also actual time units seem a bit odd since everything else in Lottie is expressed as frame indices (ip, op, etc).

Actually, you are right, we can define time in frames here

fmalita · 2025-09-02T16:42:30Z

+
+{schema_object:assets/video-frame}
+
+Video frame assets reference a video asset by ID and specify a timestamp to extract a specific frame. Players MUST extract the frame at the exact timestamp specified, or the closest available frame if the exact timestamp is not available. The extracted frame should be treated as a static image for rendering purposes.


Can the same functionality be achieved using time remapping to lock a video layer's time? IOW do we actually need video frame assets, or would video assets be sufficient?

I can think of some advantages to having a dedicated asset type, mostly on the implementation side (caching) - but there are also some downsides (more verbose than video asset + time-locked video layer).

The main motivation of having VideoFrame is ability to set time or frame number, I didn't find better way to express it with existing lottie primitives

stepancar · 2025-09-09T10:07:40Z

What is the authoring story for these asset types, are you planning to add support in existing or new tools?

These assets are created in any raster editor and then exported as Lottie files with base64-encoded individual frames.

I’m not sure how easy it would be to extend Bodymovin to support wrapping a sequence of images into a video/animated image.
I still haven’t figured out where the source code is located. Therefore, I was planning to make a website where a user could upload a Lottie file and receive a compressed Lottie. I also planned to support this spec extension in lottie-web.

b-wils · 2025-10-16T18:29:45Z

The source code for bodymovin is here- https://github.com/bodymovin/bodymovin-extension. FYI it took me a bit to get the dev environment running, particularly if you are on an ARM system.

Also did you have the authoring site/updated player available? Part of the spec process is to have at least an end to end demo of the feature.

One other thing is we wanted to be sure that the video asset could also be used for video playback. I think this still looks good but we may want to flesh out that support as well.

feat(*): add video

a6a7029

mbasaglia added the Proposal (RFC 1) RFC Stage 1 label Aug 21, 2025

mbasaglia reviewed Aug 21, 2025

View reviewed changes

Aidosmf added this to lottie-spec Aug 26, 2025

Aidosmf moved this to In Progress in lottie-spec Aug 26, 2025

feat(*): add file asset

37e5f89

Merge branch 'main' into feat/add-video

6ede4d5

refactor(*): add ty property

aa467d2

refactor(*): add doc to helpers md

a2b0026

refactor(*): add ty to files

b31bc96

refactor(*): add ty to files

3d75b48

refactor(*): add ty to files

cf72cae

cla-bot Bot added the cla-signed label Sep 1, 2025

stepancar mentioned this pull request Sep 1, 2025

asset.t is not defined #154

Closed

fmalita reviewed Sep 2, 2025

View reviewed changes

fix(*): rename vsid to refid

04831c3


		{schema_object:assets/video-frame}

		Video frame assets reference a video asset by ID and specify a timestamp to extract a specific frame. Players MUST extract the frame at the exact timestamp specified, or the closest available frame if the exact timestamp is not available. The extracted frame should be treated as a static image for rendering purposes.

Conversation

stepancar commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cla-bot Bot commented Aug 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cla-bot Bot commented Aug 31, 2025

Uh oh!

cla-bot Bot commented Aug 31, 2025

Uh oh!

stepancar commented Aug 31, 2025

Uh oh!

cla-bot Bot commented Aug 31, 2025

Uh oh!

cla-bot Bot commented Aug 31, 2025

Uh oh!

cla-bot Bot commented Aug 31, 2025

Uh oh!

cla-bot Bot commented Aug 31, 2025

Uh oh!

cla-bot Bot commented Aug 31, 2025

Uh oh!

mbasaglia commented Sep 1, 2025

Uh oh!

cla-bot Bot commented Sep 1, 2025

Uh oh!

mbasaglia commented Sep 1, 2025

Uh oh!

stepancar commented Sep 1, 2025

Uh oh!

fmalita left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmalita Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stepancar commented Sep 9, 2025

Uh oh!

b-wils commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

stepancar commented Aug 21, 2025 •

edited

Loading

fmalita Sep 2, 2025 •

edited

Loading