Question with fetching flv stream frame render PTS

[ikigai-kevin-k]

Dear author,

I want to know how to correctly obtain the pts timestamp for each frame when playing an FLV stream.
I tried using the FRAME_RENDER event type to pass pts, but I can only get the pts for I-frames and not P-frames (in my case, there are no B-frame types).

My pushed stream has 1 I-frame per 25 frames and with ~24 fps, this is why I know I only fetch the I-frame pts from the browser console log ------ the log frequency is ~1s.

I've attempted modifications in AVPlayer, Controller, VideoRenderPipeline, avframe, etc., but all failed.
I noticed I couldn't see my debug logs about render events/pts in the dev tool console panel.
I'm starting to suspect this might be related to IPC message passing: whether the render task being independent from the player main thread as a worker is preventing me from getting pts messages, but I'm not sure.
I just want to achieve something similar to ffprobe:

ffprobe -i xxx.flv -select_streams v -show_frames -show_entries frame=pts_time,pict_type -of csv=p=0
...
0.010000,I,H.26[45] User Data Unregistered SEI message
0.051000,P
0.093000,P
0.135000,P
0.176000,P
0.218000,P
0.260000,P
0.301000,P
0.343000,P
0.385000,P
0.426000,P
0.468000,P
0.510000,P
0.551000,P
0.593000,I
0.635000,P
0.676000,P
0.718000,P
0.760000,P
0.801000,P
0.843000,P
0.885000,P
0.926000,P
0.968000,P
1.010000,P
1.051000,I,H.26[45] User Data Unregistered SEI message
1.093000,P
1.135000,P
1.176000,P
1.218000,P
1.260000,P
1.301000,P
1.343000,P
1.385000,P
1.426000,P
1.468000,P
1.510000,P
1.551000,P
1.593000,I
1.635000,P
1.676000,P
1.718000,P
1.760000,P
1.801000,P
1.843000,P
1.885000,P
1.926000,P
1.968000,P
2.010000,P
2.051000,P
2.093000,I,H.26[45] User Data Unregistered SEI message
...

Please help me for solving this problem, thank you!

[zhaohappy]

Can you post your code snippet so I can analyze the reason?

[ikigai-kevin-k]

Sorry, I've tried a lot of staffs, so it would look a little bit messy...

diff --git a/src/avformat/formats/IFlvFormat.ts b/src/avformat/formats/IFlvFormat.ts
index a9d94b2..8a9fb65 100644
--- a/src/avformat/formats/IFlvFormat.ts
+++ b/src/avformat/formats/IFlvFormat.ts
@@ -79,6 +79,8 @@ export default class IFlvFormat extends IFormat {
 
   private firstTagPos: int64
 
+  private context: any
+
   constructor(options: FlvFormatOptions = {}) {
     super()
 
@@ -92,6 +94,43 @@ export default class IFlvFormat extends IFormat {
     if (formatContext.ioReader) {
       formatContext.ioReader.setEndian(true)
     }
+
+    // 初始化 context
+    this.context = {
+      keyframeFilePositions: [],
+      keyFrameTimes: [],
+      lastkeyframelocation: 0,
+      lastkeyframetimestamp: 0n,
+      lasttimestamp: 0n,
+      framerate: 0,
+      filesize: 0,
+      audioSize: 0,
+      videosize: 0,
+      datasize: 0,
+      duration: 0,
+      scriptWrote: false,
+      frameCount: 0,
+      firstKeyframePositionWrote: false,
+      videoMetadataWrote: false,
+      frameInfo: {
+        pts_time: 0,
+        pict_type: '',
+        isKeyFrame: false,
+        codecType: 0,
+        size: 0,
+        videoInfo: {
+          width: 0,
+          height: 0,
+          frameType: 0,
+          codecId: 0
+        },
+        audioInfo: {
+          channels: 0,
+          sampleRate: 0,
+          codecId: 0
+        }
+      }
+    }
   }
 
   public async readHeader(formatContext: AVIFormatContext): Promise<number> {
@@ -346,6 +385,26 @@ export default class IFlvFormat extends IFormat {
             await this.readAVPacketData(formatContext, stream, avpacket, size - 1)
           }
         }
+
+        // 更新 frameInfo
+        this.context.frameInfo = {
+          pts_time: Number(timestamp) / 1000, // 轉換為秒
+          pict_type: (((videoHeader & 0x70) >> 4) === 1) ? 'I' : 'P',
+          isKeyFrame: (((videoHeader & 0x70) >> 4) === 1),
+          codecType: AVMediaType.AVMEDIA_TYPE_VIDEO,
+          size: size,
+          videoInfo: {
+            width: stream.codecpar.width,
+            height: stream.codecpar.height,
+            frameType: (videoHeader & 0x70) >> 4,
+            codecId: videoHeader & 0x0f
+          }
+        }
+
+        // 發送幀資訊事件
+        if (this.onFrameInfo) {
+          this.onFrameInfo(this.context.frameInfo);
+        }
       }
       else {
         stream = formatContext.createStream()
diff --git a/src/avformat/formats/flv/type.ts b/src/avformat/formats/flv/type.ts
index 02a3205..77940a6 100644
--- a/src/avformat/formats/flv/type.ts
+++ b/src/avformat/formats/flv/type.ts
@@ -54,6 +54,32 @@ export interface FlvMetaData {
   width: number
 }
 
+export interface FlvFrameInfo {
+  // 時間戳（毫秒）
+  pts_time: number,
+  // 圖片類型 (I/P/B)
+  pict_type: string,
+  // 是否為關鍵幀
+  isKeyFrame: boolean,
+  // 編碼類型
+  codecType: number,
+  // 幀大小
+  size: number,
+  // 視訊相關
+  videoInfo?: {
+    width: number,
+    height: number,
+    frameType: number,
+    codecId: number
+  },
+  // 音訊相關
+  audioInfo?: {
+    channels: number,
+    sampleRate: number,
+    codecId: number
+  }
+}
+
 export interface FlvContext {
   keyframeFilePositions: number[]
   keyFrameTimes: number[]
@@ -70,4 +96,5 @@ export interface FlvContext {
   frameCount: number
   firstKeyframePositionWrote: boolean
   videoMetadataWrote: boolean
+  frameInfo: FlvFrameInfo
 }
diff --git a/src/avpipeline/VideoRenderPipeline.ts b/src/avpipeline/VideoRenderPipeline.ts
index 896475f..428ef5d 100644
--- a/src/avpipeline/VideoRenderPipeline.ts
+++ b/src/avpipeline/VideoRenderPipeline.ts
@@ -26,7 +26,7 @@
 import Pipeline, { TaskOptions } from './Pipeline'
 import * as errorType from 'avutil/error'
 import IPCPort from 'common/network/IPCPort'
-import AVFrame, { AVFrameRef } from 'avutil/struct/avframe'
+import AVFrame, { AVPictureType } from '../avutil/struct/avframe'
 import * as is from 'common/util/is'
 import * as array from 'common/util/array'
 import List from 'cheap/std/collection/List'
@@ -60,6 +60,7 @@ import nextTick from 'common/function/nextTick'
 import isPointer from 'cheap/std/function/isPointer'
 import os from 'common/util/os'
 import MasterTimer from 'common/timer/MasterTimer'
+import generateUUID from 'common/function/generateUUID'
 
 type WebGPURenderFactory = {
   new(canvas: HTMLCanvasElement | OffscreenCanvas, options: WebGPURenderOptions): WebGPURender,
@@ -166,11 +167,23 @@ export default class VideoRenderPipeline extends Pipeline {
 
   declare tasks: Map<string, SelfTask>
 
-  constructor() {
+  private currentFrameUuid: string = ''
+  private lastRenderTime: number = 0
+  private frameQueue: AVFrame[] = []
+  private webGPURenderFactory: WebGPURenderFactory
+  private webGPURender: any = null
+
+  constructor(private readonly port: MessagePort) {
     super()
+    this.webGPURenderFactory = new WebGPURenderFactory()
   }
 
   private async createTask(options: VideoRenderTaskOptions): Promise<number> {
+    console.log("Creating video render task:", {
+      taskId: options.taskId,
+      timestamp: Date.now()
+    });
+
     const leftIPCPort = new IPCPort(options.leftPort)
     const controlIPCPort = new IPCPort(options.controlPort)
 
@@ -242,10 +255,40 @@ export default class VideoRenderPipeline extends Pipeline {
     })
 
     this.tasks.set(options.taskId, task)
-    return 0
+
+    try {
+      task.render = await this.createRender(task, options.canvas)
+      console.log("Render created for task:", {
+        taskId: options.taskId,
+        timestamp: Date.now()
+      });
+
+      task.loop = new LoopTask(() => {
+        console.log("Task loop callback triggered:", {
+          taskId: options.taskId,
+          timestamp: Date.now()
+        });
+        // ... existing loop code ...
+      })
+
+      return errorType.NONE
+    }
+    catch (error) {
+      console.error("Error creating task:", {
+        taskId: options.taskId,
+        error,
+        timestamp: Date.now()
+      });
+      this.tasks.delete(options.taskId)
+      return errorType.UNKNOWN
+    }
   }
 
   private swap(task: SelfTask) {
+    console.log("Swap called", {
+      hasFrontFrame: !!task.frontFrame,
+      hasBackFrame: !!task.backFrame
+    });
     if (task.seeking) {
       return
     }
@@ -504,6 +547,13 @@ export default class VideoRenderPipeline extends Pipeline {
       const interval = (inWorker && support.shareArrayBuffer) ? 0 : 10
 
       task.loop = new LoopTask(() => {
+        console.log("Task loop iteration start:", {
+          taskId: task.taskId,
+          hasBackFrame: !!task.backFrame,
+          currentPTS: task.currentPTS,
+          timestamp: Date.now()
+        });
+
         if (!task.backFrame) {
           if (!task.ended) {
             task.loop.emptyTask()
@@ -521,6 +571,11 @@ export default class VideoRenderPipeline extends Pipeline {
         }
 
         if (!task.isSupport(task.backFrame)) {
+          console.log("Frame format not supported:", {
+            taskId: task.taskId,
+            hasRender: !!task.render,
+            timestamp: Date.now()
+          });
           if (task.render) {
             task.renderRedyed = false
             task.renderFrame = task.backFrame
@@ -544,6 +599,14 @@ export default class VideoRenderPipeline extends Pipeline {
             AV_MILLI_TIME_BASE_Q
           )
 
+        console.log("Frame PTS calculated:", {
+          pts,
+          currentPTS: task.currentPTS,
+          diff: pts - task.currentPTS,
+          taskId: task.taskId,
+          timestamp: Date.now()
+        });
+
         if (pts < task.currentPTS) {
           // 差值大于 5s 认为从头开始了
           if (task.currentPTS - pts > 5000n) {
@@ -606,24 +669,81 @@ export default class VideoRenderPipeline extends Pipeline {
 
         const diff = pts - task.masterTimer.getMasterTime()
 
-        if (diff <= 0) {
-          // 太晚的帧跳过渲染
-          if (task.renderRedyed
-            && !task.skipRender
-            && (inWorker || (-diff < 100n) || (task.renderFrameCount & 0x01n))
-          ) {
-            task.render.render(task.backFrame)
-            task.stats.videoFrameRenderCount++
-            if (task.lastRenderTimestamp) {
-              task.stats.videoFrameRenderIntervalMax = Math.max(
-                getTimestamp() - task.lastRenderTimestamp,
-                task.stats.videoFrameRenderIntervalMax
-              )
+        // 在條件判斷之前記錄影格資訊
+        const frameType = isPointer(task.backFrame) 
+          ? {
+              type: task.backFrame.pictType,
+              typeText: AVPictureType[task.backFrame.pictType] || 'UNKNOWN',
+              isKeyFrame: (task.backFrame.flags & AVFrameFlags.AV_FRAME_FLAG_KEY) !== 0
             }
-            task.lastRenderTimestamp = getTimestamp()
-          }
-          else {
+          : {
+              type: AVPictureType.AV_PICTURE_TYPE_NONE,
+              typeText: 'NONE',
+              isKeyFrame: false
+            }
+
+        console.log('Frame check:', {
+          diff,
+          renderRedyed: task.renderRedyed,
+          skipRender: task.skipRender,
+          frameType
+        })
+
+        if (diff <= 0) {
+          if (task.renderRedyed && !task.skipRender) {
+            const now = performance.now();
+            
+            console.log("Before rendering frame:", {
+              pts,
+              taskId: task.taskId,
+              timestamp: Date.now()
+            });
+
+            task.render.render(task.backFrame);
+            
+            console.log("After rendering frame:", {
+              pts,
+              renderTime: performance.now() - now,
+              taskId: task.taskId,
+              timestamp: Date.now()
+            });
+
+            // 在發送 frame_render 事件前記錄
+            console.log("Before sending frame_render event:", {
+              pts,
+              taskId: task.taskId,
+              hasFrameInfo: true,
+              timestamp: Date.now()
+            });
+
+            // 建立 frameInfo 物件
+            const frameInfo = {
+              pts: isPointer(task.backFrame) 
+                ? task.backFrame.pts 
+                : static_cast<int64>(task.backFrame.timestamp as uint32),
+              renderTime: performance.now() - now,
+              interval: task.lastRenderTimestamp ? now - task.lastRenderTimestamp : 0
+            };
+
+            task.controlIPCPort.notify('frame_render', {
+              pts: frameInfo.pts,
+              frameInfo
+            });
+
+            console.log("After sending frame_render event:", {
+              pts,
+              taskId: task.taskId,
+              frameInfo,
+              timestamp: Date.now()
+            });
+
+            task.lastRenderTimestamp = now;
+          } else {
             task.stats.videoFrameDropCount++
+            console.log('Frame dropped:', {
+              frameCount: task.stats.videoFrameDropCount,
+              reason: task.renderRedyed ? 'skip render' : 'not ready'
+            })
           }
           task.stats.videoCurrentTime = pts
           task.renderFrameCount++
@@ -1070,8 +1190,13 @@ export default class VideoRenderPipeline extends Pipeline {
   }
 
   public async renderNextFrame(taskId: string) {
+    console.log("renderNextFrame called", { taskId });
     const task = this.tasks.get(taskId)
     if (task) {
+      console.log("Task found for renderNextFrame", {
+        hasBackFrame: !!task.backFrame,
+        taskId
+      });
       if (task.backFrame) {
         const pts = isPointer(task.backFrame)
           ? avRescaleQ2(
@@ -1084,7 +1209,13 @@ export default class VideoRenderPipeline extends Pipeline {
             AV_TIME_BASE_Q,
             AV_MILLI_TIME_BASE_Q
           )
+        console.log("Before render call", {
+          pts,
+          isPointer: isPointer(task.backFrame),
+          taskId
+        });
         task.render.render(task.backFrame)
+        console.log("After render call", { taskId });
         task.stats.videoCurrentTime = pts
         task.stats.videoFrameRenderCount++
         task.currentPTS = pts
@@ -1095,6 +1226,11 @@ export default class VideoRenderPipeline extends Pipeline {
   }
 
   public async registerTask(options: VideoRenderTaskOptions): Promise<number> {
+    console.log("Registering video render task:", { 
+      taskId: options.taskId,
+      timestamp: Date.now() 
+    });
+    
     if (this.tasks.has(options.taskId)) {
       return errorType.INVALID_OPERATE
     }
@@ -1102,6 +1238,10 @@ export default class VideoRenderPipeline extends Pipeline {
   }
 
   public async unregisterTask(id: string): Promise<void> {
+    console.log("Unregistering task:", {
+      taskId: id,
+      timestamp: Date.now()
+    });
     const task = this.tasks.get(id)
     if (task) {
       const started = !!task.loop
@@ -1143,4 +1283,44 @@ export default class VideoRenderPipeline extends Pipeline {
       this.tasks.delete(id)
     }
   }
+
+  private handleFrame(frame: AVFrame) {
+    this.currentFrameUuid = generateUUID()
+  }
+
+  public getCurrentFrame() {
+    return {
+      uuid: this.currentFrameUuid
+    }
+  }
+
+  protected async render(frame: AVFrame) {
+    const now = performance.now();
+    console.log("Video frame rendering start:", {
+      pts: frame.pts,
+      pictType: frame.pictType === AVPictureType.AV_PICTURE_TYPE_I ? 'I' : 
+                frame.pictType === AVPictureType.AV_PICTURE_TYPE_P ? 'P' : 
+                frame.pictType === AVPictureType.AV_PICTURE_TYPE_B ? 'B' : '?',
+      timestamp: now
+    });
+
+    const frameInfo = {
+      pts_time: (Number(frame.pts) / 1000000).toFixed(6),
+      pict_type: frame.pictType === AVPictureType.AV_PICTURE_TYPE_I ? 'I' : 
+                 frame.pictType === AVPictureType.AV_PICTURE_TYPE_P ? 'P' : 
+                 frame.pictType === AVPictureType.AV_PICTURE_TYPE_B ? 'B' : '?'
+    };
+
+    // 發送事件
+    this.port.postMessage({
+      type: 'notify',
+      data: {
+        method: 'frame_render',
+        params: {
+          pts: frame.pts,
+          frameInfo
+        }
+      }
+    });
+  }
 }
diff --git a/src/avplayer/AVPlayer.ts b/src/avplayer/AVPlayer.ts
index eeeaf06..2d18063 100644
--- a/src/avplayer/AVPlayer.ts
+++ b/src/avplayer/AVPlayer.ts
@@ -80,7 +80,7 @@ import isHdr from 'avutil/function/isHdr'
 import hasAlphaChannel from 'avutil/function/hasAlphaChannel'
 import SubtitleRender from './subtitle/SubtitleRender'
 import { Data, Fn } from 'common/types/type'
-import { playerEventChanged, playerEventChanging, playerEventError, playerEventNoParam, playerEventProgress, playerEventSubtitleDelayChange, playerEventTime, playerEventVolumeChange } from './type'
+import { playerEventChanged, playerEventChanging, playerEventError, playerEventNoParam, playerEventProgress, playerEventSEI, playerEventSubtitleDelayChange, playerEventTime, playerEventTTFF, playerEventVolumeChange } from './type'
 import compileResource from 'avutil/function/compileResource'
 import os from 'common/util/os'
 import IPCPort, { REQUEST, RpcMessage } from 'common/network/IPCPort'
@@ -295,7 +295,10 @@ export const AVPlayerSupportedCodecs = [
   AVCodecID.AV_CODEC_ID_TEXT
 ]
 
+// Add default options constant
 const defaultAVPlayerOptions: Partial<AVPlayerOptions> = {
+  container: null,
+  wasmBaseUrl: '',
   enableHardware: true,
   enableWebGPU: true,
   enableWorker: true,
@@ -304,7 +307,8 @@ const defaultAVPlayerOptions: Partial<AVPlayerOptions> = {
   jitterBufferMax: 4,
   jitterBufferMin: 1,
   lowLatency: false,
-  preLoadTime: 4
+  preLoadTime: 4,
+  isLive: true // 新增這行，設定預設值為 true
 }
 
 export const enum AVPlayerStatus {
@@ -459,7 +463,7 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
   private lastStatus: AVPlayerStatus
   private playChannels: number
   private seekedTimestamp: int64
-  private isLive_: boolean
+  private isLive_: boolean = true; // 設定類別屬性的預設值為 true
 
   private statsController: StatsController
   private jitterBufferController: JitterBufferController
@@ -473,12 +477,21 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
 
   private externalSubtitleTasks: ExternalSubtitleTask[]
 
+  // SEI message channel
+  private sei2PlayerChannel: MessageChannel
+  // TTFF timestamps
+  private loadStart: number = Date.now()
+
   constructor(options: AVPlayerOptions) {
     super(true)
+    // Merge default options with provided options
     this.options = object.extend({}, defaultAVPlayerOptions, options)
     this.taskId = generateUUID()
     this.status = AVPlayerStatus.STOPPED
 
+    // Initialize isLive_ from options, 如果沒有指定則使用預設值 true
+    this.isLive_ = this.options.isLive ?? true
+    
     this.volume = 1
     this.playRate = 1
     this.renderMode = RenderMode.FIT
@@ -486,7 +499,6 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
     this.flipHorizontal = false
     this.flipVertical = false
     this.seekedTimestamp = NOPTS_VALUE_BIGINT
-    this.isLive_ = !!options.isLive
 
     this.GlobalData = make<AVPlayerGlobalData>()
 
@@ -506,6 +518,11 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
     logger.info(`create player, taskId: ${this.taskId}`)
   }
 
+  // Add public getter for isLive, for debugging
+  get isLive(): boolean {
+    return this.isLive_
+  }
+
   /**
    * 当前播放时间戳（毫秒）
    */
@@ -1107,6 +1124,9 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
 
     logger.info(`call load, taskId: ${this.taskId}`)
 
+    // For TTFF calculation
+    this.loadStart = Date.now()
+
     this.status = AVPlayerStatus.LOADING
     this.fire(eventType.LOADING)
     if (is.boolean(options.isLive)) {
@@ -1389,6 +1409,22 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
       }
     }
 
+
+    // SEI message channel setup
+    this.sei2PlayerChannel = createMessageChannel(this.options.enableWorker)
+    await AVPlayer.DemuxerThread.setSEIPort
+      .transfer(this.sei2PlayerChannel.port1)
+      .invoke(this.sei2PlayerChannel.port1)
+
+    // Listen for SEI events on the other port
+    const seiPort = new IPCPort(this.sei2PlayerChannel.port2)
+    seiPort.on(REQUEST, (request: RpcMessage) => {
+      const { method, params: { value } } = request
+      this.fire(method, value)
+      seiPort.reply(request, true)
+    })
+
+
     this.fire(eventType.PROGRESS, [AVPlayerProgress.OPEN_FILE])
 
     ret = await AVPlayer.DemuxerThread.openStream(this.taskId)
@@ -3339,6 +3375,9 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
   public onFirstVideoRendered(): void {
     logger.info(`first video frame rendered, taskId: ${this.taskId}`)
     this.fire(eventType.FIRST_VIDEO_RENDERED)
+    const loadEnd = Date.now()
+    const ttff = loadEnd - this.loadStart
+    this.fire(eventType.TTFF, [ttff])
   }
 
   /**
@@ -3724,6 +3763,14 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
   public on(event: typeof eventType.VOLUME_CHANGE, listener: typeof playerEventVolumeChange, options?: Partial<EmitterOptions>): AVPlayer
   public on(event: typeof eventType.SUBTITLE_DELAY_CHANGE, listener: typeof playerEventSubtitleDelayChange, options?: Partial<EmitterOptions>): AVPlayer
 
+  // SEI event
+  public on(event: typeof eventType.SEI, listener: typeof playerEventSEI, options?: Partial<EmitterOptions>): AVPlayer
+  // TTFF event
+  public on(event: typeof eventType.TTFF, listener: typeof playerEventTTFF, options?: Partial<EmitterOptions>): AVPlayer
+
+  // Add frame render event
+  public on(event: typeof eventType.FRAME_RENDER, listener: typeof playerEventTime, options?: Partial<EmitterOptions>): AVPlayer
+
   public on(event: string, listener: Fn, options?: Partial<EmitterOptions>): AVPlayer
   public on(event: string, listener: Fn, options: Partial<EmitterOptions> = {}) {
     super.on(event, object.extend({
@@ -3738,4 +3785,56 @@ export default class AVPlayer extends Emitter implements ControllerObserver {
     }, options))
     return this
   }
+
+  public onFrameRender(pts: int64, frameInfo: any): void {
+    // 加入詳細的除錯資訊
+    console.log('AVPlayer.onFrameRender called:', {
+      eventType: eventType.FRAME_RENDER,
+      pts,
+      frameInfo,
+      hasListeners: this.hasListener(eventType.FRAME_RENDER),
+      timestamp: Date.now()
+    });
+
+    // 確保 frameInfo 存在且為物件
+    if (!frameInfo || typeof frameInfo !== 'object') {
+      console.warn('AVPlayer: Invalid frameInfo:', frameInfo);
+      return;
+    }
+
+    // 加入音訊時間戳同步
+    if (this.selectedAudioStream && frameInfo.audioInfo) {
+      console.log('AVPlayer: Adding audio stream info to frameInfo');
+      frameInfo.audioInfo.streamId = this.selectedAudioStream.id;
+      frameInfo.audioInfo.codecType = this.selectedAudioStream.codecpar.codecType;
+    }
+
+    // 加入影片時間戳同步
+    if (this.selectedVideoStream && frameInfo.videoInfo) {
+      console.log('AVPlayer: Adding video stream info to frameInfo');
+      frameInfo.videoInfo.streamId = this.selectedVideoStream.id;
+      frameInfo.videoInfo.codecType = this.selectedVideoStream.codecpar.codecType;
+    }
+
+    console.log('AVPlayer: Firing frame render event:', {
+      pts,
+      frameInfo,
+      timestamp: Date.now()
+    });
+
+    // 使用正確的事件類型常數
+    this.fire(eventType.FRAME_RENDER, [pts, frameInfo]);
+  }
+
+  // 在 fire 方法中也加入 log
+  public fire(event: string, params?: any[]): void {
+    console.log('AVPlayer.fire called:', {
+      event,
+      params,
+      hasListeners: this.hasListener(event),
+      timestamp: Date.now()
+    });
+
+    super.fire(event, params);
+  }
 }
diff --git a/src/avplayer/Controller.ts b/src/avplayer/Controller.ts
index 3a0f38c..33319d5 100644
--- a/src/avplayer/Controller.ts
+++ b/src/avplayer/Controller.ts
@@ -38,6 +38,7 @@ export interface ControllerObserver {
   onMSESeek: (time: number) => void
   onGetDecoderResource: (mediaType: AVMediaType, codecId: AVCodecID) => Promise<WebAssemblyResource | string | ArrayBuffer>
   isPictureInPicture: () => boolean
+  onFrameRender?: (pts: int64, frameInfo: any) => void
 }
 
 export default class Controller {
@@ -59,41 +60,91 @@ export default class Controller {
   private enableAudioVideoSync: boolean
 
   constructor(observer: ControllerObserver, enableWorker: boolean) {
+    console.log('Controller constructor called:', {
+      enableWorker,
+      timestamp: Date.now()
+    });
+
     this.observer = observer
     this.videoRenderControlChannel = createMessageChannel(enableWorker)
     this.audioRenderControlChannel = createMessageChannel(enableWorker)
     this.muxerControlChannel = createMessageChannel(enableWorker)
     this.demuxerControlChannel = createMessageChannel(enableWorker)
 
+    console.log('Message channels created');
+
     this.videoRenderControlIPCPort = new IPCPort(this.videoRenderControlChannel.port2)
     this.audioRenderControlIPCPort = new IPCPort(this.audioRenderControlChannel.port2)
     this.muxerControlIPCPort = new IPCPort(this.muxerControlChannel.port2)
     this.demuxerControlIPCPort = new IPCPort(this.demuxerControlChannel.port2)
+
+    console.log('IPC ports initialized');
+
     this.enableAudioVideoSync = true
 
+    console.log('Setting up video render control event listener');
+
     this.videoRenderControlIPCPort.on(NOTIFY, (request: RpcMessage) => {
+      console.log('VideoRenderControl notification listener setup complete');
+      console.log('VideoRenderControl received notification:', {
+        method: request.method,
+        params: request.params,
+        timestamp: Date.now()
+      });
+
       switch (request.method) {
         case 'ended':
+          console.log('Video ended event received');
           this.observer.onVideoEnded()
           break
         case 'updateCanvas':
+          console.log('Canvas update event received');
           this.observer.onCanvasUpdated()
           break
         case 'firstRendered':
+          console.log('First video frame rendered');
           this.observer.onFirstVideoRendered()
           break
         case 'firstRenderedAfterUpdateCanvas':
+          console.log('First video frame rendered after canvas update');
           this.observer.onFirstVideoRenderedAfterUpdateCanvas()
           break
+        case 'frame_render':
+          console.log('Frame render event received:', {
+            params: request.params,
+            timeUpdateType: this.timeUpdateListenType,
+            timestamp: Date.now()
+          });
+          
+          if (this.timeUpdateListenType === AVMediaType.AVMEDIA_TYPE_VIDEO) {
+            const { pts, frameInfo } = request.params;
+            console.log('Processing frame render:', {
+              pts,
+              frameInfo,
+              timestamp: Date.now()
+            });
+            this.observer.onFrameRender?.(pts, frameInfo);
+          }
+          break
         case 'syncPts':
+          console.log('Sync PTS event received:', {
+            params: request.params,
+            timeUpdateType: this.timeUpdateListenType
+          });
           if (this.timeUpdateListenType === AVMediaType.AVMEDIA_TYPE_VIDEO) {
-            this.observer.onTimeUpdate(request.params.pts)
+            this.observer.onTimeUpdate(request.params.pts);
           }
           break
       }
     })
 
+    console.log('Setting up audio render control event listener');
+    
     this.audioRenderControlIPCPort.on(NOTIFY, (request: RpcMessage) => {
+      console.log('Audio render control received notification:', {
+        method: request.method,
+        timestamp: Date.now()
+      });
       switch (request.method) {
         case 'syncPts':
           if (this.enableAudioVideoSync) {
@@ -137,9 +188,12 @@ export default class Controller {
     }
     this.visibilityHidden = document.visibilityState === 'hidden' && !this.observer.isPictureInPicture()
     document.addEventListener('visibilitychange', this.onVisibilityChange)
+
+    console.log('Controller constructor completed');
   }
 
   public getVideoRenderControlPort() {
+    console.log('Getting video render control port');
     return this.videoRenderControlChannel.port1
   }
 
@@ -164,7 +218,9 @@ export default class Controller {
   }
 
   public destroy() {
+    console.log('Controller destroy called');
     if (this.videoRenderControlIPCPort) {
+      console.log('Destroying video render control IPC port');
       this.videoRenderControlIPCPort.destroy()
     }
     if (this.audioRenderControlIPCPort) {
diff --git a/src/avplayer/eventType.ts b/src/avplayer/eventType.ts
index 53ec054..8dc6d0f 100644
--- a/src/avplayer/eventType.ts
+++ b/src/avplayer/eventType.ts
@@ -66,3 +66,9 @@ export const PROGRESS = 'progress'
 export const VOLUME_CHANGE = 'volumeChange'
 
 export const SUBTITLE_DELAY_CHANGE = 'subtitleDelayChange'
+
+export const SEI = 'SEI'
+
+export const TTFF = 'TTFF'
+
+export const FRAME_RENDER = 'frame_render'
diff --git a/src/avutil/util/avframe.ts b/src/avutil/util/avframe.ts
index f55371e..5812927 100644
--- a/src/avutil/util/avframe.ts
+++ b/src/avutil/util/avframe.ts
@@ -45,7 +45,7 @@ export enum AV_FRAME_SIDE_DATA_FLAG {
 }
 
 export function createAVFrame(): pointer<AVFrame> {
-  const frame = reinterpret_cast<pointer<AVFrame>>(avMallocz(sizeof(AVFrame)))
+  const frame: pointer<AVFrame> = avMallocz(sizeof(AVFrame))
   getAVFrameDefault(frame)
   return frame
 }
@@ -375,7 +375,7 @@ export function refAVFrame(dst: pointer<AVFrame>, src: pointer<AVFrame>) {
   }
 
   if (src.extendedBuf) {
-    dst.extendedBuf = reinterpret_cast<pointer<pointer<AVBufferRef>>>(avMallocz(reinterpret_cast<int32>(sizeof(accessof(dst.extendedBuf))) * src.nbExtendedBuf))
+    dst.extendedBuf = reinterpret_cast<pointer<pointer<void>>>(avMallocz(reinterpret_cast<int32>(sizeof(accessof(dst.extendedBuf))) * src.nbExtendedBuf))
     if (!dst.extendedBuf) {
       unrefAVFrame(dst)
       return NO_MEMORY
diff --git a/test-react/src/App.tsx b/test-react/src/App.tsx
new file mode 100644
index 0000000..e8f6d4f
--- /dev/null
+++ b/test-react/src/App.tsx
@@ -0,0 +1,170 @@
+// test-react/src/App.tsx
+import { useEffect, useState } from "react";
+import "./App.css";
+
+import AVPlayer from "../../dist/avplayer";
+import { eventType } from "../../dist/avplayer/eventType";
+
+function FrameInfoDisplay({ frameInfo }) {
+  if (!frameInfo) return null;
+  
+  // 完全模仿 ffprobe 的輸出格式：pts_time,pict_type
+  return (
+    <div className="frame-info" style={{ fontFamily: 'monospace', whiteSpace: 'pre' }}>
+      {`${frameInfo.pts_time},${frameInfo.pict_type}`}
+    </div>
+  );
+}
+
+function App() {
+  const [currentFrameInfo, setCurrentFrameInfo] = useState(null);
+  const playerId = "stream-player";
+
+  useEffect(() => {
+    const loadPlayerAsync = async () => {
+      try {
+        const container = document.getElementById(playerId) as HTMLDivElement;
+
+        const { origin, pathname } = window.location;
+        const baseUrl = `${origin}${pathname}`;
+        const wasmBaseUrl = `${baseUrl}libmedia/wasm`;
+
+        const player = new AVPlayer({ container, wasmBaseUrl });
+        
+        // Add player to window object for console access, for debugging
+        (window as any).player = player;
+        console.log('Player is now accessible via window.player in console');
+
+        player.on("SEI", (sei) => {
+          console.log("React - SEI:", sei);
+        });
+
+        player.on("TTFF", (ttff) => {
+          console.log("React - Time to first frame (seconds):", ttff / 1000);
+        });
+
+        // const source =
+        // "https://pull-tc.stream.iki-utl.cc/live/r1234_dice_tc.flv";
+        // const source = "http://192.168.20.10:8080/live/r1234_dice.flv";
+        // const source = "http://192.168.20.10:8080/live/testrc.flv"
+        // const source = "http://192.168.20.10:8080/live/r1234_dice.flv"
+        // const source = "http://pull-tc.stream.iki-utl.cc/live/sdp001_roulette_hq.flv"
+        // const source = "http://pull-tc.stream.iki-utl.cc/live/sdp002_dice_hq.flv"
+        // const source = "http://192.168.20.10:8080/live/r1234_dice.flv"
+        const source = "http://192.168.20.10:8080/live/testrc.flv"
+        // const source = "https://video-tx-int.lv-play.com/live/4825298Y.flv"
+        // const source = "https://video-ws-aws.lv-play.com/live/1218368Y.flv"
+        // const source = "https://localhost:8090/stream.flv"
+        // const loadStart = Date.now();
+
+        await player.load(source);
+        await player.play();
+
+        player.on("time", (ms) => {
+          // current time in seconds
+          const currentTime = Number(ms) / 1000;
+
+          // console.log("Current time:", currentTime);
+          // console.log(player.getOptions());
+
+          const container = player.getOptions().container;
+          if (!container) return;
+
+          const shouldZoomIn = currentTime >= 3 && currentTime < 6;
+          if (shouldZoomIn) {
+            container.classList.add("zoomed");
+          } else {
+            container.classList.remove("zoomed");
+          }
+        });
+
+        // Add frame render event listener with frameInfo
+        player.on('frame_render', (pts, frameInfo) => {
+          // 加入除錯日誌
+          console.log('React component received frame_render event:', {
+            pts,
+            frameInfo,
+            timestamp: Date.now()
+          });
+
+          // 確保 frameInfo 存在且有效
+          if (!frameInfo) {
+            console.warn('Received invalid frameInfo:', frameInfo);
+            return;
+          }
+
+          // Convert pts to seconds with 6 decimal places (ffprobe format)
+          const pts_time = (Number(pts) / 1000000).toFixed(6);
+          
+          // 取得圖片類型 (I/P/B)
+          const pict_type = frameInfo.isKeyFrame ? 'I' : 'P';  // 假設只有 I 和 P 幀
+
+          // 建立完全符合 ffprobe 格式的幀資訊
+          const probeFrameInfo = {
+            pts_time,
+            pict_type
+          };
+
+          // 輸出完全符合 ffprobe 的格式
+          console.log(`Frame info: ${pts_time},${pict_type}`);
+
+          // 更新 UI
+          updateFrameInfo(probeFrameInfo);
+        });
+
+        // 加入更多除錯事件監聽
+        player.on('error', (error) => {
+          console.error('Player error:', error);
+        });
+
+        // 監聽播放狀態變化
+        player.on('playing', () => {
+          console.log('Player started playing');
+        });
+
+        console.log("Player loaded:", player);
+      } catch (error) {
+        console.error("Failed to load player:", error);
+      }
+    };
+
+    loadPlayerAsync();
+  }, []);
+
+  const playerStyle = { width: "50vw", height: "calc(50vw * 9 / 16)" };
+
+  // 更新幀資訊的函數
+  const updateFrameInfo = (frameInfo) => {
+    setCurrentFrameInfo(frameInfo);
+  };
+
+  return (
+    <>
+      <div style={{ ...playerStyle, overflow: "hidden" }}>
+        <div id={playerId} style={playerStyle}></div>
+      </div>
+      <h1>Studio Player Test</h1>
+      {/* 可選：顯示幀資訊 */}
+      <FrameInfoDisplay frameInfo={currentFrameInfo} />
+    </>
+  );
+}
+
+// 加入一些基本的 CSS 樣式
+const styles = `
+.frame-info {
+  position: absolute;
+  top: 10px;
+  left: 10px;
+  background: rgba(0, 0, 0, 0.7);
+  color: white;
+  padding: 5px 10px;
+  border-radius: 4px;
+  font-family: monospace;
+  font-size: 14px;
+  z-index: 1000;
+  min-width: 150px;
+}
+`;
+
+export default App;

[ikigai-kevin-k]

Currently, the workable solution for I-frame corresponds to these snippets:
AVPlayer.ts:

public onFrameRender(pts: int64, frameInfo: any): void {
    // 加入詳細的除錯資訊
    console.log('AVPlayer.onFrameRender called:', {
      eventType: eventType.FRAME_RENDER,
      pts,
      frameInfo,
      hasListeners: this.hasListener(eventType.FRAME_RENDER),
      timestamp: Date.now()
    });

    // 確保 frameInfo 存在且為物件
    if (!frameInfo || typeof frameInfo !== 'object') {
      console.warn('AVPlayer: Invalid frameInfo:', frameInfo);
      return;
    }
...
    console.log('AVPlayer: Firing frame render event:', {
      pts,
      frameInfo,
      timestamp: Date.now()
    });

    this.fire(eventType.FRAME_RENDER, [pts, frameInfo]);
  }

eventType.ts:

export const FRAME_RENDER = 'frame_render'

Controller.ts:

     switch (request.method) {
        case 'ended':
...
        case 'updateCanvas':
...
        case 'firstRendered':
...
        case 'frame_render':
          console.log('Frame render event received:', {
            params: request.params,
            timeUpdateType: this.timeUpdateListenType,
            timestamp: Date.now()
          });
          
          if (this.timeUpdateListenType === AVMediaType.AVMEDIA_TYPE_VIDEO) {
            const { pts, frameInfo } = request.params;
            console.log('Processing frame render:', {
              pts,
              frameInfo,
              timestamp: Date.now()
            });
            this.observer.onFrameRender?.(pts, frameInfo);
          }
          break
        case 'syncPts':
...

Hope these snippets will be more comprehensible~

[zhaohappy]

I followed the code above and got the pts of all frames in the console

There is no need to judge the condition this.timeUpdateListenType === AVMediaType.AVMEDIA_TYPE_VIDEO in Controller.ts, otherwise it will only be executed when no audio stream in the flv

switch (request.method) {
        case 'ended':
...
        case 'updateCanvas':
...
        case 'firstRendered':
...
        case 'frame_render':
          console.log('Frame render event received:', {
            params: request.params,
            timeUpdateType: this.timeUpdateListenType,
            timestamp: Date.now()
          });
          
          //  if (this.timeUpdateListenType === AVMediaType.AVMEDIA_TYPE_VIDEO) {
            const { pts, frameInfo } = request.params;
            console.log('Processing frame render:', {
              pts,
              frameInfo,
              timestamp: Date.now()
            });
            this.observer.onFrameRender?.(pts, frameInfo);
          // }
          break
        case 'syncPts':
...

[Kuo-TingKai]

Wierd...
What is your test flv live stream url?
Could you provide it for me thx

[zhaohappy]

I tested it using local flv file. Here is my patch:
0001-test-test-frame_render-event.patch

[ikigai-kevin-k]

Since I am applying the player in a live streaming scenario, could you please use live stream instead of local file? Thank you very much.

[zhaohappy]

I think there is no difference between live streaming and local file in this scenario. I test with live streaming also got the pts of all frames in the console. Have you tested the patch I gave above? Is the video rendering normal? If not, It may be a problem with your live streaming.

[Kuo-TingKai]

The problem solved, thanks a lot!