Skip to content

Bug: HLS stream URLs get truncated in ffmpeg #1548

New issue
New issue
Open
Open
Bug: HLS stream URLs get truncated in ffmpeg#1548
Labels
ffmpeg-produceros/windowstype/bug

Description

@leftearsons
leftearsons
opened on Jun 25, 2024

Observed Behavior

It is observed that in some cases, the internal ffmpeg library, truncates the HLS URL which causes valuable authentication data to be discarded. This leads to invalid video stream urls and eventually to stopping HLS playback.
By looking at the logs, the issue seems to take place inside the 'https' thread, since the 'hls' thread always logs the correct (full length) URL path.

The pattern of the failing URLs makes me think that the root cause of this case, could be the query parameter length of the HLS URL, since it is only observed when the URL contains the authentication data in query parameters, instead of inside the URL path.
We have been loading HLS video streams from the same video providers for a long time and no change has been made on the video source, meaning that the same HLS URLs format used to work well with CCG v2.07.

I am adding for your reference the log files for those two cases, a faulty one (in which the URL contains path parameters) and a successful one (in which the URL contains only path parameters).

Faulty case:

[info]    Received message from 127.0.0.1: PLAY 1-50 "https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k.m3u8?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad77c74e068ac5e898cdf8e7a270c3ffc30f62ba160c5b8efa5856c9"\r\n
[info]    Sent message to 127.0.0.1:202 PLAY OK\r\n
[info]    [ffmpeg] [hls @ 0000022E3604E300] Skip ('#EXT-X-VERSION:3')
[info]    
[info]    [ffmpeg] [hls @ 0000022E3604E300] Opening 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000690.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad77c74e068ac5e898cdf8e7a270c3ffc30f62ba160c5b8efa5856c9' for reading
[info]    
[info]    [ffmpeg] [hls @ 0000022E3604E300] Opening 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000691.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad77c74e068ac5e898cdf8e7a270c3ffc30f62ba160c5b8efa5856c9' for reading
[info]    
[info]    [ffmpeg] [https @ 0000022E328F2040] Opening 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000692.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad' for reading
[info]    
[warning] ffmpeg[https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k.m3u8?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad77c74e068ac5e898cdf8e7a270c3ffc30f62ba160c5b8efa5856c9|0.0000/0.0000] Latency: 41
[warning] [ffmpeg] [https @ 0000022E5DEA41C0] HTTP error 403 Forbidden
[warning] 
[warning] [ffmpeg] [hls @ 0000022E3604E300] keepalive request failed for 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000692.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad' with error: 'Server returned 403 Forbidden (access denied)' when opening url, retrying with new connection
[warning] 
[info]    [ffmpeg] [hls @ 0000022E3604E300] Opening 'https://lc-live-http-ipv4.akamaized.net/30714/4830385/desktop/table_tennis_international_tt_elite_series_retail_4830385_1364k_1718772001_0000692.ts?cid=30714&mid=50853377&ecid=4830385&pid=1&dtid=1&sid=848252974901&gc=pu4&gsd=oA&grm=1&hdnts=ip=85.73.218.100~exp=1718859799~acl=%2F30714%2F4830385%2Fdesktop%2F%2A~hmac=d3766115ad' for reading
[info]    
[warning] [ffmpeg] [https @ 0000022E57BF8D40] HTTP error 403 Forbidden
[warning] 
[warning] [ffmpeg] [hls @ 0000022E3604E300] Failed to open segment 692 of playlist 0

Successful case:

[info]    Received message from 127.0.0.1: PLAY 1-50 "https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/video.m3u8"\r\n
[info]    Sent message to 127.0.0.1:202 PLAY OK\r\n
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-VERSION:3')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-INDEPENDENT-SEGMENTS')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-PROGRAM-DATE-TIME:2024-06-25T10:34:00.000Z')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-PROGRAM-DATE-TIME:2024-06-25T10:34:02.000Z')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Skip ('#EXT-X-PROGRAM-DATE-TIME:2024-06-25T10:34:04.000Z')
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Opening 'https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/4/file2357.ts' for reading
[info]    
[info]    [ffmpeg] [hls @ 000001DB7B8BE0C0] Opening 'https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/4/file2358.ts' for reading
[info]    
[info]    [ffmpeg] [https @ 000001DB7BA9A2C0] Opening 'https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/4/file2359.ts' for reading
[info]    
[warning] ffmpeg[https://744133.akamaized.net/hls/live/716232/ch12/89be75f87f0cbc976470fc2feb7ddd34db9a5390/video-m5_video/hdntl=exp=1719398046~acl=%2f*~id=668b56aecfe14366a26602c686807954~data=hdntl~hmac=be026824432c7fdbeb164ebe0cf92d159a38e2e18191633937b89732a8367be1/video.m3u8|0.0000/0.0000] Latency: 39

Expected behaviour

When loading an HLS stream, we would expect CasparCG to start the playback of it's chunks without altering their URL, eventually causing validation errors on the video streaming server.

Steps to reproduce

  1. Load an HLS video stream with PLAY AMCP command, which uses long list of query parameters with authentication tokens
  2. Observe the CasparCG logs and check for truncated HLS video stream URL when comparing the hls ffmpeg thread and the https ffmpeg thread logs, which would eventually cause HTTP validation errors (403 in our case)

Environment

  • Server version 2.4.0 1e25c7a Stable
  • Operating system: Windows 10

Issue was not present in version 2.07

Activity

  1. leftearsons
    added
    type/bug
    on Jun 25, 2024

  2. tdotmgc commented on Sep 23, 2024

    @tdotmgc
    tdotmgc
    on Sep 23, 2024

    I have the same problem when downloading videos from PornHub. The site generates three types of links to videos. Two types of links are short and one type is long. FFmpeg cuts off part of the address, so the video does not download.

    For example:
    Source URL of the video
    https://cv-h.phncdn.com/hls/videos/202106/29/000000000/1080P_4000K_000000000.mp4/index-v1-a1.m3u8?6MB50lVyuE3XsoDstxOgtlFIS5WiZafP8gykHFLhkeDN6QJHJBdWP584qIVpgkovqyG90vdUcANTJdoyQlEEIfY01AREoa2bE6fFAIgGstWPRzp1XagL4Q1HkrjiEh4ABMzCcuW3ncZCD-Hczh2S2Gc_Jq2YdMpAmbrbbQwUJO0FbaTzlGiD5mri4xWZD0MTNeizdYeJK-w
    Last 16 simbols are cutted by FFmpeg
    https://cv-h.phncdn.com/hls/videos/202106/29/000000000/1080P_4000K_000000000.mp4/index-v1-a1.m3u8?6MB50lVyuE3XsoDstxOgtlFIS5WiZafP8gykHFLhkeDN6QJHJBdWP584qIVpgkovqyG90vdUcANTJdoyQlEEIfY01AREoa2bE6fFAIgGstWPRzp1XagL4Q1HkrjiEh4ABMzCcuW3ncZCD-Hczh2S2Gc_Jq2YdMpAmbrbbQwUJO0FbaTzlGiD5mri4xW

  3. Julusian commented on Sep 27, 2024

    @Julusian
    Julusian
    on Sep 27, 2024
    Member

    I'm not able to reproduce this on linux, even when using much longer urls.

  4. leftearsons commented on Sep 30, 2024

    @leftearsons
    leftearsons
    on Sep 30, 2024
    Author

    @Julusian are there any chances that this only takes place in Windows?
    Just to add here that this seems to be happening only with some specific video providers and not always (8-9 out of 10 times though).
    Any thoughts what could possibly cause such issues?
    Is it possible that this could be related to any 3rd party library installed or even to the specific stream URLs format?
    This issue was first started about a year ago and was not there in the past, even for the same video provider that currently faces the issue.
    I would really appreciate if you could have a second look on this, since this affects our production systems, causing a lot of issues.
    Thank you!

  5. thetcorp commented on Oct 20, 2024

    @thetcorp
    thetcorp
    on Oct 20, 2024

    This bug is Windows-specific.

    It can be reproduced with the following command:

    ffmpeg -y -i "https://cv-h.phncdn.com/hls/videos/202410/07/458776431/1080P_4000K_458776431.mp4/index-v1-a1.m3u8?YZV3BADjVrQI7TsPwCzpCBVtarJIGx4L5kK03HOOK8Z1DxlgTqArmrQatvpRliqGZa9WfJYhLrq_0SyVyQhdW2TvDrPLKAl2_a2WGAz58hH7Z2QwqAA3A_sURHrVh1pjgglylaPCGmDd7ry-frsOzGajIfvh1Z01NQ26JP1C7Ke7pyIz2ra9fvisGdRwGMElQ3uXATBMUw" -c:v copy -bsf:a aac_adtstoasc "test.mp4"

    Result:

    [hls @ 00000230ba7e0640] Opening 'https://cv-h.phncdn.com/hls/videos/202410/07/458776431/1080P_4000K_458776431.mp4/seg-68-v1-a1.ts?YZV3BADjV' for reading

[https @ 00000230bba55bc0] HTTP error 403 Forbidden

[hls @ 00000230ba7e0640] Failed to open segment 68 of playlist 0
  6. Julusian
    added
    os/windows
    ffmpeg-producer
    on Jan 15, 2025

  7. itsjnz007 commented on Oct 1, 2026

    @itsjnz007
    itsjnz007
    on Oct 1, 2026

    I am using a hls proxy and ffmpeg for some reason sends "Range" headers to the proxy. When this "Range" is sent to the upstream server, somehow the truncation of segments happen. It is weird and I cannot explain why, but intercepting the "Range" header for the master.m3u8 url fixed the problem for me.

    is_m3u8 = (
    ".m3u8" in media_url
    or (content_type and "mpegurl" in content_type.lower())
    or (content_type and "application/vnd.apple.mpegurl" in content_type)
)
request_range = request_headers.get("Range") or request_headers.get("range")
for header_name in list(arg_headers):
    if header_name.lower() == "range": del arg_headers[header_name]
if request_range and not is_m3u8: arg_headers["Range"] = request_range
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ffmpeg-produceros/windowstype/bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions