A subjective and objective video quality dataset for neural and traditional video codecs

This repository contains all the data related to the paper: "Evaluating Video Quality Metrics for Neural and Traditional Codecs using 4K/UHD-1 Videos"

[Dataset] [arXiv Preprint]

Dataset

The subjective and metric results are organized in the following way:

• The ./metrics/ directory contains all raw outputs from the evaluated quality metrics, stored as .json files.
• The file subjective.csv holds the subjective scores collected from the user study.
• An aggregated version of all results is available in results.json for easier access and analysis.

The corresponding video files (Sources: ~13GB / PVS: ~74GB) can be downloaded using the link AVT-VQDB-UHD-1-NVC or by using the included script:

chmod +x ./download.sh
./download.sh

The videos are provided as HEVC lossless transcodes for the encoded and source video files. To reproduce the results from the paper the videos need to be upscaled to 3840x2160 using ffmpeg with a lanczos filter and a lossless codec (eg. ffvhuff, ffv1 or lossless HEVC). Make sure ffmpeg is installed on your system.

ffmpeg -i INPUT.mkv -c:v ffvhuff -vf "scale=3840x2160:param0=5" -sws_flags lanczos+accurate_rnd+bitexact OUTPUT.mkv
ffmpeg -i INPUT.mkv -c:v ffv1 -level 3 slicecrc 1 -vf "scale=3840x2160:param0=5" -sws_flags lanczos+accurate_rnd+bitexact OUTPUT.mkv
ffmpeg -i INPUT.mkv -c:v libx265 -x265-params lossless=1 -vf "scale=3840x2160:param0=5" -sws_flags lanczos+accurate_rnd+bitexact OUTPUT.mkv

The included helper script prepare_videos.py can also be used to upscale all videos at once.

python prepare_videos.py --input_dir decoded  -output_dir pvs

To save space (at the cost of longer encoding times), use the --codec/-c ffv1 or h265 option:

python prepare_videos.py --input_dir decoded  -output_dir pvs --codec ffv1

Citation

Please cite the following paper if you use any part of the data or code provided in this repository.

@inproceedings{herb_evaluating_2025,
    title     = {Evaluating Video Quality Metrics for Neural and Traditional Codecs using 4K/UHD-1 Videos},
    author    = {Herb, Benjamin and Ramachandra Rao, Rakesh Rao and Göring, Steve and Raake, Alexander},
    booktitle = {Picture Coding Symposium (PCS)}, 
    year      = {2025},
    address   = {Aachen, Germany},
    note      = {To appear}
}