Voice Privacy Evaluation - Extension of VPC 2024

This fork extends the evaluation scripts of the Voice Privacy Challenge 2024 by ASV tests during training of the ASV model.

According to the evaluation setup of the challenge, the ASV model is retrained on anonymized data. The training data that is used for this retraining is supposed to be anonymized in exactly the same way as the evaluation data. This means that a new ASV model needs to be trained for every change in an anonymization system in order to evaluate this system. During this process, errors might happen that lead to a mismatch between training and evaluation data. At the same time, an anonymization system might be constructed in a way that it prevents the ASV model to learn something useful from the training data. As a consequence, the ASV model performs worse on the evaluation data than it should have if being trained correctly, leading to an overestimation of the privacy protection provided by the anonymization. In order to detect if such a case occurred, we propose an extension to the ASV model training in which we evaluate the model already during training on the same task and with the same procedure as during evaluation, speaker verification, but on a subset of the training data. We provide several protocols for these extensions. For more information, please see our protocols readme and our paper (soon at SPSC Symposium 2025).

For general information about the overall code base, please check the original repository of the VPC 2024.

Note that you need access to LibriSpeech train-clean-360 to use the scripts in this repository.

Usage

• Follow the install instructions of the Voice Privacy Challenge 2024
• Anonymize training and evaluation data with the anonymization system of your choice (since this a fork of the VPC, all baselines of the challenge are included in this code)
• Run the privacy post evaluation either (a) with or (b) without ASV training.

(a) Post evaluation with ASV training

Choose the post evaluation with ASV training to first train the ASV model on the provided training data and then use this model to evaluate your anonymized evaluation data. This corresponds to the setup of the VPC if training and evaluation data have been anonymized in the same way. Note that ASV training takes longer with our code than with the one of the VPC because the ASV test is performed after every training epoch.

python run_evaluation.py --config configs/eval_post.yaml --overwrite "{\"anon_data_suffix\": \"$anon_data_suffix\"}" --force_compute True

(b) Post evaluation without ASV training

If you already have a trained ASV model and want to skip retraining it again, you can choose the config that uses an external model. You can choose this option e.g. when you want to test how well a different ASV model (trained on data anonymized by a different method) would perform in breaking your anonymization on the evaluation data. Note that this will likely result in a mismatched evaluation.

python run_evaluation.py --config configs/eval_post_external_model.yaml --overwrite "{\"anon_data_suffix\": \"$anon_data_suffix\"}" --force_compute True

Reference

tba

Acknowledgments

Some parts of the code and structure are based on VoicePAT (Paper: https://ieeexplore.ieee.org/document/10365329)