Automatically detect non-dialog audio and display them as short text info

[cutegitcat]

Many videos, fictional and non-fictional use non-speech sounds and noises – such as footsteps, doors, animals, music, traffic, explosions as integral elements of the storytelling. For deaf people like me, this kind of information is essential to understand the context of a scene, but it lacks often in available subtitles/captions. Automatic ambient sound detection would significantly improve media accessibility for many deaf and hard of hearing users. But currently, most tools do not detect or transcribe such sounds accurately and reliably, if at all. However, there are toolkits available that might help to enhance this.

Two projects I found:
• YAMNet – a sound classification model that can run locally:
https://www.tensorflow.org/hub/tutorials/yamnet
• Vosk – offline speech recognition, works without internet:
https://github.com/alphacep/vosk-api

It would be great if the functionality of such a toolkit could be integrated into your tool to create subtitles/captions that reliably include not only dialogue, but also noises, and other non-speech sound elements. I am not a developer, and cannot judge the feasibility of this idea. Please understand my feature request as an enhancement to your already great product, that helps us to already in many ways better access the world of sounds and speech in media.

Thank you to the development team for all their hard work so far, and best of luck with future improvements!

[kaixxx]

Thank you, that's an interesting suggestion. NoScribe is not really optimized for subtitles (SubtitelEdit seems quite popular).
However, I might still come back to this idea for a future version, if I find the time. YAMNet in particular looks promising. Including non-speech events in the transcript could also be interesting for qualitative research.

[cutegitcat]

Hello and dear kaixxx

I have received some additional information: there is another tool that can automatically transcribe background and environmental sounds. It is an academic development called NonSpeech7k.

Unfortunately, I cannot personally assess whether this tool is better than YAMNet. It would be very helpful if someone from the team could compare both tools and recommend the most suitable one for integration into the workflow.

For the final transcription, any subtitle/caption editor can be used – either by copying and pasting the text or by importing the file.

Especially with the vast number of audio files available online, such as podcasts, there is a major accessibility issue. These types of content are often not accessible, which poses a significant disadvantage for people who are deaf or hard of hearing. A high-quality transcription that includes descriptions of background and ambient sounds could help reduce these barriers.

In today's age of digitalisation and artificial intelligence, very few people have the time to manually create full subtitle/caption files or transcriptions for things like podcasts. Instead, they are usually generated automatically and then reviewed for content and spelling. If necessary, manual corrections are made. This saves a lot of time and effort – and in the end, everyone benefits!

[kaixxx]

Thank you for this extra information, I will take a look whenever I have a little more time available.
Transcribing podcasts would indeed be a very suitable application for noScribe, and I totally agree that improving access to this medium for people with hard of hearing would be very desirable.