SonifyCPP

SonifyCPP is a powerful program for transforming image data into sound. It provides a flexible framework to sonify visual information — enabling new ways to interpret, analyze, or experience images through audio.

Warning

This is still a work in progress

Table of Contents

1. Introduction
2. Features
3. Traversal Methods
   1. Left to Right (Linear)
   2. Right to Left (Linear)
   3. Top to Bottom (Linear)
   4. Bottom to Top (Linear)
   5. Circular Outwards
   6. Circular Inwards
   7. Clockwise (Linear)
   8. Anti-Clockwise (Linear)
   9. Draw Path
4. Pixel Mapping
5. Reference
6. Things to do
7. Demo
8. Installation
9. Changelogs and Bug fixes
10. Inspirations

Introduction

Image sonification refers to the process of converting images to audio signals. This is done by mapping the pixels of the image to frequencies of sound. The way in which the image is traversed changes the resultant audio, and the way in which the pixels are mapped to frequencies also changes the audio. So, there can be an infinite possibilites of mapping things.

This project is a performant successor to Sonify which was written in Python.

Features

Apart from sonifying the images, sonifyCPP ships with few features.

• Tone Generator

  Just generate single frequency tones. It's fun

2024-08-01.14-00-46.mp4

• Waveform visualizer

  Visualize the sonified sound waveform

  

• Frequency Spectrum Analyzer

  Visualize the frequency vs amplitude plot

  

Traversal Methods

SonifyCPP is currently able to traverse images in the following manner:

1. Left to Right (Linear)

2. Right to Left (Linear)

3. Top to Bottom (Linear)

4. Bottom to Top (Linear)

5. Circular Outwards

6. Circular Inwards

7. Clockwise (Linear)

8. Anti-Clockwise (Linear)

9. Draw Path

10. Region

Note

Not implemented yet!

Pixel mapping

Currently, two kinds of pixel to sound mapping is supported:

1. grayscale of the input image, mapping the pixel position (x, y) to (t, frequency) and pixel intensity to amplitude.

2. hsv, map the hue to amplitude.

Reference

Credit: Link

TODO

• Nice GUI
• Save resulting audio
• Waveform visualizer
• Path-based Sonification: Draw a path on the image and sonify along that path.
• Color Mapping: Map different colors to different sounds or musical notes.
• Frequency Analyzer
• Implement multi-threading
• Option for mapping selection
• Support for custom mapping user functions (add scripting support ?)
• Pixel audio visualiser (inpsecting each column or row of pixel from the image and producing the audio instantly).
• Video Export: Create a video file that combines the image and the sonification, showing the progression over time.
• Reverse audio to produce image
• Sonify specific regions or objects within the image.
• Map textures or patterns within the image to different sound effects.
• Effect Processing: Add reverb, echo, or other audio effects to enhance the sonification experience.
• Dynamic Range Compression: Ensure that the sound levels are balanced and not too harsh or too quiet.
• Adjustable Speed: Allow users to slow down or speed up the sonification process.
• MIDI Support: Allow users to export sonification data as MIDI files for further musical manipulation.
• Spectrogram: Display a spectrogram that shows the frequency content of the sonified sound over time.
• 3D Visualization: Provide a 3D view of the image and its corresponding soundscape.
• Handle Large Images

Demo

Warning

Lower your volume when listening to the video.

These are demo from when the software was in version 0.1. Have to update the videos.

2024-07-24.09-58-30.mp4

2024-07-24.10-17-33.mp4

The new GUI looks like this

Installation

Note

I haven't tested this software on Windows, but it works on Linux.

1. Make sure you have the following dependencies

   • Qt6 (GUI)
   • libsndfile (reading and writing audio files)
   • SDL2 (audio playback)
   • ninja (build dependency)
   • fftw3 (fast fourier transforms and inverse transforms)

Since I use Arch Linux, the commands to install these packages is sudo pacman -S qt6 libsndfile sdl2 fftw ninja

2. Clone this repo and go to the directory

   git clone https://github.com/dheerajshenoy/sonifycpp && cd sonifycpp

3. Create a build directory and run cmake`

   mkdir build && cmake -G Ninja -Bbuild -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release

4. Run make inside the build directory

   make -j$(nproc)

Please see CHANGELOG.md

FAQ

Why ?

Main reason why I started this is because of an earlier basic Python version I started on 2023 when I participated in the 2023 NASA Hackathon. It was slow... it was python, and I love C++, so I thought I'll create this in C++.

Why the wierd name ?

Because I already had a python project called sonify, and i wanted CPP in the name, hence sonifyCPP.

Inspirations

• NASA has been a big inspiration for making me create this software. If you haven't checked out yet, they have good sonification videos on youtube.