Sample (or milisecond) accurate cursor time

[Beyley]

I'm working on a rhythm game, and i'm trying to get sample accurate (or even just milisecond accurate) cursor time to draw my visuals at the right positions, miniaudio at the moment when using ma_sound_get_cursor_in_seconds returns a value which updates slowly, causing constant visual jitter.

Attempting to interpolate this myself then "syncing back" to the audio time just leads to occasional jitter, as i dont "sync" to the miniaudio cursor time perfectly, since i dont know exactly when the miniaudio cursor time changes, is there a way to get some callback to run code exactly when it updates the cursor time?

As if i use SDL_GetPerformanceCounter exactly when it updates the audio time, then im able to call SDL_GetPerformanceCounter whenever i need an accurate time, and subtract by the new value of it by the value of it when the time updated, giving me a (accurate enough) estimate of the current real cursor time

[mackron]

So miniaudio updates the sound's cursor every time the audio thread does a processing step. The size of that step will be fixed intervals, defaulting to 10ms. You can configure that to something smaller if need be. The fact that it's running on the audio thread could be why you're getting jittering.

The engine maintains a running timer in frames that is updated on the audio thread after each processing step. This might be helpful for you. ma_engine_get_time_in_pcm_frames() and ma_engine_get_time_in_milliseconds().

is there a way to get some callback to run code exactly when it updates the cursor time?

There's several ways you might be able to do this.

One way is to implement your own ma_device object. miniaudio uses a layered architecture where the ma_engine object sits on top of a lower level object called ma_device. The concept of ma_device is basically the same kind of concept that SDL uses for audio - it uses a callback for data delivery. Within that callback, you can call ma_engine_read_pcm_frames() and output directly to the callback's output buffer. That function will do all of the engine's audio processing. Once that returns, all active sounds will have had their cursor's updated. See code below:

void data_callback(ma_device* pDevice, void* pOutput, const void* pInput, ma_uint32 frameCount)
{
    ma_engine_read_pcm_frames((ma_engine*)pDevice->pUserData, pOutput, frameCount, NULL);

    // Inspect the value of your sound's cursors. Keep an accumulation of `frameCount` to keep track of the audio timer.
}

// Initialize the device using miniaudio's standard config/init pattern.
ma_device_config deviceConfig = ma_device_config_init(ma_device_type_playback);
deviceConfig.playback.format = ma_format_f32; // Must be f32 when using with the engine.
deviceConfig.playback.channels = 0; // 0 means to use the device's native channel count.
deviceConfig.sampleRate = 48000; // Or whatever sample rate you need. Leave as 0 to use the device's native rate.
deviceConfig.dataCallback = data_callback;
deviceConfig.pUserData = &engine;

ma_device_init(NULL, &deviceConfig, &myDevice);

// Plug your device into your engine config.
ma_engine_config engineConfig = ma_engine_config_init();
engineConfig.pDevice = &myDevice;

ma_engine_init(&engineConfig, &engine);

So it'll be slightly annoying because you'll have to manage your own ma_device object, but you'll get a bit more control over things. The engine_advanced example covers this kind of setup.

Another option is to plug in a custom passthrough node immediately after your sound. That will have it's processing callback fired from the audio thread, straight after the sound has had it's processing completed, and therefore it's cursor updated. Here's an example for custom node: https://github.com/mackron/miniaudio/tree/master/extras/nodes/ma_reverb_node. You would set the MA_NODE_FLAG_PASSTHROUGH flag in your node's vtable.

Finally, if you're brave, you can use the dev-0.12 branch. That branch will eventually become the 0.12 release of miniaudio, and it includes the ability to plug in a callback that get's fired just after the sound has been processed. It gets called on the audio thread so therefore it'll also be fired straight after the cursor has been updated. That branch introduces the notion of sound notifications. Do a search for the ma_sound_notifications object if this is something you might be interested in. Just note that the dev-0.12 branch is still WIP and I might change APIs.

Happy to go through any of these ideas in more detail if need be.

[mackron]

OK, let's take a step back and rethink this. I don't think the current approach is going to work cleanly.

Internally, the ma_engine object wraps around a ma_device and calls ma_engine_read_pcm_frames() in the device's data callback when it needs to process audio. I'm wondering if I should add support for plugging in a callback that get's fired from the device's callback:

static void ma_engine_data_callback_internal(ma_device* pDevice, void* pFramesOut, const void* pFramesIn, ma_uint32 frameCount)
{
    ma_engine* pEngine = (ma_engine*)pDevice->pUserData;

    // .. Some unrelated Emscripten-specific stuff is here ...

    ma_engine_read_pcm_frames(pEngine, pFramesOut, frameCount, NULL);

    // Add this? This will fire a callback in the device's data callback and will only be called after all the splitting
    // logic has been completed which means timers should have all been resolved for the current period.
    if (pEngine->onProcess) {
        pEngine->onProcess(pEngine->onProcessUserData, pFramesOut, frameCount);
    }
}

And you can set the callback with something like ma_engine_set_process_callback(&engine, my_engine_process_callback, pMyData). This callback would get fired after everything has been processed, including the whole splitting thing, so those numbers should all be in line with real time. miniaudio will fire the device's data callback in known fixed intervals so you can know the PCM frame index that'll be coming next which you can use for your interpolation.

That make sense?

[Beyley]

OK, let's take a step back rethink this. I don't think the current approach is going to work cleanly.

Internally, the ma_engine object wraps around a ma_device and calls ma_engine_read_pcm_frames() in the device's data callback when it needs to process audio. I'm wondering if I should add support for plugging in a callback that get's fired from the device's callback:

static void ma_engine_data_callback_internal(ma_device* pDevice, void* pFramesOut, const void* pFramesIn, ma_uint32 frameCount)
{
    ma_engine* pEngine = (ma_engine*)pDevice->pUserData;

    // .. Some unrelated Emscripten-specific stuff is here ...

    ma_engine_read_pcm_frames(pEngine, pFramesOut, frameCount, NULL);

    // Add this? This will fire a callback in the device's data callback and will only be called after all the splitting
    // logic has been completed which means timers should have all been resolved for the current period.
    if (pEngine->onProcess) {
        pEngine->onProcess(pEngine->onProcessUserData, pFramesOut, frameCount);
    }
}

And you can set the callback with something like ma_engine_set_process_callback(&engine, my_engine_process_callback, pMyData). This callback would get fired after everything has been processed, including the whole splitting thing, so those numbers should all be in line with real time. miniaudio will fire the device's data callback in known fixed intervals so you can know the PCM frame index that'll be coming next which you can use for your interpolation.

That make sense?

That does make sense, i think that would work

[mackron]

OK, I've updated both dev and dev-0.12. To be honest, this is probably a good feature to have anyway. A few things to consider:

• The function will be called after processing which means the timers and cursors will start greater than zero (equivalent to the size of the period) at the time of the callback.
• It will continuously be ticking away even when the sound isn't playing, so that seeking issue might need to be handled in some way.
• I've updated the sound processing callback to take a 64-bit frame count instead of 32-bit. In practice it'll never exceed 32-bits, but I wanted to make it 64-bit for consistency with this new callback. You might therefore get a compilation error - just change it to 64 and you'll be right.

You need to set it via the engine config:

engineConfig.onProcess = on_engine_process;
engineConfig.pProcessUserData = NULL;

Hopefully this will simplify the interpolation stuff. With this I don't think you'll need your sound callback anymore?

[Beyley]

Unfortanately still does not help, my interp logic is still broken but in a different way now, where im overshooting and undershooting the interpolated time, seemingly alternating between overshooting and undershooting the value each time i get a process callback, heres a video explaining the full logic of the interp/audio stuff so far, as you may be able to tell from the video im very tired, since ive been working on this for about 8 hours now, so i will be going to sleep and probably pick this up again in the morning
https://www.youtube.com/watch?v=8a_DX-J7gHE

[mackron]

Very frustrating! Hard to see the jumping in the YouTube video, but I think I understand what you're describing. So the engine timer will be running as soon as the engine starts. Do you need to sync your system timer and engine's timer, maybe? ma_engine_set_time_in_pcm_frames() will allow you to update the engine's timer (again, this is just a simple setter - it doesn't do any seeking or anything). Just a random thought off the top of my head...

By the way, if you like, you can jump on my Discord server and we can go through this through DMs if that would make this easier. Invite link is in the readme.