Real Time Voice Chat With ChatDKU using Web Speech API

[hafikhan11]

Currently, we don't have real time voice chatting with chatdku. We only have dictation which means the user speaks something and its converted to text written on the screen. This is what the frontend ui would look like:

As long as you are within the voice chat, web speech api would result in the ai speaking out the answer to you. The answer will also be written in case the user needs to read the text. Clicking on exit voice chat means it would stop speaking.

2026-03-28.17-00-17.mp4

What I plan to implement and enhance:

• how to handle interruptions without having to click on exit voice chat (real time voice chat). With my current code, it just takes the input and reads out the answer to you.
• Natural conversation. (eleven labs might be a better option but its paid)

Any suggestions would be much appreciated.

[pomegranar]

Interesting idea, though I think you may want to weigh the time it will take to develop against the user benefit of such a feature before proceeding. Conversational AI interface is still something that's quite challenging to implement well.
Perhaps you can start small by adding a button for TTS using a decent free voice model (if good free options even exist)
Afterwards you could choose to make the conversation flow naturally using a dictation-controlled loop (similar to ChatGPT voice mode), alongside a new mode for the agent to compose answers in conversational language. You can also look through GitHub to find a solution that's adaptable for ChatDKU

[hafikhan11]

Great, i'll start with the button for text to speech first (my current code does that i guess since once you are in the voice chat button, it will start reading out the answer to you.) So instead of voice chat, we can simply have it read text to speech for now and further progression would include:

• looking for good voice models for tts
• dictation controlled loop
• new mode for the agent (conversational mode)

I'll make these sub-issues and start working on them.

Interesting idea, though I think you may want to weigh the time it will take to develop against the user benefit of such a feature before proceeding. Conversational AI interface is still something that's quite challenging to implement well. Perhaps you can start small by adding a button for TTS using a decent free voice model (if good free options even exist) Afterwards you could choose to make the conversation flow naturally using a dictation-controlled loop (similar to ChatGPT voice mode), alongside a new mode for the agent to compose answers in conversational language. You can also look through GitHub to find a solution that's adaptable for ChatDKU

[Ar-temis]

I would also second Anar and remind you of the development time and energy to benefit ratio. But this could be a good feature for you to learn from. Try to make it modular so that this mode is hot-swappable if anything goes down.

[pomegranar]

@hafikhan11 any updates?

[hafikhan11]

@hafikhan11 any updates?

I have identified what opensource model would be best to use for the tts part.
https://github.com/hexgrad/kokoro

I am now working on utilizing it (would need some help for the backend). I will update here if I hit a roadblock.

[Ar-temis]

Is this a closed source model? If not, Are we hosting it on our server? Remember no data must get out of our server.