Technical Feasibility

[tifilipebr]

Objective

This issue aims to discuss and document the technical feasibility of SpeechHub, detailing the key aspects that will enable its implementation. From this discussion, we seek to consolidate a pull request containing a complete document on the technical feasibility of the solution.

Context

SpeechHub aims to be a real-time dubbing and translation tool for online meetings, integrating with platforms such as Google Meet, Microsoft Teams, and Skype. To make this feasible, it is essential to explore the technologies, APIs, and algorithms that will enable the solution's implementation.

Discussion Topics

1. Integration with Communication Platforms

• Available APIs: Review the APIs of Google Meet, Microsoft Teams, and Skype to determine which support real-time audio capture.
• Audio injection methods: Explore techniques to send translated audio into the call without interfering with the platform's functionality.
• Middleware or extensions: Evaluate the possibility of using browser extensions, bots, or intermediary software to capture and inject audio.

2. Audio Processing and Translation

• Speech Recognition (ASR - Automatic Speech Recognition): Models such as Whisper (OpenAI), Google Speech-to-Text, Azure Speech API, or Deepgram.
• Text Translation (NMT - Neural Machine Translation): APIs such as Google Translate, DeepL, and Azure Translator.
• Text-to-Speech (TTS) Synthesis: Tools like Amazon Polly, Azure Speech TTS, and Google Cloud TTS for generating translated audio.
• Latency and optimization: Strategies to minimize delays in conversion and audio playback.

3. Infrastructure and Architecture

• Backend and APIs: Define a scalable architecture using Node.js, Python, or another suitable language.
• Hosting and Computing: Use services like AWS, Azure, or GCP for audio processing.
• Real-time streaming and communication: Technologies such as WebRTC to reduce latency in audio transmission.

4. Security and Privacy

• Data encryption: Measures to protect audio and text data during transmission.
• Compliance with regulations: Such as GDPR and LGPD, ensuring user privacy.
• Data storage and disposal: Strategies to avoid unnecessary storage of processed audio.

5. Technologies and Tools

• Suggested programming languages and frameworks.
• Audio processing and AI libraries.
• Monitoring and logging tools for performance analysis.

How to Contribute

1. Share insights on the most suitable APIs and frameworks.
2. Evaluate potential limitations and technical challenges.
3. Suggest alternatives for performance optimization.
4. Discuss solutions for integration with online meeting platforms.

Conclusion and Next Steps

With community contributions, we aim to consolidate a technical document validating the solution's feasibility. This issue will be closed upon submission of a pull request containing the finalized documentation.

Let's build SpeechHub together! 🚀

[ThyagoMartins0]

The proposed idea is very valid, but I believe that the sequence of priorities described above needs to be reviewed. It is essential to start with the basic functionalities that the tool should offer, and only then think about usability on specific platforms (X or Y). Understanding that scalability for different levels and integrations is a crucial point of the project; however, designing everything with integration with several APIs in mind can “restrict” the size and focus of the solution at this early stage.

1. Integration with Communication Platforms
   Prioritize one platform at a time
   Initially focus on the most popular one — for example, Google Meet, which is more open and has fewer usage restrictions. This way, it will be possible to understand the real application difficulties before expanding to other platforms.

2. Audio Injection Methods
   Architecture before the method
   Before discussing injection methods, it is important to define the architecture. One possibility would be to create your own AI (without relying on third-party APIs) that converts text to audio.

Avoid working on the project
Using an external API directly at this point may limit the flexibility for future growth or even make the project dependent on services that may become unavailable.

3. Middleware or Extensions
   Impact on Immediacy
   The use of middleware or extensions that add multiple steps to the process may break the “instant” response proposal of the project. If the goal is a fast flow, we will compensate by adopting robust middleware.

4. Audio Processing and Translation
   In-house training and creation
   Developing and training your own AI solution would be interesting, as it would avoid dependence on external APIs that may become unavailable or become paid.

Open source alternatives
If you want ready-made models for speech recognition or translation, Hugging Face offers open source AI models that can be used for free.

5. Infrastructure and Architecture
   APIs and backend
   When using pure Node, consider TypeScript, which can offer the best typing and organization. If there are parts that are not necessarily in Python, Go can also become an option for specific microservices.

Hexagonal architecture (Ports and Adapters)
It can be a great choice to allow scalability and the addition of new external APIs in the future, without modifying the business layer. This simplifies maintenance when other communication platforms are added.

6. Hosting and Computing
   Cloud services
   Using AWS or similar services would be interesting due to the scalability and ready-made infrastructure.

Real-time transmission and communication
Technologies such as WebRTC can reduce latency in audio transmission and ensure a more fluid experience.

[ThyagoMartins0]

Regarding Topic 5 (Infrastructure and Architecture), we have two main approaches:

Use two distinct languages

Divide the system into two projects that communicate with each other:

A project in TypeScript (or Node.js), responsible for the layers that do not involve audio processing/translation; it can act as a BFF (Backend for Frontend) or call orchestrator.

Another project in Python, focused on the audio processing and translation core, running as a dedicated microservice.

This scenario favors the microservices architecture and allows each team to work in their preferred language. It also allows each part to be scaled, maintained and deployed independently.

Adopt Python for everything

If the team prefers to simplify the ecosystem, the entire backend can be developed in Python (including APIs, business logic and integrations).

There are several Python frameworks (Flask, FastAPI, Django, etc.) that can meet the needs of APIs and microservices, in addition to libraries for AI/translation.

This choice unifies the stack, but centralizes more responsibilities in a single language.

Considerations
Trade-offs of Multiple Languages

Pros: Specialization in each part of the system, better organization by responsibility and team scalability.

Cons: Increased operational complexity (multiple repositories, pipelines, deployments and integrations).

Maintenance and Team

If the team is proficient in Python for AI but also wants to leverage TypeScript for APIs and business logic, the dual-language option may be attractive.

If the team is small and wants to reduce complexity, using only Python can simplify things a lot.

Performance and Scalability

Microservices in different languages ​​can scale independently, but involve network communication between services.

If the focus is to minimally distribute the layers, the “TS + Python” model brings flexibility for future additions of services in other languages ​​(for example, Go).

[tifilipebr]

Hi Thyago,

I'm glad you liked the idea and took the time to contribute to the project.
It makes me really happy to see one more person helping me with this dream project.

---

✅ Alignment on Minimum Viable Scope

I agree with you that we should define the minimum viable scope so we can get the project into production as soon as possible. This is essential for having something functional quickly and starting to deliver value.

---

🔌 Integrations with External APIs

Regarding designing the project with external API integrations in mind:

• The goal here was precisely to reduce the project’s responsibilities by leveraging what’s already available.
• It also helps us avoid direct competition with major players, especially in each micro-SaaS that might emerge from this project.
• This approach makes more sense, at least in this initial development phase.
  Does that make sense to you as well?

---

🎥 Integration Platform Priority

I agree with prioritizing Google Meet as the initial platform.
I think it's a valid strategic decision, considering its openness and fewer restrictions.

---

🏗️ Architecture vs. Audio Injection Methods

I'd like to better understand your view on why you believe we should prioritize the architecture before the audio injection methods.

Also, regarding your suggestion to create our own API for text-to-speech conversion:

• I believe this would increase the product scope, which is not the goal at this moment.
• We want to keep the focus lean and get to production as soon as possible.

---

⚠️ Use of External APIs

At this point, I disagree with the idea of avoiding the use of external APIs.
From my perspective:

• To move to production quickly, third-party APIs are allies to our project.
• To avoid infrastructure costs, I believe our solution should run in the browser, on the user's or company's machine or server.

But I’m open to discussion!
Feel free to explain more about why you suggest avoiding this type of dependency.

---

🔄 Robust Middleware

You mentioned the use of a "robust middleware."

Could you explain more about what you mean by that?
I'm curious about how this component would work within our flow.

---

🧠 Audio Processing and Translation

I found your considerations and suggestions on open-source alternatives very valuable.

• SpeechHub, for example, should allow integration with different AI models.
• This would give users the option to choose their preferred AI model for the process.

---

🧱 Topic-Based Organization

I'd like to take this opportunity to reinforce that:

• In this specific issue, the main focus is to discuss the feasibility of the project.
• I saw that you brought up many valuable points about architecture, which are definitely important, but maybe we can dive deeper into that in the issue already opened specifically for architecture.

---

💻 Programming Languages

Regarding the use of TypeScript and Python, I agree with your suggestion.

But it’s worth noting that this decision will depend on which developers get involved in the implementation.
We can leave that open until the technical team is defined.

[tifilipebr]

I’d like to invite more people to contribute to this topic, but I must admit I’m feeling a bit anxious to move the project forward. With that in mind, I’m sharing below the main points that support the technical feasibility of the proposal. I believe this can help clarify the idea, give the project more credibility, and hopefully encourage more contributions.

---

1. Choosing Google Meet as the pilot platform

Before diving into the technical details, I’d like to tag @jpoehnelt (Developer Relations Engineer at Google), just to inform you that we’ve chosen Google Meet as the pilot platform for this project.

If there are any limitations regarding the use of Google Meet for this solution, we’d be happy to hear your feedback here.

---

2. Use of the Meet Media API

We’ve identified that Google Meet provides the [Meet Media API](https://developers.google.com/workspace/meet/media-api/guides/ts), which allows real-time access to the audio from meetings. This feature is essential for the core proposal of the application: capturing spoken audio in real time. We consider this the first step in our translation workflow.

Points of attention regarding the API:

• Configuration requires access credentials, and apparently each meeting host will need to obtain their own credentials.
• When installing or configuring the Speech Hub, it will be necessary to register these credentials individually.

---

3. Audio injection on the user’s machine

To enable playback of the translated audio, we’ve researched two approaches for audio injection:

• Virtual microphone: Creating and managing a virtual microphone device (e.g., VB-Cable, VoiceMeeter, Loopback) seems to be the best option. This approach would also be useful for platforms beyond Google Meet.
• Browser automation: As an alternative, tools like Puppeteer or Playwright could be used to play audio via browser automation, although this solution has more limitations.

---

4. System workflow summary

Below is a general overview of how the solution works:

1. Audio capture via Meet Media API
2. Speech recognition (ASR - Automatic Speech Recognition)
3. Text translation (NMT - Neural Machine Translation)
4. Speech synthesis (TTS - Text-to-Speech)
5. Playback of translated audio via virtual audio driver (VB-Cable, VoiceMeeter, Loopback, etc.)

---

5. Choosing AI services (ASR, NMT, and TTS)

To move forward, I’d like to ask you to share your preferences for the AI services we should use in the following steps:

• ASR (Automatic Speech Recognition)
• NMT (Neural Machine Translation)
• TTS (Text-to-Speech)

The idea is to select one service for each step to be used as the pilot for the project.

---

Thank you in advance for your support and contributions! Let’s keep going 🚀

[jpoehnelt]

Please send me an email jpoehnelt@google.com if you have questions about implementation.

[diegogit03]

What is the latency expected? The latency is a bigger problem something above 4s will get a bad experience