[AI] Implement RealtimeInputConfig and Manual Activity Signals for Live API by milaGGL · Pull Request #8080 · firebase/firebase-android-sdk

milaGGL · 2026-04-30T16:23:16Z

Add support for RealtimeInputConfig to configure voice activity detection and manual activity signals in the Live API.

gemini-code-assist · 2026-04-30T16:23:21Z

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

github-actions · 2026-04-30T16:23:34Z

📝 PRs merging into main branch

Our main branch should always be in a releasable state. If you are working on a larger change, or if you don't want this change to see the light of the day just yet, consider using a feature branch first, and only merge into the main branch when the code complete and ready to be released.

milaGGL · 2026-04-30T16:56:20Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces comprehensive support for realtime input configuration in the Live API, specifically adding LiveRealtimeInputConfig and LiveActivityDetection. These additions allow for fine-grained control over how the model detects user activity, handles interruptions, and defines turn coverage. The PR includes new public methods sendStartActivityRealtime and sendStopActivityRealtime for manual activity signaling, along with corresponding updates to the Java-friendly LiveSessionFutures and internal serialization logic. Comprehensive unit and instrumentation tests have been added to verify the new configuration options. Review feedback focuses on minor documentation refinements to ensure consistent KDoc linking syntax and improved readability in the LiveSession class.

milaGGL · 2026-04-30T17:44:52Z

+   *
+   * Only required when automatic activity detection is disabled via [LiveRealtimeInputConfig].
+   */
+  public suspend fun sendStartActivityRealtime() {


func name is not decided yet. same for sendStopActivityRealtime.

milaGGL · 2026-04-30T17:45:22Z

+ * starting and ending of the activity.
+ */
+@PublicPreviewAPI
+public class LiveRealtimeInputConfig


name is not decided yet. could be changed to RealtimeInputConfig.

milaGGL · 2026-04-30T17:46:55Z

+            gotTurnComplete = true
          }
+          // Stop collecting when there's a new handle AND turnComplete is true
+          !(gotTurnComplete && lastResumptionUpdate?.newHandle != null)


fixes takeWhile early return error when it received lastResumptionUpdate with null newHandle

If the takeWhile bug starts breaking CI we can spin a different PR just with this fix.

have removed the test fix from this PR.

milaGGL · 2026-04-30T17:48:59Z

+    )
+
+  @Serializable
+  internal data class Internal(


toInternal() and internal are a bit long. Open for suggesstion.

You can make Sensitivity have it's own Internal class and dedup here.

milaGGL · 2026-04-30T18:02:44Z

+   * Only required when automatic activity detection is disabled via [LiveRealtimeInputConfig].
+   */
+  public suspend fun sendStartActivityRealtime() {
+    sendFrame(BidiGenerateContentRealtimeInputSetup(activityStart = true).toInternal())


activity_start and activity_end are expected to be an object with no field: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/reference/rpc/google.cloud.aiplatform.v1beta1#activitystart

these two fields set to booleans in BidiGenerateContentRealtimeInputSetup to be serialized to {} later.

open for code optimization here.

I do have considered using a sealed interface instead ,like this:

internal sealed interface BidiRealtimeInput { data class Text(val text: String) : BidiRealtimeInput data class Audio(val data: InlineData) : BidiRealtimeInput data class Video(val data: InlineData) : BidiRealtimeInput data class MediaStream(val chunks: List<InlineData>) : BidiRealtimeInput object ActivityStart : BidiRealtimeInput object ActivityEnd : BidiRealtimeInput }

and seriaization works like this

internal class BidiGenerateContentRealtimeInputSetup(val input: BidiRealtimeInput) { @Serializable internal class ActivityStart @Serializable internal class ActivityEnd @Serializable internal class Internal(val realtimeInput: BidiGenerateContentRealtimeInput) { @Serializable internal data class BidiGenerateContentRealtimeInput( val mediaChunks: List<InlineData.Internal>? = null, ... ) } fun toInternal() = Internal( when (input) { is BidiRealtimeInput.Text -> Internal.BidiGenerateContentRealtimeInput(text = input.text) ... is BidiRealtimeInput.ActivityStart -> Internal.BidiGenerateContentRealtimeInput(activityStart = ActivityStart()) ... } ) }

By doing so, we are no longer using boolean for activityStart, activityEnd. but we are also fixed to mutually exclusive input action (Only one input action,Text OR Audio OR Start signal can be sent once).

If encoding works, and given that this is internal, feel free to go that way!

rlazo · 2026-05-01T15:45:02Z

+) {
+
+  /** How sensitive the model interprets speech activity. */
+  public enum class Sensitivity {


public API shouldn't be using enums

rlazo · 2026-05-01T15:46:28Z

+    )
+
+  @Serializable
+  internal data class Internal(


You can make Sensitivity have it's own Internal class and dedup here.

rlazo · 2026-05-01T15:47:45Z

+  ) {
+    @Serializable
+    internal enum class StartSensitivity {
+      @SerialName("START_SENSITIVITY_UNSPECIFIED") UNSPECIFIED,


Not 100% necessary, but if the LiveActivityDetection class is only ever created by devs, there's no need for an "unspecified" case.

rlazo · 2026-05-01T15:48:38Z

+) {
+
+  /** How a model handles user input activity. */
+  public enum class ActivityHandling {


public API should have no enums

rlazo · 2026-05-01T16:00:26Z

+  }
+
+  /** How the model considers which input is included in the user's turn. */
+  public enum class TurnCoverage {


rlazo · 2026-05-01T16:22:07Z

+   * Only required when automatic activity detection is disabled via [LiveRealtimeInputConfig].
+   */
+  public suspend fun sendStartActivityRealtime() {
+    sendFrame(BidiGenerateContentRealtimeInputSetup(activityStart = true).toInternal())


If encoding works, and given that this is internal, feel free to go that way!

milaGGL added 2 commits April 30, 2026 12:00

[AI] implement RealtimeInputConfig

826d43b

Merge branch 'main' into mila-realtime-input-config

e98395c

gemini-code-assist Bot reviewed Apr 30, 2026

View reviewed changes

Comment thread ai-logic/firebase-ai/src/main/kotlin/com/google/firebase/ai/type/LiveSession.kt Outdated

Comment thread ai-logic/firebase-ai/src/main/kotlin/com/google/firebase/ai/type/LiveSession.kt Outdated

add changelog, format code

a8edacd

milaGGL changed the title ~~[AI] support realtime input config~~ [AI] Implement RealtimeInputConfig and Manual Activity Signals for Live API Apr 30, 2026

milaGGL commented Apr 30, 2026

View reviewed changes

milaGGL marked this pull request as ready for review May 1, 2026 14:39

Merge branch 'main' into mila-realtime-input-config

34783ad

milaGGL added the do-not-merge/work-in-progress label May 1, 2026

Update LiveSessionTests.kt

a708b30

rlazo requested changes May 1, 2026

View reviewed changes

Conversation

milaGGL commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 PRs merging into main branch

Uh oh!

milaGGL commented Apr 30, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milaGGL Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

milaGGL commented Apr 30, 2026 •

edited

Loading

github-actions Bot commented Apr 30, 2026 •

edited

Loading

milaGGL Apr 30, 2026 •

edited

Loading