Struct Live 

pub struct Live { /* private fields */ }

Fluent builder for constructing and connecting Gemini Live sessions.

Accumulates model configuration, callbacks, extractors, phases, watchers, temporal patterns, and tool execution modes, then connects via one of the connect_* methods.

Control-lane callbacks can be registered with _concurrent suffixed methods for fire-and-forget execution. Tools can be marked for background execution via tool_background().

Example

let session = Live::builder()
    .model(GeminiModel::Gemini2_0FlashLive)
    .voice(Voice::Kore)
    .instruction("You are a weather assistant")
    .tools(dispatcher)
    .on_audio(|data| playback_tx.send(data.clone()).ok())
    .on_text(|t| print!("{t}"))
    .on_interrupted(|| async { playback.flush().await; })
    .connect_vertex("project", "us-central1", token)
    .await?;

Extraction Pipeline

let handle = Live::builder()
    .model(GeminiModel::Gemini2_0FlashLive)
    .instruction("You are a restaurant order assistant")
    .extract_turns::<OrderState>(
        flash_llm,
        "Extract: items ordered, quantities, modifications, order_phase",
    )
    .on_extracted(|name, value| async move {
        println!("Extracted {name}: {value}");
    })
    .connect_vertex(project, location, token)
    .await?;

// Read latest extraction from shared State at any time:
let order: Option<OrderState> = handle.extracted("OrderState");

Implementations

impl Live

pub fn before_tool_response<F, Fut>(self, f: F) -> Self

where F: Fn(Vec<FunctionResponse>, State) -> Fut + Send + Sync + 'static, Fut: Future<Output = Vec<FunctionResponse>> + Send + 'static,

Intercept tool responses before they are sent back to Gemini.

Use this to rewrite, augment, or filter tool results based on conversation state. The callback receives the tool responses and the shared State, and returns (potentially modified) responses.

Example

.before_tool_response(|responses, state| async move {
    let order: OrderState = state.get("OrderState").unwrap_or_default();
    responses.into_iter().map(|mut r| {
        r.response["current_order"] = serde_json::to_value(&order).unwrap();
        r
    }).collect()
})

pub fn on_turn_boundary<F, Fut>(self, f: F) -> Self

where F: Fn(State, Arc<dyn SessionWriter>) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Hook called at turn boundaries — after extractors run, before on_turn_complete.

Receives the shared State and a SessionWriter for injecting content into the conversation. Use for context stuffing, K/V data injection, condensed state summaries, or any outbound content interleaving.

Example

.on_turn_boundary(|state, writer| async move {
    let summary = state.get::<String>("summary").unwrap_or_default();
    writer.send_client_content(
        vec![Content::user().text(format!("[Context: {summary}]"))],
        false,
    ).await.ok();
})

pub fn on_audio(self, f: impl Fn(&Bytes) + Send + Sync + 'static) -> Self

Called for each audio chunk from the model (PCM16 24kHz).

pub fn on_text(self, f: impl Fn(&str) + Send + Sync + 'static) -> Self

Called for each incremental text delta.

pub fn on_text_complete(self, f: impl Fn(&str) + Send + Sync + 'static) -> Self

Called when model completes a text response.

pub fn on_input_transcript( self, f: impl Fn(&str, bool) + Send + Sync + 'static, ) -> Self

Called for input (user speech) transcription.

pub fn on_output_transcript( self, f: impl Fn(&str, bool) + Send + Sync + 'static, ) -> Self

Called for output (model speech) transcription.

pub fn on_thought(self, f: impl Fn(&str) + Send + Sync + 'static) -> Self

Called when the model emits a thought/reasoning summary.

Requires .include_thoughts() on the session config. Fast lane callback (sync, must complete in < 1ms).

pub fn on_vad_start(self, f: impl Fn() + Send + Sync + 'static) -> Self

Called when server VAD detects voice activity start.

pub fn on_vad_end(self, f: impl Fn() + Send + Sync + 'static) -> Self

Called when server VAD detects voice activity end.

pub fn on_usage( self, f: impl Fn(&UsageMetadata) + Send + Sync + 'static, ) -> Self

Called when server sends token usage metadata.

Receives a reference to the full UsageMetadata including prompt, response, cached, tool-use, and thoughts token counts plus per-modality breakdowns. Fires on the telemetry lane (not the fast lane).

pub fn on_interrupted<F, Fut>(self, f: F) -> Self

where F: Fn() -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when model is interrupted by barge-in.

pub fn on_tool_call<F, Fut>(self, f: F) -> Self

where F: Fn(Vec<FunctionCall>, State) -> Fut + Send + Sync + 'static, Fut: Future<Output = Option<Vec<FunctionResponse>>> + Send + 'static,

Called when model requests tool execution. Return None to auto-dispatch, Some(responses) to override. Receives State for natural state promotion from tool results.

pub fn on_turn_complete<F, Fut>(self, f: F) -> Self

where F: Fn() -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when model turn completes.

pub fn on_go_away<F, Fut>(self, f: F) -> Self

where F: Fn(Duration) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when server sends GoAway.

pub fn on_connected<F, Fut>(self, f: F) -> Self

where F: Fn(Arc<dyn SessionWriter>) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when session connects (setup complete).

Receives a SessionWriter for sending messages on connect.

pub fn on_disconnected<F, Fut>(self, f: F) -> Self

where F: Fn(Option<String>) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when session disconnects.

pub fn on_error<F, Fut>(self, f: F) -> Self

where F: Fn(String) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called on non-fatal errors.

pub fn on_turn_complete_concurrent<F, Fut>(self, f: F) -> Self

where F: Fn() -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when model turn completes (spawned concurrently).

pub fn on_connected_concurrent<F, Fut>(self, f: F) -> Self

where F: Fn(Arc<dyn SessionWriter>) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when session connects (spawned concurrently).

pub fn on_disconnected_concurrent<F, Fut>(self, f: F) -> Self

where F: Fn(Option<String>) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when session disconnects (spawned concurrently).

pub fn on_error_concurrent<F, Fut>(self, f: F) -> Self

where F: Fn(String) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called on non-fatal errors (spawned concurrently).

pub fn on_go_away_concurrent<F, Fut>(self, f: F) -> Self

where F: Fn(Duration) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when server sends GoAway (spawned concurrently).

pub fn on_extracted_concurrent<F, Fut>(self, f: F) -> Self

where F: Fn(String, Value) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when a TurnExtractor produces a result (spawned concurrently).

pub fn on_extraction_error_concurrent<F, Fut>(self, f: F) -> Self

where F: Fn(String, String) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when a TurnExtractor fails (spawned concurrently).

impl Live

pub fn model(self, model: GeminiModel) -> Self

Set the Gemini model.

pub fn voice(self, voice: Voice) -> Self

Set the output voice.

pub fn instruction(self, instruction: impl Into<String>) -> Self

Set the system instruction.

pub fn text_only(self) -> Self

Switch to text-only mode (no audio output).

Sets response modality to Text and disables speech config. Use with GeminiModel::Gemini2_0FlashLive for text-only conversations.

pub fn add_tool(self, tool: Tool) -> Self

Add a raw Tool declaration to the session configuration.

Use this for tools that aren’t registered through the ToolDispatcher (e.g., raw FunctionDeclaration lists, Google Search, code execution).

pub fn greeting(self, prompt: impl Into<String>) -> Self

Set a greeting prompt to trigger the model to initiate the conversation.

When set, this text is sent immediately after the session connects, causing the model to respond first (e.g. with a greeting or introduction).

let handle = Live::builder()
    .model(GeminiModel::Gemini2_0FlashLive)
    .instruction("You are a friendly assistant")
    .greeting("Greet the user warmly and introduce yourself.")
    .connect_vertex(project, location, token)
    .await?;
// Model will speak first without any user input

pub fn temperature(self, temp: f32) -> Self

Set the temperature.

pub fn tools(self, dispatcher: ToolDispatcher) -> Self

Set the tool dispatcher (auto-dispatches tool calls).

pub fn with_tools(self, composite: ToolComposite) -> Self

Register tools from a T module composition.

use gemini_adk_fluent_rs::prelude::*;

Live::builder()
    .with_tools(
        T::simple("get_weather", "Get weather", |args| async move {
            Ok(serde_json::json!({"temp": 22}))
        })
        | T::google_search()
    )

pub fn agent_tool( self, name: impl Into<String>, description: impl Into<String>, agent: impl TextAgent + 'static, ) -> Self

Register a text agent as a tool the live model can call.

The agent shares the session’s State, so it can read live-extracted values and its mutations are visible to watchers and phase transitions.

Live::builder()
    .agent_tool("verify_identity", "Verify caller identity", verifier_agent)
    .agent_tool("calc_payment", "Calculate payment plans", calc_pipeline)

pub fn agent_tool_arc( self, name: impl Into<String>, description: impl Into<String>, agent: Arc<dyn TextAgent>, ) -> Self

Register a text agent (already Arc’d) as a tool.

pub fn google_search(self) -> Self

Enable Google Search built-in tool.

pub fn code_execution(self) -> Self

Enable code execution built-in tool.

pub fn url_context(self) -> Self

Enable URL context built-in tool.

pub fn tool_background(self, tool_name: impl Into<String>) -> Self

Mark a tool for background execution (zero dead-air).

When the model calls this tool, an immediate “running” acknowledgment is sent back while the tool executes in a background task. The final result is delivered asynchronously when complete.

pub fn tool_background_with_formatter( self, tool_name: impl Into<String>, formatter: Arc<dyn ResultFormatter>, ) -> Self

Mark a tool for background execution with a custom result formatter.

The formatter controls the shape of the acknowledgment (“running”), completion, and cancellation messages sent to the model.

pub fn tool_background_with_scheduling( self, tool_name: impl Into<String>, scheduling: FunctionResponseScheduling, ) -> Self

Mark a tool for background execution with a specific scheduling mode.

The scheduling mode controls how the model handles async results:

• Interrupt: halts current output, immediately reports the result
• WhenIdle: waits until current output finishes before handling
• Silent: integrates the result without notifying the user

pub fn transcription(self, input: bool, output: bool) -> Self

Enable input and/or output transcription.

pub fn thinking(self, budget: u32) -> Self

Enable thinking/reasoning with a token budget (Gemini 2.5+).

Sets the thinking budget for the Live session. Use with .include_thoughts() and .on_thought() to receive thought summaries.

Live::builder()
    .thinking(1024)
    .include_thoughts()
    .on_thought(|text| println!("[Thought] {text}"))

Platform support: Google AI only. On Vertex AI, thinkingConfig is automatically stripped from the setup message.

pub fn include_thoughts(self) -> Self

Include the model’s thought summaries in responses.

When enabled, the model emits SessionEvent::Thought events containing its reasoning process. Register an .on_thought() callback to receive them.

Platform support: Google AI only. Stripped on Vertex AI.

pub fn affective_dialog(self, enabled: bool) -> Self

Enable affective dialog (emotionally expressive responses).

pub fn proactive_audio(self, enabled: bool) -> Self

Enable proactive audio.

pub fn media_resolution(self, res: MediaResolution) -> Self

Set media resolution for video/image input.

pub fn vad(self, detection: AutomaticActivityDetection) -> Self

Configure server-side VAD.

pub fn activity_handling(self, handling: ActivityHandling) -> Self

Set activity handling mode (interrupts vs no-interruption).

pub fn turn_coverage(self, coverage: TurnCoverage) -> Self

Set turn coverage mode.

pub fn session_resume(self, enabled: bool) -> Self

Enable session resumption.

pub fn context_compression( self, trigger_tokens: u32, target_tokens: u32, ) -> Self

Enable context window compression.

pub fn soft_turn_timeout(self, timeout: Duration) -> Self

Enable soft turn detection for proactive silence awareness.

When proactiveAudio is enabled, the model may choose not to respond. After VAD end, if the model stays silent for timeout, a lightweight “soft turn” updates state and fires watchers without forcing a response.

pub fn steering_mode(self, mode: SteeringMode) -> Self

Set the steering mode for how the phase machine delivers instructions.

• InstructionUpdate (default): Replace system instruction on transition.
• ContextInjection: Inject steering via send_client_content.
• Hybrid: Instruction on transition, context injection per turn.

pub fn context_delivery(self, mode: ContextDelivery) -> Self

Set when model-role context turns are delivered to the wire.

• Immediate (default): Send as a single batched frame during TurnComplete processing.
• Deferred: Queue context and flush before the next user send (send_audio/send_text/send_video). Eliminates isolated WebSocket frames during silence that can confuse the model.

Live::builder()
    .steering_mode(SteeringMode::ContextInjection)
    .context_delivery(ContextDelivery::Deferred)
    .phase("greeting")
        .instruction("Welcome the guest")
        .done()
    .initial_phase("greeting")

pub fn repair(self, config: RepairConfig) -> Self

Enable the conversation repair protocol.

Tracks unfulfilled needs per phase. After nudge_after stalled turns, injects a gentle nudge. After escalate_after turns, sets repair:escalation in state for phase guards to handle.

pub fn persistence(self, backend: Arc<dyn SessionPersistence>) -> Self

Set a session persistence backend for surviving process restarts.

pub fn session_id(self, id: impl Into<String>) -> Self

Set the session ID for persistence.

pub fn tool_advisory(self, enabled: bool) -> Self

Enable or disable tool availability advisory on phase transitions.

When enabled (default), the SDK injects a model-role context turn telling the model which tools are available in the new phase.

impl Live

pub async fn connect_google_ai( self, api_key: impl Into<String>, ) -> Result<LiveHandle, AgentError>

Connect using a Google AI API key.

pub async fn connect_vertex( self, project: impl Into<String>, location: impl Into<String>, access_token: impl Into<String>, ) -> Result<LiveHandle, AgentError>

Connect using Vertex AI credentials.

pub async fn connect( self, config: SessionConfig, ) -> Result<LiveHandle, AgentError>

Connect using a pre-configured SessionConfig for auth and model.

Merges the provided config’s endpoint and model into the builder’s config, preserving system instruction, tools, voice, transcription, and all other settings configured via the fluent API.

impl Live

pub fn extract_turns<T>( self, llm: Arc<dyn BaseLlm>, prompt: impl Into<String>, ) -> Self

where T: DeserializeOwned + Serialize + JsonSchema + Send + Sync + 'static,

Add a turn extractor that runs an OOB LLM after each turn to extract structured data from the transcript window.

Automatically enables both input and output transcription. The extraction result is stored in State under the type name (e.g., "OrderState") and can be read via handle.extracted::<T>(name).

The type T must implement JsonSchema for schema-guided extraction. The window size defaults to 3 turns.

pub fn extract_turns_windowed<T>( self, llm: Arc<dyn BaseLlm>, prompt: impl Into<String>, window_size: usize, ) -> Self

where T: DeserializeOwned + Serialize + JsonSchema + Send + Sync + 'static,

Like extract_turns but with a custom window size.

pub fn extract_turns_triggered<T>( self, llm: Arc<dyn BaseLlm>, prompt: impl Into<String>, window_size: usize, trigger: ExtractionTrigger, ) -> Self

where T: DeserializeOwned + Serialize + JsonSchema + Send + Sync + 'static,

Like extract_turns_windowed but with a custom extraction trigger.

Use ExtractionTrigger::AfterToolCall when tool calls are the primary state source, ExtractionTrigger::Interval(n) to reduce extraction frequency, or ExtractionTrigger::OnPhaseChange for phase-entry extraction.

pub fn extract_turns_configured<T>( self, llm: Arc<dyn BaseLlm>, prompt: impl Into<String>, window_size: usize, trigger: ExtractionTrigger, configure: impl FnOnce(LlmExtractor) -> LlmExtractor, ) -> Self

where T: DeserializeOwned + Serialize + JsonSchema + Send + Sync + 'static,

Like extract_turns_triggered, but lets callers configure the underlying LlmExtractor before registration.

Use this for field promotion rules, custom minimum word counts, or other extraction policies that should live at the SDK layer instead of app callback glue.

pub fn extractor(self, extractor: Arc<dyn TurnExtractor>) -> Self

Add a custom TurnExtractor implementation.

pub fn on_extracted<F, Fut>(self, f: F) -> Self

where F: Fn(String, Value) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when a TurnExtractor produces a result.

The callback receives the extractor name and the extracted JSON value.

pub fn on_extraction_error<F, Fut>(self, f: F) -> Self

where F: Fn(String, String) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Called when a TurnExtractor fails.

The callback receives the extractor name and error message. Use this for custom error handling (alerting, retry logic, etc.).

impl Live

pub fn instruction_template( self, f: impl Fn(&State) -> Option<String> + Send + Sync + 'static, ) -> Self

State-reactive system instruction template.

Called after extractors run on each turn. If it returns Some(instruction), the system instruction is updated mid-session (deduped — same instruction is not sent twice). Returns None to leave the instruction unchanged.

Example

.instruction_template(|state| {
    let phase: String = state.get("phase").unwrap_or_default();
    match phase.as_str() {
        "ordering" => Some("Focus on taking the order accurately.".into()),
        "confirming" => Some("Summarize and confirm the order.".into()),
        _ => None,
    }
})

pub fn instruction_amendment( self, f: impl Fn(&State) -> Option<String> + Send + Sync + 'static, ) -> Self

State-reactive instruction amendment (additive, not replacement).

Unlike instruction_template (which replaces the entire instruction), this appends to the current phase instruction. The developer never needs to know or repeat the base instruction.

Example

.instruction_amendment(|state| {
    let risk: String = state.get("derived:risk").unwrap_or_default();
    if risk == "high" {
        Some("[IMPORTANT: Use empathetic language. Do not threaten.]".into())
    } else {
        None
    }
})

pub fn computed( self, key: impl Into<String>, deps: &[&str], f: impl Fn(&State) -> Option<Value> + Send + Sync + 'static, ) -> Self

Register a computed (derived) state variable.

The compute function receives the full State and returns Some(value) to write to derived:{key}, or None to skip.

pub fn phase_defaults( self, f: impl FnOnce(PhaseDefaults) -> PhaseDefaults, ) -> Self

Set default modifiers and prompt_on_enter inherited by all phases.

Phase-specific modifiers are applied after defaults, so they extend (not replace).

Live::builder()
    .phase_defaults(|p| {
        p.with_state(&["emotional_state", "risk_level"])
         .when(risk_is_elevated, "Show extra empathy.")
         .prompt_on_enter(true)
    })
    .phase("greet").instruction("...").done()
    .phase("close").instruction("...").done()
    // Both phases inherit the modifiers and prompt_on_enter.

pub fn phase(self, name: impl Into<String>) -> PhaseBuilder

Start building a conversation phase.

Returns a PhaseBuilder that flows back to this Live via .done().

pub fn initial_phase(self, name: impl Into<String>) -> Self

Set the initial phase name (must match a registered phase).

pub fn watch(self, key: impl Into<String>) -> WatchBuilder

Start building a state watcher.

Returns a WatchBuilder that flows back to this Live via .then().

pub fn when_sustained<F, Fut>( self, name: impl Into<String>, condition: impl Fn(&State) -> bool + Send + Sync + 'static, duration: Duration, action: F, ) -> Self

where F: Fn(State, Arc<dyn SessionWriter>) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Register a sustained condition pattern.

Fires when the condition remains true for at least duration.

pub fn when_rate<F, Fut>( self, name: impl Into<String>, filter: impl Fn(&SessionEvent) -> bool + Send + Sync + 'static, count: u32, window: Duration, action: F, ) -> Self

where F: Fn(State, Arc<dyn SessionWriter>) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Register a rate detection pattern.

Fires when at least count matching events occur within window.

pub fn when_turns<F, Fut>( self, name: impl Into<String>, condition: impl Fn(&State) -> bool + Send + Sync + 'static, turn_count: u32, action: F, ) -> Self

where F: Fn(State, Arc<dyn SessionWriter>) -> Fut + Send + Sync + 'static, Fut: Future<Output = ()> + Send + 'static,

Register a turn count pattern.

Fires when the condition is true for turn_count consecutive turns.

impl Live

pub fn builder() -> Self

Start building a Live session.

Examples

Minimal live session setup:

use gemini_adk_fluent_rs::prelude::*;

let handle = Live::builder()
    .model(GeminiModel::Gemini2_0FlashLive)
    .voice(Voice::Kore)
    .instruction("You are a helpful assistant")
    .greeting("Hello! How can I help?")
    .on_audio(|data| { /* send to speaker */ })
    .on_text(|t| print!("{t}"))
    .connect_google_ai("API_KEY")
    .await?;

handle.send_text("What is the weather?").await?;
handle.disconnect().await?;

With phases and state-based transitions:

let handle = Live::builder()
    .model(GeminiModel::Gemini2_0FlashLive)
    .phase("greeting")
        .instruction("Welcome the user")
        .transition("main", S::is_true("greeted"))
        .done()
    .phase("main")
        .instruction("Help the user")
        .terminal()
        .done()
    .initial_phase("greeting")
    .connect_google_ai("API_KEY")
    .await?;

pub fn telemetry_interval(self, interval: Duration) -> Self

Set the periodic telemetry emission interval.

When set, the processor emits LiveEvent::Telemetry snapshots and LiveEvent::TurnMetrics at this rate.