How WhatsApp Voice Communication Works in Hermes Agent

User sends a voice message

The user records a voice note in WhatsApp. The WhatsApp server pushes it to the connected Baileys client via the WhatsApp Web protocol.

bridge.js receives the message

The Node.js bridge uses Baileys (@whiskeysockets/baileys) to connect to WhatsApp Web and listens for the messages.upsert event.

socket.ev.on('messages.upsert', async ({ messages }) => {
  for (const msg of messages) {
    // Enqueue message internally
    messageQueue.push(msg);
  }
});

Incoming messages are buffered in an internal queue exposed via GET /messages. WhatsApp voice notes arrive as .ogg Opus audio, tagged mediaType="audio" or "ptt".

WhatsAppAdapter polls for messages

The adapter in whatsapp.py runs a long-polling loop, detects voice by mediaType, downloads the audio to a local cache, and builds a MessageEvent.

response = requests.get("http://127.0.0.1:3000/messages")
for msg_data in response.json():
    event = self._build_message_event(msg_data)
    if event.message_type == MessageType.VOICE:
        # Download and cache the audio file
        event = self._process_voice_attachment(event)
    self.handle_message(event)

BasePlatformAdapter processes the message

handle_message() creates a session key, manages concurrent session locking (supporting interruption), and calls _process_message(), which:

• STT transcription — voice is transcribed via Whisper (local), Groq (cloud), or the OpenAI Whisper API; the text becomes the LLM message body.
• Session management — a session is created/loaded from the database with a ~30 min idle TTL.
• Agent invocation — the text goes to AIAgent.run_conversation(), fed to the LLM (DeepSeek V4 Flash) with history + skills.

LLM generates a response

The LLM processes the transcribed message — optionally calling tools like web search or file ops — and returns a response string. It may include MEDIA:<path> tags if the agent called text_to_speech itself.

Decision — should we send voice?

_should_auto_tts_for_chat() evaluates three conditions: voice.auto_tts enabled, channels.whatsapp.voiceReply true, and the agent did not already call TTS itself. Two paths follow:

_process_message()
  → text_to_speech_tool(text=response_text)
    → asyncio.to_thread()   # runs in thread pool
    → returns .mp3 file path
  → play_tts() → send_voice()
    → bridge POST /send-media audio

_process_message()
  → extract_media(response)   # parses MEDIA:<path> tags
  → send_voice()              # sends the generated audio

Audio format conversion (bridge.js)

Before delivery, bridge.js normalizes the format. Supported inputs: ogg / opus / mp3 / wav / m4a — anything that isn't already ogg/opus is converted with ffmpeg on the bridge side.

TTS produces .mp3 (or other format)
        ↓
Not ogg/opus? → convert with ffmpeg to ogg
        ↓
Send with ptt=true (WhatsApp voice bubble mode)
        ↓
User hears the Edge TTS voice on WhatsApp

Delivery to user

send_voice() POSTs the audio to bridge.js at /send-media; Baileys sends it as PTT (push-to-talk) so it renders as the familiar voice-note bubble. The text reply is sent separately via /send.

{
  "chatId": "***@lid",
  "filePath": "/tmp/hermes_tts_abc123.mp3",
  "mediaType": "audio",
  "caption": ""
}