AICaller: A real-time voice interviewer

The problem

Voice applications have to manage several systems at once: speech detection, transcription, reasoning, response generation, and audio playback. AICaller was built to explore that complete real-time loop while preserving a usable conversation record.

System design

• Pyannote detects voice activity and helps identify meaningful pauses.
• Faster Whisper produces live transcripts.
• GPT-4o mini runs within an AutoGen workflow so specialist agents can join when needed.
• ElevenLabs converts responses back into speech.

Engineering focus

The main challenge was conversational timing. Responding too early interrupts the speaker; waiting too long makes the system feel unresponsive. Voice-activity signals and incremental transcription are used to balance those two failure modes.

The orchestration layer separates conversational roles from the audio pipeline. That makes it possible to change the interview or assistance workflow without rebuilding speech processing.

Result

AICaller demonstrates a reusable architecture for browser and phone-based voice workflows. Conversations can be transcribed into structured records, allowing teams to review what happened and derive new insights without asking users to repeat the same discussion.