Telephone inquiry desks have long faced problems such as labor shortages, rising personnel costs, and inconsistent service quality. The traditional solution was IVR, an interactive voice response system operated with keypad tones, but the need to navigate fixed menus often damaged the customer experience. Conversational voice AI agents combining large language models with speech processing are now spreading rapidly.
Telephone service is undergoing a major transformation into systems that understand customers' natural speech on the spot and respond flexibly according to context.
From Cascades to Architectures That Handle Speech Directly
There are two broad approaches to building voice AI. One is the conventional cascade, which connects speech-to-text (STT), a language model (LLM), and text-to-speech (TTS) in sequence. It offers freedom to choose the best specialist provider for each stage, but latency accumulates with every API boundary, and total response time frequently exceeds one second. Once it passes 500 milliseconds, often treated as a benchmark for natural human conversation, customers begin to feel that they are 'talking to a machine.'
By contrast, the approach now becoming mainstream is a speech-to-speech (S2S) architecture that integrates recognition and synthesis within the model, receives audio directly, and returns audio directly. Because it avoids the heavy step of conversion to text, latency falls substantially. The model can also interpret information lost in writing, such as vocal tone and intonation, and reflect it in the response. This approach has made it realistic to break through the '500-millisecond barrier' that was previously commonplace.
The Latest Models Serving as the Brain
OpenAI and Google are leading this field.OpenAI's Realtime API became generally available in August 2025 and was designed around bidirectional streaming over WebSocket alongside models that handle audio directly.
Its strength is the ability to keep returning audio with low latency while carrying out complex processing with tool calls and allowing developers to specify the AI's persona and how speech segments should be detected.
Meanwhile, Google's Gemini Multimodal Live API accepts not only audio but also text, image, and video input. It dynamically changes the tone of responses to match a user's expression and offers real-time translation across more than seventy languages, making it effective in designs intended for global deployment.
Security considerations are also essential in commercial use, including the use of temporary tokens rather than ordinary API keys.
Twilio Connects the Telephone Network and AI
The role of connecting online AI to the telephone network is fulfilled by Twilio, the de facto standard. There are two main ways to build the connection.
One is a low-level method using Media Streams to exchange raw call audio bidirectionally. Building your own relay server allows detailed control. The same foundation can handle inbound calls as well as outbound calling initiated by the system.
The other is ConversationRelay, in which Twilio bundles speech recognition, connection to a language model, and speech synthesis. Developers are freed from detailed audio-data processing and can construct a conversation using only structured text exchanges. It is easy to support Japanese and other languages by fixing the recognition language or automatically identifying the language of response text for speech, allowing developers to focus on business logic.
Designing 'Natural Conversation' for Japanese
No matter how far the technology advances, if a telephone conversation does not feel natural, customers will abandon it midway and transfers to human representatives will increase. Japanese especially requires embedding language- and culture-specific conversational practices into the AI.
Backchannel responses come first. People do not listen to someone in complete silence; they form the rhythm of a conversation by signaling understanding and interest with short phrases such as 'yes' and 'I see.' If an AI silently listens to a long explanation, customers worry that it has frozen or failed to hear them properly. Instructing it to provide appropriate positive acknowledgments at breaks in the conversation creates reassurance that the customer is being heard attentively.
The next requirement is to keep latency from being felt. Database lookups described below inevitably create processing waits. To prevent these intervals from becoming awkward silence, insert bridging phrases such as 'Please hold for a moment' and 'Let me check that for you' to ease the perceived delay. Faster speech synthesis is another option.
Cartesia's Sonic uses a proprietary state-space model to generate speech with latency in the ninety-millisecond range and supports more than forty languages.
Interruption handling also matters. In human conversation, people often ask questions while the other party is still speaking. When a customer begins talking during the AI's response, 'barge-in' must immediately stop playback and switch to listening. This works only with echo cancellation that separates the system's output audio from the customer's voice and voice activity detection (VAD) that quickly identifies human speech. Voice AI supporting such interruptions has begun appearing in Japan as well.
Prompt design for voice differs in its key points from text chat. Concise, natural spoken language that is understandable after hearing it once is essential. In Japanese, painstaking adjustments such as adding katakana readings for proper nouns and technical terms in the prompt remain effective for preventing mispronunciation.
RAG and Function Calling That Get Work Done
Fluent speech alone does not resolve practical requests. Answering needs such as 'Tell me my balance' or 'I want to change my reservation' requires access to internal knowledge and dynamic data.
RAG, or retrieval-augmented generation, searches internal FAQs and manuals and reflects the results in an answer. When used directly with voice, however, search and generation processing stack up and tend to delay the response.
Latency can be reduced by caching similar past questions and answers so that close matches can be returned immediately without a fresh search or generation, and by detecting a boundary before the customer finishes speaking so that a lookup can begin in the background in advance.
Function calling, meanwhile, handles actions such as reservations and account operations. During a conversation, the AI itself decides when it needs to call an external API, passes the necessary information to execute the process, and continues the conversation after receiving the result.
Covering these background operations with bridging phrases lets customers receive service without being aware of the complex process.
Results from Adoption in Japanese Call Centers
These technologies have already begun producing results in Japan. At one support center, Sumitomo Life Insurance completes about 10% of incoming calls through a voice bot, easing the burden on representatives.
In addition to automating email routing, Bic Camera cut the time required for recordkeeping after calls by about half.
Kobayashi Pharmaceutical introduced a system that transcribes calls in real time and searches FAQs and manuals. It reduced handling time by roughly half and substantially cut after-call work as well.
Sumitomo Mitsui Trust TA Solution reduced work by about 9,200 hours per year through summaries of customer interactions and representative assistance. By using its accumulated service records, it also shortened training for each new employee by three business days.
The objectives extend beyond reducing personnel costs to limiting abandoned calls, providing round-the-clock service, and understanding customers through call data.
Handling Recordings and Personal Information
Once customer speech is transcribed, stored, summarized, and used for learning, a data-governance system is unavoidable. Audio and text obtained from calls often contain sensitive information, including names and contact details, and their handling is governed by Japan's Act on the Protection of Personal Information.
Organizations are expected to define clear internal operating rules for storage, encryption, reliable disposal of data that is no longer required, and responses to customer requests for disclosure or cessation of use, and to state the purpose of use when necessary. Selecting a platform compliant with major international security standards is an effective way to reduce these risks at the foundation.
Two Pillars of a Successful Introduction
Telephone automation has advanced from IVR that waits for keypad input to autonomous voice AI integrating language models and real-time communication. The key is to combine an S2S model with a robust communications foundation such as Twilio and cross the '500-millisecond barrier.'
In Japanese-speaking markets, this must be accompanied by detailed UX design including backchannel responses, interruption handling, and bridging phrases. Choosing a dedicated platform is also important when development speed matters.
For developers who prioritize flexible combinations, one option is Vapi; for teams that want to reach production quality quickly without code, there is Retell AI. When suitable technology choices are combined with an understanding of conversational culture in the Japanese market, voice AI will grow beyond a mere cost-reduction tool into infrastructure for the next generation of customer support.
