Can AI Read Between the Lines? What Hesitation Sounds Like in Hindi and English, Mixed
Insights / Can AI Read Between the Lines? What Hesitation Sounds Like in Hindi and English, Mixed
Table of Contents
A customer calls support and says, “Mera order abhi tak deliver nahi hua, matlab… can you please check?” A human agent barely notices the switch from Hindi to English or the brief pause at “matlab.” It’s simply how many Indians speak. For a voice AI, however, that same moment can be harder to interpret, particularly when systems have been trained mainly on English-language conversations.
The interesting part is that the hesitation and the language switch are happening together. That’s not an unusual edge case in India—it is part of everyday conversation. Yet much of the research on AI hesitation has focused on English-only speech, leaving a gap in understanding what those pauses mean when people naturally move between Hindi and English.
Pauses Were Studied. Mostly in One Language.
Research into pauses and hesitation in conversation goes back decades, including the well-known work on conversational turn-taking by Sacks, Schegloff and Jefferson. But much of that research focused on English-language conversations.
That creates a problem when the same assumptions are applied to Indian speech. In a call centre in Bengaluru, a WhatsApp voice note from Delhi, or a support call in Mumbai, Hindi and English can appear in the same sentence without the speaker consciously switching between two languages. Research on Indian English and code-switched speech suggests that existing models often perform poorly in these situations, particularly because hesitation can increase around the point where the speaker changes language.
The Switch Point Is Where the Signal Actually Is
Research on Hindi-English conversations shows that the English parts of a mixed sentence are often spoken more slowly, with more vocal effort and greater changes in pitch than the Hindi around them. In other words, the point where someone switches languages can sound different from an ordinary pause.
That matters because many voice systems mainly look at silence to decide whether someone has finished speaking. A pause beyond a certain length can be treated as the end of a turn, prompting the system to respond or interrupt.
But in a sentence like “matlab… can you please check?”, the pause is part of the speaker’s thought. The customer hasn’t finished; they are simply switching languages while working out what to say next.
For an AI voice system, recognising that difference is important. It needs to understand the conversation around the pause, not treat the pause itself as the signal to respond.
Why This Isn't a Niche Problem in India Specifically
Roughly 600 million people in India speak Hindi, and many also use English professionally and socially. Switching between the two can happen naturally, even within the same sentence. It isn’t usually a conscious decision to move from one language to another.
That makes this more than a technical edge case for voice AI. If an assistant responds to a language switch with a generic “Sorry, I didn’t understand that. Could you repeat?”, it isn’t handling an unusual situation. It’s failing to understand how a large number of Indian conversations actually happen.
The challenge also looks different depending on the channel:
- Live calls: interrupting at the point of a language switch can make the system appear impatient or simply unaware of what the customer is saying.
- WhatsApp voice notes: there is no real-time turn-taking pressure, but the hesitation still carries meaning. The research on interpreting it in this context is still limited.
- Current research: most work on Hindi-English code-switching has focused on transcription and identifying which language is being spoken—not on understanding what a hesitation at the switch point might be telling the system.
That’s why the issue matters for AI being built for India. The challenge isn’t just recognising two languages. It’s understanding how people naturally use them together.
How Conversational AI Is Adapting — At Least on Intent
Intent detection is one area where AI has made meaningful progress with code-switched speech, even though interpreting hesitation at the point of a language switch remains much less developed.
Older systems often handled mixed-language speech in stages: identify the language, split the input, and send each part to a language-specific model. That works reasonably well when someone speaks one language at a time. It becomes much harder when a customer switches languages mid-sentence—the exact point where the system needs to understand the whole thought rather than separate fragments.
Newer systems can interpret intent directly from mixed-language input instead of first translating it into one “clean” language. Models such as MuRIL, developed by Google for Indian languages and code-mixed text, are designed to understand these mixed sentences as a single utterance. Combined with modern speech recognition, this approach is currently one of the stronger ways to handle Hinglish and Indian English disfluencies, according to the research reviewed earlier.
The practical difference is simple: the customer doesn’t have to finish their thought in one language for the system to understand what they need.
This is a more developed problem than interpreting hesitation itself, but it is an important step towards voice AI that can handle Indian conversations naturally.
Where This Leaves Products Being Built for This Market
Worktual‘s Lola is built to handle Hindi-English conversation as it happens in everyday use, including understanding intent when a customer switches languages mid-sentence. That’s a more specific claim than saying the wider research problem has been solved.
Understanding a hesitation at the exact point of a language switch is more complex. It means recognising not only that the speaker changed languages, but also what the pause, change in pace or vocal effort might be signalling. This remains an emerging area of research—for Lola and for the wider industry.
The research base here is still much smaller than the decades of work on English-language conversation. It’s worth being clear about that distinction: handling Hindi-English conversations well is one challenge; understanding the deeper meaning of hesitation at a code-switch point is another.
What Closing the Gap Actually Requires
There is research beginning to close this gap. A Hindi-English code-switching speech corpus, developed with Government of India funding, gives researchers real conversational data to work with rather than relying on translated or simulated speech.
There is also evidence that people notice the difference when AI handles mixed-language conversation well. A Microsoft Research study found that multilingual users preferred chatbots that could naturally code-mix, rating them as more natural and capable than monolingual alternatives.
The direction is encouraging, but the research is still catching up with how people actually speak.
Conclusion
Indian conversations don’t fit neatly into one language at a time. Hindi-English code-switching, pauses and changes in speaking style are part of how people communicate naturally, yet much of the research behind conversational AI has been built around English-only speech.
The next step isn’t simply teaching AI more languages. It is teaching systems to understand how people use those languages together including the hesitation, emphasis and changes in delivery that can carry meaning.
For businesses building voice AI for India, that distinction matters. Recognising the words is one part of the problem. Understanding what the speaker means, especially at the point where language and hesitation overlap, is where the field still has room to develop.
Frequently Asked Questions
1. Why do people switch between Hindi and English while speaking?
Hindi-English code-switching is a natural part of everyday communication for many Indians. Speakers can move between the two languages within the same sentence without consciously treating them as separate conversations.
2. Why is hesitation difficult for AI to interpret?
A pause does not always mean that a speaker has finished talking. It can indicate that someone is thinking, searching for a word, changing direction or switching languages. Voice systems that rely mainly on pause duration can therefore respond at the wrong moment.
3. Does code-switching affect voice AI performance?
It can. Research on Indian English and code-switched speech suggests that models trained mainly on American or English-only datasets may perform poorly when applied to Indian speech.
4. What happens when hesitation occurs during a language switch?
The switch itself can be accompanied by changes in speaking speed, vocal effort and pitch. This makes the moment different from an ordinary pause and potentially useful as a signal for AI systems trying to understand the speaker’s intent.
5. Can Lola understand Hindi-English conversations?
Lola is designed to handle Hindi-English conversations as they naturally occur, including understanding intent when a customer switches languages mid-sentence. Interpreting the deeper meaning of hesitation specifically at a code-switch point remains an emerging research area.
6. How does AI detect intent in Hindi-English speech?
Newer systems can analyse a mixed-language utterance as a whole rather than identifying each language first and routing the fragments to separate models. Models such as MuRIL, developed for Indian languages and code-mixed text, are one example of this approach.
7. Is research on Hindi-English hesitation already well established?
No. There is growing research and dedicated speech data, but much of the existing work has focused on transcription and language identification rather than understanding what hesitation at a language-switch point communicates.
Related Posts
Does Your Business Need to Register as a DPDP Consent Manager?
Financial institutions operate in a highly competitive, regulation-driven, and margin-sensitive environment where revenue growth is no longer driven by acquisition alone. Banks, fintechs, and Non-Banking Financial Companies (NBFCs) continue to invest heavily in digital acquisition, yet face persistent challenges such as low onboarding completion, rising contact centre costs, increasing churn, and fragmented customer data. At the same time, customers expect instant responses, seamless onboarding, personalised financial guidance, and consistent service across digital and assisted channels. When these expectations are not met, conversion drops, churn increases, and cost-to-serve rises, often before the impact is fully visible in performance metrics.
Autonomous Brand Responsiveness: What It Actually Means for Indian Marketing Leaders
The Indian B2B landscape is shifting fast. Customer expectations are rising, sales cycles are getting longer, and the volume of customer data most businesses generate has outpaced what traditional spreadsheets or legacy CRM systems can handle. AI CRM software was built precisely for this inflection point.
AI Contact Centres for Indian EdTech: Scaling Student Support Without Losing the Human Touch
Financial institutions operate in a highly competitive, regulation-driven, and margin-sensitive environment where revenue growth is no longer driven by acquisition alone. Banks, fintechs, and Non-Banking Financial Companies (NBFCs) continue to invest heavily in digital acquisition, yet face persistent challenges such as low onboarding completion, rising contact centre costs, increasing churn, and fragmented customer data. At the same time, customers expect instant responses, seamless onboarding, personalised financial guidance, and consistent service across digital and assisted channels. When these expectations are not met, conversion drops, churn increases, and cost-to-serve rises, often before the impact is fully visible in performance metrics.
