TL;DR
- Same text task, several languages: that’s multilingual data annotation. Labels can travel between languages. Guidelines, reviewers and quality checks can’t.
- Chinese, Japanese, Thai, Lao: no spaces between words. Vietnamese has them, but only between syllables. Someone has to say where a span starts, in writing.
- One agreement score per language, same statistic each time. Averaging hides the weak one.
- Pilot each language before you scale. Cheapest way to find gaps in a guideline.
Labelling English text is hard enough. Hand the task to a Thai team instead and it’s a different job. Same for Chinese or Japanese. Word boundaries land in other places. Scripts behave differently. Some edge cases from English don’t exist in the new language, and others turn up looking nothing like the original.
That’s the real problem with multilingual data annotation for Asian languages. The work isn’t harder in some vague way. It’s less portable. Copy an English guideline across and it fails quietly. Nobody notices until agreement scores drop. Or until a model starts mislabelling one language far more than the others.
Here’s what’s covered: text annotation across languages, with a guideline per language, reviewer pools, quality scored language by language, and a pilot. Recording or collecting speech is out of scope, and our speech dataset collection case covers that side. Coming in cold? Start with what data annotation is. There’s a Malaysia angle in here too, since Jawi, Rumi and Manglish all end up on an annotator’s desk.
What is multilingual data annotation?
Multilingual data annotation is the labelling of text in more than one language so a model can learn each of them. The label set is usually shared, for example the same entity classes or intent categories in every language. What sits around the labels isn’t shared. Every language gets its own guideline section, its own native-reading reviewers and its own quality score. Skip that and you get the classic mistake. Take one English guideline stretched over Chinese, Japanese and Thai. The labels look consistent on paper. Where do words and entities begin and end? They can’t agree. A fair test: report agreement per language, then explain every gap between them. Can you? As of 2026 that means text only. Think entity tagging, intent classification and sentiment, not speech, audio or image labelling. Those are separate jobs.
So how much should the languages share? That’s the question every project hits early. Share too little and you’ve got several unrelated projects whose results you can’t compare. Share too much and one language’s logic gets forced onto another.
Universal Dependencies shows the usual compromise. Its aim is cross-linguistically consistent treebanks. It covers nearly 200 human languages, with treebanks and guidelines of varying size and quality. Categories and guidelines are universal. Extensions for a specific language come in when necessary, and each language team keeps its own guidelines (Universal Dependencies). The same split works in a smaller commercial project, and it’s the structure the rest of this guide assumes.
Why can’t one guideline cover every language?
One guideline can’t stretch to every language because languages mark words, scripts and grammar in different ways. The clearest case is the word itself. Chinese and Japanese don’t put spaces between words, and neither do Thai or Lao. So an annotator has to decide where a word or an entity starts and ends. That call changes the label. Write it down.
Wikipedia’s article on text segmentation lays this out. In Chinese and Japanese, sentences are delimited but words aren’t. In Thai and Lao, phrases and sentences are delimited but words aren’t. It also shows a Chinese string with two valid splits and two meanings: “The US will not agree” or “The US Congress does not agree” (Wikipedia, Text segmentation). Same characters, different entity.
Vietnamese looks easier and isn’t. It uses spaces, but per Wikipedia’s article on word dividers, virtually all syllables are separated by spaces whether or not they form word boundaries (Wikipedia, Word divider). A space tells you where a syllable ends. It doesn’t tell you where a word does. An annotator who treats every space as a boundary will split compound words, and a second annotator who doesn’t will disagree with them all day.
Why does this matter for annotation? Take span labelling, such as person names or product terms. If the guideline doesn’t say how to treat boundaries in languages without clear word spaces, two annotators draw two different spans. Agreement drops, and skill has nothing to do with it. The guideline was the problem.
How does script change the work?
Script changes the work because the same language can arrive in more than one writing system, and a labelled dataset has to say which one it holds. Per the W3C, a script subtag belongs on a language tag when it adds useful information. Russian transcribed into Latin script (ru-Latn) is the W3C’s own example (W3C, language tags). From the start, tag every item with its language and script. Filtering, sampling and per-language reporting stay possible later.
Malaysia gives a good local example. Malay is written today mostly in Latin letters, known as Rumi, but Jawi is still around. Jawi is based on the Arabic script, with 31 original Arabic letters plus extra letters built to fit sounds native to Malay. Until the 20th century it was the standard script of the Malay language. Today it’s one of two official scripts in Brunei, and in Malaysia its position is protected under Section 9 of the National Language Act 1963/67, as it retains a degree of official use in religious and cultural contexts (Wikipedia, Jawi script).
What does that mean for a project? If your Malay data could include Jawi, decide up front whether it’s in scope, and tag it. If romanised text is in scope for any language, say so in the guideline. An item that’s out of scope should still have a rule, even if the rule is “skip and flag”. Leave it unspecified and each annotator invents their own answer.
What goes into a guideline for each language?
Start with one shared core guideline. Then add a short section for each language covering boundary rules, script handling, mixed-language text and worked examples. Write those examples in the language itself. They matter more than translated rules, so budget time for them in the plan.
Here’s the structure to give each language section:
| Section | What it settles | A question it answers |
|---|---|---|
| Boundaries | Where a labelled span starts and ends | With no spaces in the text, who decides what counts as a word? |
| Script and spelling | Which script you expect, and how transliteration is handled | Is romanised text in scope, or out? |
| Language and script tag | How each item is tagged | Is the tag recorded per item or per file? |
| Edge cases | Items that fit no label, or two labels | What happens with names, loanwords and abbreviations? |
| Worked examples | A real item, its correct label and the reason | Does every label have a clear case and a hard case? |
| Mixed-language text | One sentence holding two languages | Is the item labelled once, or split? |
Keep the shared core short. Every annotator reads a few pages. Each language team also reads its own section, and nobody else does. That’s easier to maintain than one long document. When a decision changes, update the core or the language section, record the date, and tell the team. A guideline nobody can find the latest version of is worse than a short one.
Planning a multilingual project and not sure how many language sections you’ll need? A scoping conversation settles that quickly. You can talk through your languages and task types with the SummitNext team before you commit to a schema.
How should you build reviewer pools for each language?
Each language gets its own reviewer pool, filled with native-level readers (the kind of people who’d spot a shift in meaning that a translation flattened), plus a backup or two so one sick day doesn’t stall the language.
Reviewers should work in the original language, not from a translation. Give each language a senior reviewer who settles disputed items and writes the ruling into that language’s section the same day. And calibrate twice, once with every language in the room so the shared labels mean the same thing everywhere, then once per language for the local rules.
Does translation cover it? Not for original text. Take MASSIVE: it began as SLURP, an English set, then professional translators localized it into 50 more languages, 51 in total (arXiv, MASSIVE), which suits some tasks. A post written in Thai has no English twin, so Thai readers do the labelling and the review. Where your data came from matters too, since translated and native sets behave differently and blending them quietly muddies every score you report.
How do you measure quality for each language?
Use one statistic for every language and report the scores side by side, never as an average, which can look fine while one language falls apart. That gap is the whole point. When it shows up, ask why (often it’s a fuzzy boundary rule).
Per Wikipedia, inter-rater reliability is the degree of agreement among independent observers who rate, code or assess the same phenomenon, and the article lists Cohen’s kappa, Fleiss’ kappa and Krippendorff’s alpha among the options (Wikipedia, Inter-rater reliability). Whichever you pick, use the same one for every language. For the wider process, see how annotation quality is measured.
Here’s a made-up example, not client data: three languages, 100 double-labelled items each, and raw percent agreement of 82%, 80% and 58%. The average is 73%, which sounds fine and isn’t. Open the 42 items where language three’s annotators disagreed. Is a boundary rule missing? Were the guideline’s examples translated from English, so they look nothing like real text? Whatever turns up, the fix goes in that language’s section, and nobody needs retraining.
Each batch, per language, double-label a sample and score it alone. Then walk the disagreements with a senior reviewer, calling each a guideline gap or an annotator slip, and write the decision down. If a language’s score keeps sliding across batches, that team needs recalibrating.
Low agreement doesn’t always mean weak annotators. Check the guideline first.
What about text that mixes languages?
Decide early whether mixed-language text is in scope, and write it down. Code-switching, where people shift from one language or dialect to another depending on the social context, shows up wherever data gets informal (Wikipedia, Code-switching). Posts. Chats. Emails.
Malaysia’s the obvious case. Manglish is an informal form of Malaysian English, heavily influenced by Malay, Tamil and varieties of Chinese, and not one of the country’s official languages (Wikipedia, Manglish). One message can mix English, Malay and a variety of Chinese. So which language is the item? Good luck deciding mid-batch.
To label mixed-language text, first decide whether it is in scope, then pick one rule and apply it to every item. Two rules are common. One labels the whole item once and gives it a mixed-language tag, which keeps the item intact and suits tasks like intent classification. The other splits the item and labels each language span on its own, which suits entity tagging, where a name may sit inside a sentence in another language. Either works when every annotator applies it the same way. Neither works when annotators choose for themselves. So write the rule into the guideline, attach a worked example from real text, and ask the senior reviewer for each language whether the rule fits how that language is written. This applies to informal text such as social posts, chat and email, as of 2026.
Pick a rule. Add an example to the guideline. Most teams skip that part.
Where do multilingual annotation projects go wrong?
Mostly in places you can spot before labelling begins. An English guideline gets translated, so it keeps edge cases that don’t exist in the new language and misses the ones that do. Agreement gets averaged, and one weak language vanishes into the mean. Reviewers work from translations, so labels that read fine in English and wrong in the original get approved. Boundaries are left to annotators, and everyone draws their own spans. Script and tags are forgotten, and months on nobody can filter by script. Mixed-language text gets dismissed as noise, yet real users switch languages all the time, so a model trained on tidy single-language data meets a surprise in production.
No special tooling fixes any of it. A written decision, made before labelling starts, does.
What should a pilot batch test?
The guideline, not the annotators. Run a small batch per language, double-label every item, then read the disagreements. You’re hunting for gaps in the instructions, and a pilot is the cheapest place to find them.
Stack it with hard cases: names, loanwords, mixed-language sentences, items in more than one script. Easy items prove nothing. Ask each team what confused them, turn the answers into examples in the language section, and rerun a smaller batch.
Agree what passing means before results land. A target score per language, or a rule that every disagreement gets traced to a cause. Either works if it’s set early, so nobody argues about the number afterwards. If one language fails, hold it back. The rest can go.
How should you evaluate a provider for multilingual annotation?
Don’t count the languages on a provider’s list. Ask how each one is run. A long list tells you about the sales deck. These questions tell you about the work:
- Who writes and maintains the guideline section for each language, and who signs off changes?
- Do reviewers read the language natively, and is there a senior reviewer for each one?
- Is agreement reported per language, with the same statistic across all of them?
- How are boundaries, script and mixed-language text handled in the guideline?
- What does a pilot look like, and what happens if one language fails it?
- Who owns the schema and the final dataset?
Doing it yourself is a real option too. If you’re weighing it, in-house versus outsourced operations covers the cost and risk trade-offs in general terms, and for text annotation specifically, our page on AI data annotation outsourcing for US AI and SaaS companies explains how that setup is run.
How does SummitNext approach multilingual text annotation?
SummitNext runs data annotation as a scoped service. Pilot batch first. Prove quality on your schema. Scale once accuracy sits where you need it. There’s no minimum commitment, and responsibility splits cleanly: SummitNext recruits, trains and manages the annotation team and its quality, while you own the schema, the standards and the final dataset. Delivery runs from centres in India and the Philippines. All of that comes from our data annotation services page, and the wider offer sits under AI data preparation.
For multilingual text work, tell us which languages and task types you have in mind. Entity labelling in one language and intent classification across five are different projects, so the scope follows from that. Client results from SummitNext partnerships show how engagements run, and the artificial intelligence page shows where annotation fits.
FAQ
What is multilingual data annotation?
Labelling text in several languages so a model can learn each one. That’s multilingual data annotation. Labels are often shared, while guidelines, reviewers and quality checks are set per language. Word boundaries, scripts and edge cases differ from one language to the next, so a single guideline rarely stretches across all of them.
Why is annotating Asian languages different from English?
Because several Asian languages, Chinese, Japanese, Thai and Lao among them, don’t put spaces between words. Annotators have to decide where a word or span begins and ends, and scripts and transliteration add more choices. A guideline written for English leaves all of that open, so annotators fill the gaps differently and agreement drops.
Do I need a separate guideline for each language?
In a way, yes: a shared core plus one short section per language. The core defines the labels. The language sections cover boundaries, script, edge cases and examples written in that language. Universal Dependencies follows a similar pattern, with shared categories and language-specific extensions added when necessary.
How do you measure annotation quality across languages?
Choose one agreement statistic, Krippendorff’s alpha or Cohen’s kappa for instance, and run it on every language. Report each score on its own, because a blended average can hide a weak language. Then review the disagreements with a senior reviewer who reads that language and update the guideline so the gap doesn’t return.
Is translating the data enough for multilingual annotation?
Sometimes. MASSIVE was built by localizing an English set into other languages with professional translators. Text originally written in each language can’t be handled that way, since there’s no English original to localize. You need annotators and reviewers who work in each language directly, using guidelines with real examples from it.
How should mixed-language text be labelled?
Pick one rule and apply it to every item. Either label the whole item once with a mixed-language tag, or split it and label each language span separately. Both work if all annotators do the same thing. Write the rule into the guideline with a worked example from real text, and review it with each language’s senior reviewer.
Should every item be tagged with its script?
Yes, from the start. The W3C advises a script subtag when it carries useful information, such as Russian written in Latin letters. For Malay, that means telling Rumi and Jawi apart. Tagging early keeps filtering, sampling and per-language reporting possible later.
Conclusion
Multilingual annotation fails when one language’s assumptions get applied to another. The fix is a short shared core, a guideline section per language, reviewers who read the language natively, and one agreement score per language instead of an average. Decide early how to handle boundaries, script and mixed-language text, and write each decision down. Then pilot, read the disagreements, fix the guideline, and only then scale. It’s slower at the start. By the third language it’s much cheaper, because every gap you close once stays closed.
Planning a multilingual annotation project? Book a consultation with SummitNext and tell us your languages, task types and timeline.
