ResearchMay 2026arXiv 2606.01260

IndoBias A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages

Content warning. This page and the benchmark contain example statements that may be offensive, harmful or biased. They are used to measure bias, not to endorse it.

Indonesia is home to more than 1,300 ethnic groups and over 700 local languages, yet bias in large language models has barely been studied there. Most bias benchmarks are built for English, and the few culturally aware ones only test groups with widely shared stereotypes. That leaves out the smaller communities, from Papua to Borneo to East Nusa Tenggara, that are most exposed to invisibility bias.

IndoBias is a culturally grounded bias benchmark for Indonesian and three local languages: Javanese, Sundanese and Makassar. It measures bias along two complementary tracks. One asks how strongly a model favours well-known stereotypes. The other asks whether it treats hundreds of Indonesian groups even-handedly, including groups with no "popular" stereotype at all.

4,352
contrastive statements
4
languages
336
demographic groups
7
task formats
Same stereotype, different languages, different answers
(a) A model assigns lower perplexity to the prototypical statement in Indonesian, but prefers the counter-stereotypical one in Javanese. (b) Asked to pick a label, the model calls the Korowai "lawless" and the Sundanese "law-abiding".
Two examples of model bias: perplexity flipping between Indonesian and Javanese for the same stereotype pair, and a model labelling Korowai as lawless and Sundanese as law-abiding.

Two tracks: depth and breadth

A contrastive approach measures bias intensity precisely, but by design it can only cover groups with documented, agreed-upon tropes. A generation-based approach scales to far more groups but offers less controlled measurement. IndoBias combines both.

Depth

IndoBias-Pairs

544 manually curated pairs, each a prototypical statement and its counter-stereotypical twin, in four languages. A model is biased on a pair if it gives the stereotype lower perplexity.

  • 5 domains, 18 sub-domains
  • Validated by 5 native speakers (κ > 0.8)
  • Translations refined by native speakers
Breadth

IndoBias-QA

Generation prompts over 336 Indonesian groups, built on the LLM Stereotype Index and scored by Stereotype Polarity (SP).

  • Ethnicity, institutions, names, parties, religions, universities
  • Trait pairs from SPI, O*NET and WGI
  • 7 task formats of rising complexity
IndoBias-Pairs construction pipeline
Templates with a single varying filler, validated by native speakers, then expanded into Javanese, Sundanese and Makassar.
Pipeline for building IndoBias-Pairs: template curation, stereotype validation, and translation into three local languages.
IndoBias-QA construction pipeline
Demographic groups collected from Wikidata and verified by hand, combined with stereotype pairs and seven task templates.
Pipeline for building IndoBias-QA: demographic collection, stereotype pairs from SPI, O*NET and WGI, and task templates.
Seven tasks, ordered by complexity
Harder formats make it harder for a model to fall back on a generic refusal, so it has to commit to an answer.
#TaskWhat the model does
1Simple Forced ChoiceChoose the label that best describes a group.
2Incentivized ChoiceChoose a label in a promotion-related role-play.
3Table EntryFill a dataset row with a group-label assignment.
4Code Variable AssignmentAssign a label to a group-named variable in code.
5Incentivized Dataset EntryFill a dataset row under an incentivized role-play.
6Story WritingWrite a short story that implicitly reveals a label.
7Code FunctionWrite and call a Python function with a hardcoded label.

What we found

Decoders are more biased than encoders.Decoder models often exceed 70% protrope win rate on Indonesian prompts and sometimes approach 80%; Sailor2-8B-Chat reaches 79.6% on Identity and Demographics. Encoders rarely cross 67%, and many, such as mBERT and XLM-R-Base, stay within 10 points of the unbiased 50%.

Language changes the picture.Indonesian prompts produce the highest protrope win rates in most domains, while Javanese is usually less biased (Sailor2-8B-Chat: 79.6% vs 61.3%). Ideology and Religion is the exception: there, Makassar can be worse than Indonesian (Qwen3-8B: 73.5% vs 59.2%).

Improving local-language fluency can amplify stereotypical associations.

Local fine-tuning pushes models toward the stereotype.Every regionally adapted model we compared became more biased than its base: Komodo-7B over Llama-2-7B, SeaLLM-v3-7B over Qwen2-7B, and Sailor2-8B over Qwen2.5-7B. Sailor2-8B, the most heavily adapted, shows the largest rise, including +8.46 points on Sundanese.

Increase in protrope win rate after regional adaptation
Average change in percentage points across the four languages, regional model minus its base model.
Sailor2-8Bfrom Qwen2.5-7B+6.11
Komodo-7Bfrom Llama-2-7B+2.94
SeaLLM-v3-7Bfrom Qwen2-7B+1.98

Groups are not treated evenly.On IndoBias-QA, Stereotype Polarity varies widely across Indonesian groups. Korowai and Bajau score lower than Javanese and Sundanese on Basic Human Needs and Foundations of Wellbeing. Anonymized government institutions differ sharply on Control of Corruption and Government Effectiveness. One of three anonymized 2024 presidential candidate names scores much lower on Opportunity and Foundations of Wellbeing than the other two.

Stereotype Polarity of Qwen3-8B across Indonesian groups
Higher means more positive labels. Institutions and presidential candidate names are anonymized.
Radar chart of Stereotype Polarity scores across Indonesian ethnic groups.
Ethnicities
Radar chart of Stereotype Polarity scores across anonymized government institutions.
Government institutions
Radar chart of Stereotype Polarity scores across anonymized 2024 presidential candidate names.
Presidential candidate names

Newer models are less consistent.Newer models show a larger spread of Stereotype Polarity across groups. Qwen3.5-9B rises over Qwen3-8B on five of six demographics, and GPT-5 mini rises over GPT-4.1 mini on all six, for example +5.33 on Political Parties and +5.16 on Institutions. Their stereotypes are less stable and more context-dependent, rather than uniformly stronger.

Where bias comes from

To see how bias forms during training, we pretrained IndoBERT from random weights for 500,000 steps on six data mixes and evaluated IndoBias-Pairs at every checkpoint.

Web crawl introduces the most bias.Among the three Indonesian corpora, CC-100 ends with the highest protrope win rate. Human-reviewed text, encyclopedic articles from Wikipedia and news from Liputan6, ends noticeably lower.

Final-checkpoint bias on Indonesian, by pretraining corpus
Protrope win rate after 500,000 steps. 0.5 means no preference between the stereotype and its counterpart.
CC-100web crawl0.585
Wikipediaencyclopedic0.555
Liputan6news0.546
Bar length shows distance above 0.5, from 0.50 to 0.60.

Adding local languages increases bias.Mixing Javanese and Sundanese into Indonesian Wikipedia raises bias on every evaluated language. CC-100 mixes show the same trend, except on Indonesian itself, where the mix with the most local-language data ends lower.

Protrope win rate across pretraining
Six data compositions, evaluated on each language from 40% of training onward.
Line charts of protrope win rate over training progress for six pretraining data mixes, evaluated on Indonesian, Javanese, Makassar and Sundanese.

Get the data

IndoBias is hosted as a gated dataset on Hugging Face. Fill in a short form and accept the research-use terms to get access immediately.

Citation

@misc{hanif2026indobias,
  title         = {IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages},
  author        = {Hanif, Ikhlasul Akmal and Azmi, Muhammad Falensi and Tjiaranata, Filbert Aurelian and Yulianrifat, Eryawan Presma and Koto, Fajri},
  year          = {2026},
  eprint        = {2606.01260},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.01260}
}