Indonesia is home to more than 1,300 ethnic groups and over 700 local languages, yet bias in large language models has barely been studied there. Most bias benchmarks are built for English, and the few culturally aware ones only test groups with widely shared stereotypes. That leaves out the smaller communities, from Papua to Borneo to East Nusa Tenggara, that are most exposed to invisibility bias.
IndoBias is a culturally grounded bias benchmark for Indonesian and three local languages: Javanese, Sundanese and Makassar. It measures bias along two complementary tracks. One asks how strongly a model favours well-known stereotypes. The other asks whether it treats hundreds of Indonesian groups even-handedly, including groups with no "popular" stereotype at all.
- 4,352
- contrastive statements
- 4
- languages
- 336
- demographic groups
- 7
- task formats
Two tracks: depth and breadth
A contrastive approach measures bias intensity precisely, but by design it can only cover groups with documented, agreed-upon tropes. A generation-based approach scales to far more groups but offers less controlled measurement. IndoBias combines both.
IndoBias-Pairs
544 manually curated pairs, each a prototypical statement and its counter-stereotypical twin, in four languages. A model is biased on a pair if it gives the stereotype lower perplexity.
- 5 domains, 18 sub-domains
- Validated by 5 native speakers (κ > 0.8)
- Translations refined by native speakers
IndoBias-QA
Generation prompts over 336 Indonesian groups, built on the LLM Stereotype Index and scored by Stereotype Polarity (SP).
- Ethnicity, institutions, names, parties, religions, universities
- Trait pairs from SPI, O*NET and WGI
- 7 task formats of rising complexity


| # | Task | What the model does |
|---|---|---|
| 1 | Simple Forced Choice | Choose the label that best describes a group. |
| 2 | Incentivized Choice | Choose a label in a promotion-related role-play. |
| 3 | Table Entry | Fill a dataset row with a group-label assignment. |
| 4 | Code Variable Assignment | Assign a label to a group-named variable in code. |
| 5 | Incentivized Dataset Entry | Fill a dataset row under an incentivized role-play. |
| 6 | Story Writing | Write a short story that implicitly reveals a label. |
| 7 | Code Function | Write and call a Python function with a hardcoded label. |
What we found
Decoders are more biased than encoders.Decoder models often exceed 70% protrope win rate on Indonesian prompts and sometimes approach 80%; Sailor2-8B-Chat reaches 79.6% on Identity and Demographics. Encoders rarely cross 67%, and many, such as mBERT and XLM-R-Base, stay within 10 points of the unbiased 50%.
Language changes the picture.Indonesian prompts produce the highest protrope win rates in most domains, while Javanese is usually less biased (Sailor2-8B-Chat: 79.6% vs 61.3%). Ideology and Religion is the exception: there, Makassar can be worse than Indonesian (Qwen3-8B: 73.5% vs 59.2%).
Improving local-language fluency can amplify stereotypical associations.
Local fine-tuning pushes models toward the stereotype.Every regionally adapted model we compared became more biased than its base: Komodo-7B over Llama-2-7B, SeaLLM-v3-7B over Qwen2-7B, and Sailor2-8B over Qwen2.5-7B. Sailor2-8B, the most heavily adapted, shows the largest rise, including +8.46 points on Sundanese.
Groups are not treated evenly.On IndoBias-QA, Stereotype Polarity varies widely across Indonesian groups. Korowai and Bajau score lower than Javanese and Sundanese on Basic Human Needs and Foundations of Wellbeing. Anonymized government institutions differ sharply on Control of Corruption and Government Effectiveness. One of three anonymized 2024 presidential candidate names scores much lower on Opportunity and Foundations of Wellbeing than the other two.



Newer models are less consistent.Newer models show a larger spread of Stereotype Polarity across groups. Qwen3.5-9B rises over Qwen3-8B on five of six demographics, and GPT-5 mini rises over GPT-4.1 mini on all six, for example +5.33 on Political Parties and +5.16 on Institutions. Their stereotypes are less stable and more context-dependent, rather than uniformly stronger.
Where bias comes from
To see how bias forms during training, we pretrained IndoBERT from random weights for 500,000 steps on six data mixes and evaluated IndoBias-Pairs at every checkpoint.
Web crawl introduces the most bias.Among the three Indonesian corpora, CC-100 ends with the highest protrope win rate. Human-reviewed text, encyclopedic articles from Wikipedia and news from Liputan6, ends noticeably lower.
Adding local languages increases bias.Mixing Javanese and Sundanese into Indonesian Wikipedia raises bias on every evaluated language. CC-100 mixes show the same trend, except on Indonesian itself, where the mix with the most local-language data ends lower.

Get the data
IndoBias is hosted as a gated dataset on Hugging Face. Fill in a short form and accept the research-use terms to get access immediately.
Citation
@misc{hanif2026indobias,
title = {IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages},
author = {Hanif, Ikhlasul Akmal and Azmi, Muhammad Falensi and Tjiaranata, Filbert Aurelian and Yulianrifat, Eryawan Presma and Koto, Fajri},
year = {2026},
eprint = {2606.01260},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2606.01260}
}