Multilingual embedding
An embedding model trained so that text meaning the same thing lands nearby regardless of language, letting a query in one language retrieve passages in another.
Last updated
An English-only embedding model puts an English sentence and its Bengali translation in unrelated parts of the space — it has no representation of Bengali at all. A multilingual model is trained so that the two land close together, which means a query in one language can retrieve passages in another.
For archival work this is the difference between searching your corpus and searching the part of it you happen to share a language with.
Illustrative passages
-
Bengali
এ বৎসর বর্ষা বিলম্বে আসিয়াছে; ধান্যের অবস্থা আশাপ্রদ নহে।
Gloss: This year the rains have come late; the state of the paddy gives no hope.
A Bengali weekly, 1896 · page 3
-
English
The late setting-in of the monsoon has occasioned grave anxiety for the rice crop in the eastern districts.
An English daily, 1896 · page 5
-
Urdu
اس سال بارش دیر سے ہوئی، دھان کی فصل کی حالت تسلی بخش نہیں۔
Gloss: The rain came late this year; the condition of the paddy crop is not reassuring.
An Urdu weekly, 1896 · page 2
A theme, found across languages
Late monsoon and the rice crop
title written by machine · rename it
3 passages · Bengali, English, Urdu
- A Bengali weekly, 1896 · page 3open on the page
- An English daily, 1896 · page 5open on the page
- An Urdu weekly, 1896 · page 2open on the page
These passages share no words — not even a script. They are grouped because they say the same thing.
Two cautions. First, cross-lingual similarity is usually weaker than same-language similarity, so a mixed-language result set can rank the same-language matches too highly. Second — and this is the one that bites — a model’s absolute similarity scores tell you very little. A model that rates a true cross-lingual match at 0.62 looks better than one rating it 0.48 until you notice the first also rates an unrelated sentence at 0.45. What matters is the gap between related and unrelated, not the height of either.