Language Identification At Scale. Building a LID pipeline on 1TB for less than $30.
Introduction Language Identification is an important part of preprocessing data for pretraining or finetuning language models. We typically want to filter out data that includes languages we don’t want to train on, or quantify precisely how much data we have on certain languages. Pretraining datasets tend to be multi-terabyte scale data typically derived from crawling web pages, books, news, and magazines etc. However this data must be cleaned for multiple reasons. Model performance is extremely dependent on the quality of data, the classic machine learning anecdotes still hold here. Ethical concerns such as alignment and model safety are also big concerns that underline the preprocessing stage. ...