IMPROVEMENT OF CHARACTER SET DETECTOR CHARDET

摘要

There are many encoding schemes which represent characters in text files. If the program displaying these texts cannot determine the right encoding, the text may become unreadable. Thanks to the widely spread universal charset detector from Netscape, it is possible to display text correctly in any software on any device. Language models for the automatic character set detection have been created only for a small group of languages. Our aim was to create language models for more countries so that the probability of successful determination of the encoding increased. The most problematic was the increase in accuracy of detecting the character set for languages using ISO-8859-1 encoding. The original algorithm was not sufficiently precise, and we have therefore designed a different procedure.