An Estimation of the Entropy of Chinese

• Main Authors: Yuh-Juh Lin, 林玉柱
• Other Authors: Jyun-Sheng Chang
• Format: Others
• Language: en_US
• Published: 1994
• Online Access: http://ndltd.ncl.edu.tw/handle/82305916772416048578

碩士 === 國立清華大學 === 資訊科學學系 === 82 === There are three tasks in this thesis: the first is to develop a new approach to building class-based n-gram models,the second is to estimate the entropy of Chinese, and the last is to ana- lyze some bottlenecks in Chinese processing. We accomplish these three closely related tasks by estimating the cross- entropy of Chinese and our language model, and then analyse the bottle- necks in Chinese processing from the results. The cross- entropy of Chinese and our language model is 12.66 bits per word or 3.88 bits per byte, which is better than IWCB by 0.6 bit per word. At last, we diagnose the bottlenecks in Chinese processing as Name and Unknown classes in that they have enormous perplexities in our model, which seem hard to be much improved.