A new model for persian multi-part words edition based on statistical machine translation

• Main Authors: M. Zahedi, A. Arjomandzadeh
• Format: Article
• Language: English
• Published: Shahrood University of Technology 2016-01-01
• Series: Journal of Artificial Intelligence and Data Mining
• Subjects: Persian Multi-Part Words; Statistical Machine Translation; Fertility-based IBM Model; Syntax-Based Decoder; Spacing Rules
• Online Access: http://jad.shahroodut.ac.ir/article_495_4779dc10c5df9c7ba8cddf674d016abd.pdf

Multi-part words in English language are hyphenated and hyphen is used to separate different parts. Persian language consists of multi-part words as well. Based on Persian morphology, half-space character is needed to separate parts of multi-part words where in many cases people incorrectly use space character instead of half-space character. This common incorrectly use of space leads to some serious issues in Persian text processing and text readability. In order to cope with the issues, this work proposes a new model to correct spacing in multi-part words. The proposed method is based on statistical machine translation paradigm. In machine translation paradigm, text in source language is translated into a text in destination language on the basis of statistical models whose parameters are derived from the analysis of bilingual text corpora. The proposed method uses statistical machine translation techniques considering unedited multi-part words as a source language and the space-edited multi-part words as a destination language. The results show that the proposed method can edit and improve spacing correction process of Persian multi-part words with a statistically significant accuracy rate.