Entropy-Based Approach for the Detection of Changes in Arabic Newspapers’ Content

• Main Authors: Olga Bernikova, Oleg Granichin, Dan Lemberg, Oleg Redkin, Zeev Volkovich
• Format: Article
• Language: English
• Published: MDPI AG 2020-04-01
• Series: Entropy
• Subjects: publishing model modeling; anomaly detection; word embedding
• Online Access: https://www.mdpi.com/1099-4300/22/4/441

A new method for the recognition of meaningful changes in social state based on transformations of the linguistic content in Arabic newspapers is suggested. The detected alterations of the linguistic material in Arabic newspapers play an indicator role. The currently proposed approach acts in an “online” fashion and uses pre-trained vector representations of Arabic words. After a pre-processing stage, the words in the issues’ texts are substituted by vectors obtained within a word embedding methodology. The approach typifies the consistent linguistic template by the similarity of the embedded vectors. A change in the distributions of the issue-grounded samples indicates a difference in the underlying newspaper template. A two-step procedure implements the concept, where the first step compares the similarity distribution of the current issue versus the union of ones corresponding to several of its predecessors. A repeating under-sampling approach accompanied by a two-sample test stabilizes the sampling and returns a collection of the resultant <i>p</i>-values. In the second stage, the entropy of these sets is sequentially calculated, such that the change points of the time series obtained in this way indicate the changes in the newspaper content. Numerical experiments provided on the following issues of several Arabic newspapers published in the Arab Spring period demonstrate the high reliability of the method.