A Close-to-linear Topic Detection Algorithm using Relative Entropy based Relevance Model and Inverted Indices Retrieval

• Main Authors: Steve Kansheng Shi, Lemin Li
• Format: Article
• Language: English
• Published: Atlantis Press 2012-08-01
• Series: International Journal of Computational Intelligence Systems
• Subjects: Topic Detection; Link Topic Detection; Retrospective Event Detection; Information Retrieval; Relevance Models; Inverted Indices
• Online Access: https://www.atlantis-press.com/article/25868005.pdf

Although timely access to information is becoming increasingly important and gaining such access is no longer a problem, the capacity for humans to assimilate such huge amounts of information is limited. Topic Detection(TD) is then a promising research area that addresses speedy access of desired information. However, ironically, the time complexity of existing TD algorithms themselves is usually up to the -th power of . Linear performance requirement of real world topic detection has not been significantly addressed. This paper reveals a new patented topic detection algorithm called that combines elevance odel with nformation etrieval technique to improve on time efficiency. Relevance Model(RM) is a theoretical extension of statistical language modeling that was developed for the task of document retrieval. To reduce the costs of fetching RM, we reduce the number of comparisons for stories by a query-based approach that makes similar stories exist in the top-k query results. We also build our query based on inverted indices, which have the complexity close to linear. The time cost of rest of operations in the topic detection process is a constant. Hence, the total complexity of topic detection algorithm should be close to linear as shown in experimental results. In addition, also gains better detection rates and robustness by relative entropy based topic model design