A Method for Filtering Pages by Similarity Degree based on Dynamic Programming

• Main Authors: Ziyun Deng, Tingqin He
• Format: Article
• Language: English
• Published: MDPI AG 2018-12-01
• Series: Future Internet
• Subjects: method for filtering pages; similarity degree; dynamic programming; combination method
• Online Access: https://www.mdpi.com/1999-5903/10/12/124

To obtain the target webpages from many webpages, we proposed a Method for Filtering Pages by Similarity Degree based on Dynamic Programming (MFPSDDP). The method needs to use one of three same relationships proposed between two nodes, so we give the definition of the three same relationships. The biggest innovation of MFPSDDP is that it does not need to know the structures of webpages in advance. First, we address the design ideas with queue and double threads. Then, a dynamic programming algorithm for calculating the length of the longest common subsequence and a formula for calculating similarity are proposed. Further, for obtaining detailed information webpages from 200,000 webpages downloaded from the famous website &#8220;www.jd.com&#8222;, we choose the same relationship Completely Same Relationship (CSR) and set the similarity threshold to 0.2. The Recall Ratio (RR) of MFPSDDP is in the middle in the four filtering methods compared. When the number of webpages filtered is nearly 200,000, the <i>PR</i> of MFPSDDP is highest in the four filtering methods compared, which can reach 85.1%. The <i>PR</i> of MFPSDDP is 13.3 percentage points higher than the <i>PR</i> of a Method for Filtering Pages by Containing Strings (MFPCS).