A study of different web-crawler behaviour

A study of different web-crawler behaviour

The article deals with a study of web-crawler behaviour on different websites. A classification of web-robots, information gathering tools and their detection methods are provided. Well-known scrapers and their behaviour are analyzed on the base of large web-server log set. Experimental results demo...

Full description

Bibliographic Details
Main Authors: Alexander Menshchikov, Antonina Komarova, Yurij Gatchin, Anatoly Korobeynikov, Nina Tishukova
Format: Article
Language:English
Published: FRUCT 2017-04-01
Series:Proceedings of the XXth Conference of Open Innovations Association FRUCT
Subjects:
web robots
parsing
information gathering
web robot detection
information security
data protection
Online Access:https://fruct.org/publications/fruct20/files/Men.pdf

Internet

https://fruct.org/publications/fruct20/files/Men.pdf

Similar Items