Focused crawlers aim to search only the subset of the web related to a specific topic, and offer a potential solution to the problem. The major problem is how to retrieve the maximal set of relevant and quality pages. In this paper, We propose an architecture that concentrates more over page selection policy and page revisit policy The three-step algorithm for page refreshment serves the purpose. The first layer contributes to decision of page relevance using two methods. The second layer checks for whether the structure of a web page has been changed or not, the text content has been altered or whether an image is changed. Also a minor variation to the method of prioritizing URLs on the basis of forward link count has been discussed to accommodate the purpose of frequency of update. And finally, the third layer helps to update the URL repository.

1. Bidoki, Yazdani et el, “FICA: A fast intelligent crawling algorithm”, Web Intelligence, IEEE/ACM/WIC International conference on Intelligent agent technology, Pages 635-641, 2007.
2. Cui Xiaoqing Yan Chun,” An evolutionary relevance calculation measure in topic crawler ” CCCM 2009, ISECS International Colloquium on Computing, Communication, Control, and Management, 267 – 270, aug 2009
3. Junghoo Cho, Hector Garcia-Molina, Lawrence Page, |Efficient crawling through URL ordering”, 7th International WWW Conference , April 14-18, Brisbane, 1998.
4. Mukhopadhyay et al, “A New Approach to Design Domain Specific Ontology Based Web Crawler”, ICIT 2007, 10th International Conference on Information Technology, 289 - 291, Dec. 2007
5. Peisu, Ke et el, “A Framework of deep web crawler”, 27th Chinese Proceedings of the 27th Chinese Control Conference, Pages 582-586, July 16-18, 2008.
6. Yadav, Sharma et el, “Architecture for parallel crawling and algorithm for change detection in web pages”, 10th International Conference on Information Technology, Pages 258-264, ICIT 2007.
7. Yuan, Yin et el, “Improvement of pagerank for focused crawler”, 8th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing, Pages 797-802, SNPD 2007.
8. Zheng, Chen, “HAWK: a Focused crawler with content and link analysis”, E-business engineering, 2008, ICEBE’08, IEEE international conference, pages 677-680, Oct 2008.
9. Zheng, Zhaou ET el, “URL Rule based focused crawler”, E-business engineering, ICEBE’08, IEEE international conference, Oct 2008, pages 147-154, 2008.

Focused crawler page change detection crawler policies crawler database