Smart Crawler for Hidden Web Interfaces

Abstract: Deep web is growing day by day. There has been increase in the techniques to locate deep web interfaces efficiently with the help of deep web interfaces. To locate deep web resources is being very difficult due to large volume of web resources and dynamic nature of deep web. The biggest challenge is to achieve wide coverage and high efficiency. We propose a two stage framework such as smart crawler for hidden web interfaces to efficiently gather deep web. There are two stages namely Site locating and Insite exploring. In Site locating the centre pages are located with the help of search engine. In second stage Insite exploration, the wide coverage of the sites is obtained and the relevant links are formed. The page is fetched to obtain form. The representative set of domain shows accuracy, and efficiency than other crawlers.

Keywords: selection, Deep web interfaces, Two stage crawler, Page ranking.