Implementation of Web Scraper Bot: Web Harvesting

Abstract: Web Harvesting or Web-Scraping also called web data extraction, are various methods of collecting information from across the internet. It is essentially a form of data mining. Programs are written to mine the data and to convert it into a meaningful and useful structure. In this paper we demonstrate a code/program written to harvest web-data from a particular web-site and to display the same in different file formats. The university results of a class of Students arescraped from the web and are being stored and calculated. This code can be reused several number of times and may also be altered to suit the desired/intended application. Creating a customized score sheet of all students in the college or university is a tedious task. In this paper a web scraper bot is employed to do the same within minutes.

Keywords: Web Harvesting or Web-Scraping, essentially a form of data mining, reused several number of times, Creating a customized score sheet, web scraper bot, regular expression (regex).