A Web Crawler is a program that crawls through the sites in the Web and find URL’s. Normally Search Engines uses a crawler to find URL’s on the Web. Google uses a crawler written in Python. There are some other search engines that uses different types of crawlers.
For Web crawling we have to perform following steps-
1.Firstly make url of page which we have to crawl.
2.Then we have to fetch link of that particular website.Following curl () function fetches link of website–
3.Function crawl is for crawling the website and get all links of the webpage.