The functions of crawlers or bots are very important, as they are responsible for the page or web visibility ti google search engine.

In SEO theory, google records 3 activities crawling, indexing ot ranking. If the page is not crawled that means that inspite of being published google is not allowing people to search or read the website.