With billions of pages to crawl daily, it makes no sense for Google to go over a page that hasn’t changed. This is where the If Modified Since header comes in.
What is “If Modified Since”?
If Modified Since is a HTTP header that is sent to a server as a conditional request. This feature prompts your web server to tell Googlebot if your content has changed since it last crawled your site. Supporting the If Modified Since HTTP header saves you both bandwidth and overhead.
What does If Modified Since do?
Essentially, If Modified Since tells Googlebot whether or not it needs to check a page again. If a page hasn’t changed, it will tell Googlebot not to bother re-downloading as it already has an up to date copy on the index. However, if it has changed it will tell Googlebot to download again as there is new information.
The first time Googlebot visits your page it will see a 200 status code which means the content loaded fine. Googlebot notes when it accessed your page and the next time it returns to this page, one of two things will happen:
304 status code – If the content has not changed since Googlebot last visited, it will receive a 304 status code which lets it know that the page does not need to be accessed or re-downloaded again.
200 status code – This indicates that content has changed since Googlebot’s last visit. If it gets a 200 status code, Googlebot will receive the full requested resource.
Status codes are retained in your log files and your statistics report (we have another article breaking down some of the most common status codes). To determine whether your server supports the If Modified Since header, simply check your logs for any 304 status codes. If you see any 304s there, you know that your web server supports If Modified Since.
With billions of pages to crawl each day, it makes no sense for Google to re-crawl a page that hasn’t changed. For people with large sites, the crawling process can consume a lot of bandwidth which means extra cost. It therefore saves a lot of bandwidth if your web server supports the If Modified Since header, so it’s recommended that you use a web server with this feature.