What you need to know about how Google indexes your website
After the Frogs push the Big Red Button and a client’s brand new website goes live, it’s only a matter of time before the client calls and asks why they can’t see their beautiful, new site. It’s not that the site isn’t there; it’s that the Internet, like all technology, works in mysterious ways. Or maybe it’s not so mysterious if you understand a couple of key technical concepts.
When dealing with any kind of technology, particularly the types that are related to the Internet, patience is definitely a virtue. For reasons we’ll get into shortly, it can take up to 48 hours before everyone online will be able to see your new website.
TTL and DNS and caches, oh my!
There’s a common perception that when a new site goes live, it creeps out over the Internet like pond scum over a lake until it’s completely covered the cyber-ether. Not that we want to compare your awesome, new site to pond scum, but that’s the general belief out there as to how the Internet works. It’s not reality, though.
Not to get too technical, but there’s a concept called Time To Live (TTL) (that’s “live” as in “live on stage” or “going live,” not “live” as in “I live in an expensive condo”). TTL can take up to 48 hours, although some Internet users will be able to see your new site much sooner. The reason has to do with domain names and server addresses.
Think of your domain name like a licence plate and your server like the licence plate number. Someone out there wants to find you on the Internet. They punch in your domain name, but their browser doesn’t know where it is. Your browser has to check with the Domain Name System (DNS), a vast database of all the domain names out there, to get your server’s coordinates. Servers are, of course, found on the Net by a string of numbers, also known as your IP (Internet Protocol) address. Then the DNS tells your browser what the domain name’s server numbers are, and voila, that someone has just been pointed to your website.
Because this takes a few seconds to do (ever notice when you first visit a website that it takes longer to load than subsequent visits?), computers store the information in a DNS cache so the browser doesn’t have to keep pinging the Internet Powers That Be each time you want to visit a website. The cache stores the information for whatever the tech guy has set it to, or more likely, the default of almost all DNS systems, which is – you guessed it – 48 hours.
With changes in servers, this means it’ll take up to 48 hours before someone can find your new website.
Flushing the cache
There is a little trick that will empty your DNS cache so you can instantly see your new website. A DNS cache flush requires some serious technical knowledge, but the Frogs can walk you through the process if you find your patience is wearing thin.
The Google problem
Another question related to websites going live the Frogs often get asked is, “Why is my site not listed on Google?”
If you think 48 hours is a long time to wait, consider that Google may not find your new website for six to eight weeks. Google attempts to query the whole internet every day, which is an impossible task. So it takes them a little bit of time to come around to your neighbourhood, knock on your door, and start spidering your website.
Spidering is a technique used by search engines to spread out across the Web to find every site out there. The main way it does this, though, is by finding links on pages it already knows. A brand new domain name isn’t likely to be linked anywhere, so it can take Google quite some time to find it. There are no links or pathways that lead to your website. Add to that that Google adds a “cooling off” period to make sure you aren’t a spammer or nere’do’well, and your website might take months to get indexed. In fact, new sites will take longer, where older sites will get picked up much faster.
While it’s usually best to let Google find your site through spidering, there are sometimes reasons that clients need their new sites to be found as soon as possible. New domain names can be submitted to Google manually, but even then, it takes about a week before Google gets around to checking it out. Keep this in mind when formulating any marketing plans that revolve around a new site.
Not sure if your site has been indexed yet by Google? Try This!
Go to www.google.com and type in “site:www.yourcompanydomain.com” in the search field. If your site appears in the search, then congratulations! …you’ve been indexed! If it does not, you have a little while longer to wait. Sorry! Try sending Matt Cutts a box of chocolates or something to see if he’ll come around to visit sooner.
Another Tip: Try having another website link to you. Oftentimes we find that this speeds up the process a bit as well, as it creates a pathway to your site from another site that Google has already indexed.
Patience is a virtue
It would be nice if as soon as the Frogs pushed the Big Red Button that everyone could suddenly find it and you’d have a million visitors instantaneously, but the reality of the Internet is that everything takes time.
For a variety of technical reasons mentioned above, your website will not be instantly found by visitors or Google. Have some patience and take some advice from Douglas Adams: Don’t panic.