How to Block Your Website From The Search Engines

 

If you own a website and don’t want your site indexed in the major search engines, you can create a simple robot text file or add a small piece of meta code to your website that will stop search engines from listing your site. An advantage to NOT having your site listed on the major search engines is that your site will be kept private. If you name your site using your first and last name (e.g., www.firstnamelastname.com) it won’t stop someone from guessing the domain name and typing the URL.

If you want to keep your website private and WANT the domain named after your first and last name, you might want to consider running off of WordPress. WordPress is an easy-to-use content management platform. WordPress is easy to install and has features where you can make some areas of your website under password protection. WordPress is a free content management system.

 

Block Your Website From Search Engines Using Meta No Index Tag

Add the following meta tag to the head section of all of your web pages:

<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>

 

See the sample highlighted code below:

Create a Robot Text File

To block the search engines from your website, simply open a new notepad file from your computer.

 

Copy and paste the following code below to your notepad file:

User-agent: *
Disallow: /

See sample screenshot below:

 

Now name your file “robots” and click “Save.” Make sure that the file extension is saved as an *.txt file. See example below:

Upload your robots.txt file to your public_html directory on your web server. Make sure that the file is at the top level of your site (e.g., www.yoursite.com/robots.txt).

If you want to dig further, please visit: http://www.robotstxt.org/

Block Your Website From Google

To remove your site from Google’s index, create the robot text file as discussed in part one of this article. If your website is already established and you want to remove web pages from the Google index faster, you will want to join Google Webmaster Tools. Follow the directions to verify your website in Google Webmaster Tools. Once you have verified your site, click on “Site configuration.”

Next, click on “Crawler access.”

Next, click on the “Remove URL” link.

 

Now click on the “New removal request” button.

Enter the URL’s that you want removed from Google’s index. You will also need to make sure that your site is meets either one of the following guidelines as specified by Google:

a.) The web page returns an HTTP 404 (not found) or 410 status code.
b.) Blocks content using a robots.txt file.
c.) Blocks content using a meta noindex tag.

 

Block Your Website From Bing

To remove your website from the Bing index, simply create the robot text file as discussed in part one of this article.

Block Your Website From Yahoo

To remove your website from the Bing index, simply create the robot text file as discussed in part one of this article.

 

Block Your Website From Ask

To remove your website from the Bing index, simply create the robot text file as discussed in part one of this article.

 

Using Dogpile To Check If Your Site is Still Listed On The Major Search Engines

You can use Dogpile search to see if your site is still listed on the major search engines: Google, Bing, Ask, and Yahoo.

Top