|
|
 |
| HTML Code Clinic Do you have some HTML code you'd like to share and get suggestions from others for tweaking or improving it? This discussion is the place. |
Welcome to the p2p.wrox.com Forums.
You are currently viewing the HTML Code Clinic section of the Wrox p2p Programmer to Programmer discussion community. This is a community of more than 40,000 computer programmers including Wrox book authors and readers. As a guest, you can read any forum posting. By joining our free Wrox p2p community you can post your own programming questions and respond to other programmers’ questions. Registered users also don't have to see the ads that are displayed to guests. Registration is fast, simple and absolutely free so please, join today!
Join today and post to win prizes! Post more to increase your chances of being Wrox’s top poster of the month.
|
 |
April 5th, 2009, 07:26 PM
|
|
Friend of Wrox
|
|
Join Date: Jan 2005
Location: Mauchline, East Ayrshire, Scotland
Posts: 1,518
Thanks: 0
Thanked 0 Times in 0 Posts
|
|
prevent Google/Yahoo!/MSN spidering my webpage
Hi,
Am i safe enough with the following meta tags to prevent Google/Yahoo!/MSN Search from spidering my webpage?
Code:
<meta name="robots" content="noindex" />
<meta name="robots" content="nofollow" />
<meta name="robots" content="noarchive" />
<meta name="robots" content="noodp" />
<meta name="robots" content="noimageindex,nomediaindex" />
<meta name="robots" content="unavailable_after: 05-Apr-2009 22:00:00 CET" />
<meta name="googlebot" content="noindex">
<meta name="googlebot" content="nosnippet" />
<meta name="slurp" content="noydir">
My robots.txt file:
Code:
User-agent: *
Disallow: /picco.html
Also - is there anyway to test if Google will pick it up?
Thanks,
Picco |
April 5th, 2009, 09:01 PM
|
 |
Wrox Staff
|
|
Join Date: May 2003
Location: Indianapolis, IN, USA.
Posts: 1,349
Thanks: 27
Thanked 49 Times in 40 Posts
|
|
I think you have it more than covered. To verify it with Google, sign up for a free Google webmaster tools account
www.google.com/webmasters/tools/
and from there, you can verify which pages Google is and isn't spidering under "URLs restricted by robots.txt." |
April 6th, 2009, 06:47 PM
|
|
Friend of Wrox
|
|
Join Date: Jan 2005
Location: Mauchline, East Ayrshire, Scotland
Posts: 1,518
Thanks: 0
Thanked 0 Times in 0 Posts
|
|
Thanks Jim, i'll do that. What about MSN Search and Yahoo!
Will this prevent those two search engines from spidering the page?
Picco
|
April 7th, 2009, 05:28 PM
|
|
Wrox Staff
|
|
Join Date: May 2003
Location: Indianapolis, IN, USA.
Posts: 1,349
Thanks: 27
Thanked 49 Times in 40 Posts
|
|
I think you're covered with the robots.txt, as I think both Yahoo and MSN (Live) search are respectable crawlers and follow the robot.txt rules. Where you may run in to problems are some of the lesser known crawlers, and even malicious crawlers that deliberately ignore robots. If there is a link to this page from some other page, there's a good chance some crawlers will index it.
|
April 21st, 2009, 05:13 AM
|
|
Friend of Wrox
|
|
Join Date: Jun 2007
Location: San Diego, CA, USA.
Posts: 278
Thanks: 0
Thanked 4 Times in 4 Posts
|
|
Yeah, the robots.txt file is all you really need to disallow the page. None of the major engines are looking to catalog things you don't want indexed.
If it's something really sensitive, you may want to look at ASP.NET 2.0. There are some fairly basic ways of setting up a login system to password protect sensitive files like that. Check out www.asp.net for video tutorials on it if you're interested.
__________________
-------------------------
Whatever you can do or dream you can, begin it. Boldness has genius, power and magic in it. Begin it now.
-Johann von Goethe
When Two Hearts Race... Both Win.
-Dove Chocolate Wrapper
Chroniclemaster1, Founder of www.EarthChronicle.com
A Growing History of our Planet, by our Planet, for our Planet.
|
May 8th, 2009, 03:24 PM
|
|
Authorized User
|
|
Join Date: Nov 2008
Location: virudhunagar, tamil nadu, India.
Posts: 22
Thanks: 0
Thanked 0 Times in 0 Posts
|
|
Try php
Try php. It provide some ways to satisfy your needs.
|
| Thread Tools |
Search this Thread |
|
|
|
| Display Modes |
 Linear Mode
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
|
|
on their post!
