[rfc-i] URIs in RFC references, was: feedback on draft-iab-styleguide-01

John R Levine johnl at taugh.com
Wed Mar 26 20:14:08 PDT 2014

>> Yes, this is a kludge, but it turns out to be a really simple and
>> effective one.  If you recognize Googlebot and bingbot and Slurp
>> (Yahoo) and maybe a few others, you'll seed all of the indexes anyone
>> cares about.
> I'd recommend against this one. First, it makes entries for new search 
> engines tougher. But more importantly, if our documents don't contain the 
> necessary information (be it in a visible header or in hidden meta-data), 
> then I think we are doing a bad job on the main form of the document.

If the kludge sends the full page for any user agent that includes the 
string "bot" or "spider", it won't miss many.  I look at my web server 
logs all the time, and the amount of spider traffic is impressive.

John Levine, johnl at taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

More information about the rfc-interest mailing list