RFC 9309

Robots Exclusion Protocol, September 2022

File formats:

icon for HTML icon for text file icon for v3pdf icon for XML
Status:
PROPOSED STANDARD
Authors:
M. Koster
G. Illyes
H. Zeller
L. Sassman
Stream:
IETF
Source:
NON WORKING GROUP

Cite this RFC: TXT  |  XML

DOI:  10.17487/RFC9309

Discuss this RFC: Send questions or comments to iesg@ietf.org

Other actions: View Errata  |  Submit Errata  |  Find IPR Disclosures from the IETF  |  View History of RFC 9309


Abstract

This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.


For the definition of Status, see RFC 2026.

For the definition of Stream, see RFC 8729.




Advanced Search