Apple has confirmed the existence of its Applebot web crawler. In a support document (http://apple.co/1H0Mp37) the company had this to say:
“Applebot is the web crawler for Apple, used by products including Siri and Spotlight Suggestions. It respects customary robots.txt rules and robots meta tags. It originates in the 17.0.0.0 net block.
“User-agent strings will contain ‘Applebot’ together with additional agent information. For example:
“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) “AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
“If robots instructions don’t mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.”
