All options for user-agent in robots.txt file

From The IT Community
Jump to: navigation, search




Here is a list of all options you can set for the user-agent parameter in your robots.txt file to specify certain bots:


1.ABCdatos BotLink
2.Acme.Spider
3.Ahoy! The Homepage Finder
4.Alkaline
5.Anthill
6.Walhello appie
7.Arachnophilia
8.Arale
9.Araneo
10.AraybOt
11.ArchitextSpider
12.Aretha
13.ARIADNE
14.arks
15.AskJeeves
16.ASpider (Associative Spider)
17.ATN Worldwide
18.Atomz.com Search Robot
19.AURESYS
20.BackRub
21.Bay Spider
22.Big Brother
23.Bjaaland
24.BlackWidow
25.Die Blinde Kuh
26.Bloodhound
27.Borg-Bot
28.BoxSeaBot
29.bright.net caching robot
30.BSpider
31.CACTVS Chemistry Spider
32.Calif
33.Cassandra
34.Digimarc Marcspider/CGI
35.Checkbot
36.ChristCrawler.com
37.churl
38.cIeNcIaFiCcIoN.nEt
39.CMC/0.01
40.Collective
41.Combine System
42.Conceptbot
43.ConfuzzledBot
44.CoolBot
45.Web Core / Roots
46.XYLEME Robot
47.Internet Cruiser Robot
48.Cusco
49.CyberSpyder Link Test
50.CydralSpider
51.Desert Realm Spider
52.DeWeb(c) Katalog/Index
53.DienstSpider
54.Digger
55.Digital Integrity Robot
56.Direct Hit Grabber
57.DNAbot
58.DownLoad Express
59.DragonBot
60.DWCP (Dridus' Web Cataloging Project)
61.e-collector
62.EbiNess
63.EIT Link Verifier Robot
64.ELFINBOT
65.Emacs-w3 Search Engine
66.ananzi
67.esculapio
68.Esther
69.Evliya Celebi
70.FastCrawler
71.Fluid Dynamics Search Engine robot
72.Felix IDE
73.Wild Ferret Web Hopper #1, #2, #3
74.FetchRover
75.fido
76.Hämähäkki
77.KIT-Fireball
78.Fish search
79.Fouineur
80.Robot Francoroute
81.Freecrawl
82.FunnelWeb
83.gammaSpider, FocusedCrawler
84.gazz
85.GCreep
86.GetBot
87.GetURL
88.Golem
89.Googlebot
90.Grapnel/0.01 Experiment
91.Griffon
92.Gromit
93.Northern Light Gulliver
94.Gulper Bot
95.HamBot
96.Harvest
97.havIndex
98.HI (HTML Index) Search
99.Hometown Spider Pro
100.ht://Dig
101.HTMLgobble
102.Hyper-Decontextualizer
103.iajaBot
104.IBM_Planetwide
105.Popular Iconoclast
106.Ingrid
107.Imagelock
108.IncyWincy
109.Informant
110.InfoSeek Robot 1.0
111.Infoseek Sidewinder
112.InfoSpiders
113.Inspector Web
114.IntelliAgent
115.I, Robot
116.Iron33
117.Israeli-search
118.JavaBee
119.JBot Java Web Robot
120.JCrawler
121.Jeeves
122.JoBo Java Web Robot
123.Jobot
124.JoeBot
125.The Jubii Indexing Robot
126.JumpStation
127.image.kapsi.net
128.Katipo
129.KDD-Explorer
130.Kilroy
131.KO_Yappo_Robot
132.LabelGrabber
133.larbin
134.legs
135.Link Validator
136.LinkScan
137.LinkWalker
138.Lockon
139.logo.gif Crawler
140.Lycos
141.Mac WWWWorm
142.Magpie
143.marvin/infoseek
144.Mattie
145.MediaFox
146.MerzScope
147.NEC-MeshExplorer
148.MindCrawler
149.mnoGoSearch search engine software
150.moget
151.MOMspider
152.Monster
153.Motor
154.MSNBot
155.Muncher
156.Muninn
157.Muscat Ferret
158.Mwd.Search
159.Internet Shinchakubin
160.NDSpider
161.Nederland.zoek
162.NetCarta WebMap Engine
163.NetMechanic
164.NetScoop
165.newscan-online
166.NHSE Web Forager
167.Nomad
168.The NorthStar Robot
169.nzexplorer
170.ObjectsSearch
171.Occam
172.HKU WWW Octopus
173.OntoSpider
174.Openfind data gatherer
175.Orb Search
176.Pack Rat
177.PageBoy
178.ParaSite
179.Patric
180.pegasus
181.The Peregrinator
182.PerlCrawler 1.0
183.Phantom
184.PhpDig
185.PiltdownMan
186.Pimptrain.com's robot
187.Pioneer
188.html_analyzer
189.Portal Juice Spider
190.PGP Key Agent
191.PlumtreeWebAccessor
192.Poppi
193.PortalB Spider
194.psbot
195.GetterroboPlus Puu
196.The Python Robot
197.Raven Search
198.RBSE Spider
199.Resume Robot
200.RoadHouse Crawling System
201.RixBot
202.Road Runner: The ImageScape Robot
203.Robbie the Robot
204.ComputingSite Robi/1.0
205.RoboCrawl Spider
206.RoboFox
207.Robozilla
208.Roverbot
209.RuLeS
210.SafetyNet Robot
211.Scooter
212.Sleek
213.Search.Aus-AU.COM
214.SearchProcess
215.Senrigan
216.SG-Scout
217.ShagSeeker
218.Shai'Hulud
219.Sift
220.Simmany Robot Ver1.0
221.Site Valet
222.Open Text Index Robot
223.SiteTech-Rover
224.Skymob.com
225.SLCrawler
226.Inktomi Slurp
227.Smart Spider
228.Snooper
229.Solbot
230.Spanner
231.Speedy Spider
232.spider_monkey
233.SpiderBot
234.Spiderline Crawler
235.SpiderMan
236.SpiderView(tm)
237.Spry Wizard Robot
238.Site Searcher
239.Suke
240.suntek search engine
241.Sven
242.Sygol
243.TACH Black Widow
244.Tarantula
245.tarspider
246.Tcl W3 Robot
247.TechBOT
248.Templeton
249.TeomaTechnologies
250.TITAN
251.TitIn
252.The TkWWW Robot
253.TLSpider
254.UCSD Crawl
255.UdmSearch
256.UptimeBot
257.URL Check
258.URL Spider Pro
259.Valkyrie
260.Verticrawl
261.Victoria
262.vision-search
263.void-bot
264.Voyager
265.VWbot
266.The NWI Robot
267.W3M2
268.WallPaper (alias crawlpaper)
269.the World Wide Web Wanderer
270.w@pSpider by wap4.com
271.WebBandit Web Spider
272.WebCatcher
273.WebCopy
274.webfetcher
275.The Webfoot Robot
276.Webinator
277.weblayers
278.WebLinker
279.WebMirror
280.The Web Moose
281.WebQuest
282.Digimarc MarcSpider
283.WebReaper
284.webs
285.Websnarf
286.WebSpider
287.WebVac
288.webwalk
289.WebWalker
290.WebWatch
291.Wget
292.whatUseek Winona
293.WhoWhere Robot
294.Wired Digital
295.Weblog Monitor
296.w3mir
297.WebStolperer
298.The Web Wombat
299.The World Wide Web Worm
300.WWWC Ver 0.2.5
301.WebZinger
302.XGET


Feel free to add additional bots if they are not in the list.



Was this article helpful? Then please donate to keep The IT Community alive...

If you found this article helpful please share it, comment and help others by writing your own article.






Translate this page:




Articles found in the same category:
(max. 20 shown)