1
: one that crawls
2
: a vehicle (such as a crane) that travels on endless chain belts
Synonyms
Examples of crawler in a Sentence
he's always the crawler who makes everyone else late
Recent Examples on the Web
The company uses a web crawler of its own design, known as PerplexityBot, to index the Internet.
—
IEEE Spectrum, 24 Feb. 2024
Going further, The Washington Post reported that Reddit is ready to play hard ball:
If a deal can’t be reached, Reddit is considering blocking search crawlers from Google and Bing, which would prevent the forum from being discovered in searches and reduce the number of visitors to the site.
—Scharon Harding, Ars Technica, 23 Oct. 2023
The startup used Internet Archive data to establish when each website started blocking AI crawlers; many did so soon after OpenAI announced its crawler would respect robots.txt flags in August 2023.
—Kate Knibbs, WIRED, 24 Jan. 2024
The developers didn’t want to build another hack-and-slash game or dungeon crawler.
—Gieson Cacho, The Mercury News, 5 Mar. 2024
These crawlers index websites and enable the kind of search that brings back relevant and up-to-date results you are used to.
—Julius Černiauskas, Forbes, 22 Feb. 2024
Bing’s crawlers do the same, and Microsoft licenses its database to other search engines and companies.
—Peter Rubin, Longreads, 15 Feb. 2024
For example, OpenAI recently detailed a method that websites could use to block its AI-training web crawler using robots.txt.
—Benj Edwards, Ars Technica, 14 Aug. 2023
On weekends, the place is overrun by fun-hungry tourist types and causeway crawlers.
—Miami Herald Archives, Miami Herald, 8 Feb. 2024
IEEE Spectrum, 24 Feb. 2024
Going further, The Washington Post reported that Reddit is ready to play hard ball:
If a deal can’t be reached, Reddit is considering blocking search crawlers from Google and Bing, which would prevent the forum from being discovered in searches and reduce the number of visitors to the site.
—Scharon Harding, Ars Technica, 23 Oct. 2023
The startup used Internet Archive data to establish when each website started blocking AI crawlers; many did so soon after OpenAI announced its crawler would respect robots.txt flags in August 2023.
—Kate Knibbs, WIRED, 24 Jan. 2024
The developers didn’t want to build another hack-and-slash game or dungeon crawler.
—Gieson Cacho, The Mercury News, 5 Mar. 2024
These crawlers index websites and enable the kind of search that brings back relevant and up-to-date results you are used to.
—Julius Černiauskas, Forbes, 22 Feb. 2024
Bing’s crawlers do the same, and Microsoft licenses its database to other search engines and companies.
—Peter Rubin, Longreads, 15 Feb. 2024
For example, OpenAI recently detailed a method that websites could use to block its AI-training web crawler using robots.txt.
—Benj Edwards, Ars Technica, 14 Aug. 2023
On weekends, the place is overrun by fun-hungry tourist types and causeway crawlers.
—Miami Herald Archives, Miami Herald, 8 Feb. 2024
These examples are programmatically compiled from various online sources to illustrate current usage of the word 'crawler.' Any opinions expressed in the examples do not represent those of Merriam-Webster or its editors. Send us feedback about these examples.
Dictionary Entries Near crawler
Cite this Entry
“Crawler.” Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/crawler. Accessed 24 Apr. 2024.
Copy CitationLove words? Need even more definitions?
Merriam-Webster unabridged
Share