Spider Trap

What Does Spider Trap Mean?

A spider trap is an online feature that traps a web crawler or bot in an infinite loop or other recursive situation that takes up its resources and essentially ties up that crawler for a specific set of iterations.


A spider trap is also known as a crawler trap.

Techopedia Explains Spider Trap

Some spider traps are made to intentionally divert the attention of web crawlers. For example, someone may program a deep directory structure so that the web crawler goes crawling down into that structure instead of moving to other areas of a site or online space. Programmers can also overload the crawler’s lexical analyzer, or load up a session with cookies in order to drain the resources of spambots or other crawlers.

Other spider traps are made unintentionally through programming errors. Some types of calendar references can cause seemingly infinite loops and crash poorly made crawlers.

The use of spider traps and other designs to foil robotic web crawlers is going to change with the emergence of new machine learning and artificial intelligence principles. Since designers have proven that they can now make web crawlers that can respond to on-page commands in the same way that humans do, resources like CAPTCHA are no longer going to be effective against robotic users. Spider traps and crawler traps probably will not, either, because the parties using the crawlers or bots will invest them with the ability to recognize these traps and avoid them.


Related Terms

Latest Internet Terms

Related Reading

Margaret Rouse

Margaret Rouse is an award-winning technical writer and teacher known for her ability to explain complex technical subjects to a non-technical, business audience. Over the past twenty years her explanations have appeared on TechTarget websites and she's been cited as an authority in articles by the New York Times, Time Magazine, USA Today, ZDNet, PC Magazine and Discovery Magazine.Margaret's idea of a fun day is helping IT and business professionals learn to speak each other’s highly specialized languages. If you have a suggestion for a new definition or how to improve a technical explanation, please email Margaret or contact her…