Advertisement

Spiders spin tangled web of information on Net

Reading Time:3 minutes
Why you can trust SCMP
0

Not only have they been a hit at the box office recently, 'eight-legged freaks' also infest the World Wide Web. There are numerous species of spiders (alias 'bots' and 'wanderers') to be found scuttling along its threads.

But these itsy-bitsy virtual critters should neither be screamed at nor squashed. For, just like real spiders, they have a valuable role to play in the information jungle.

They are the unsung heroes of online data gathering, enslaved by an all-powerful entity that exploits them mercilessly - the search engine.

This digital despot was created to accelerate searches within the Internet. The father of all search engines, who was given the pipe-and-slippers name of Archie, arrived in 1990 to search the evolving Internet for files on FTP (File Transfer Protocol) sites.

The next major advance came in the shape of a tool with the equally stuffy name of Veronica, which searched Gopher servers that contained text databases.

Then along came a close forerunner of present search engines, the bohemian-sounding World Wide Wanderer, which rocked all over the Web, tracking its growth.

This program also succeeded in causing widespread irritation because, when it visited a Web site, it slowed down the performance of the network on which the site resided. The wayward Wanderer eventually gave way to more sophisticated searchers such as the alarmingly named WWW Worm, Jumpstation and Nasa's Repository Based Software Engineering project.

Advertisement