After denying search engines from indexing the millions of bookmarks stored on its servers for so long, del.icio.us has finally lifted the entry barrier but selectively.
According to the robots.txt of del.icio.us, only spiders from Yahoo!, Google and Ask search engines are allowed to crawl and index the content of del.icio.us - all other bots are banned including msnbot of the Microsoft Live search engine.
User-agent: *
Disallow: /
User-agent: Slurp [Yahoo]
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Teoma [Ask.com]
Allow: /
del.icio.us has also removed the meta robots (noarchive,nofollow,noindex) restriction from all their webpages that earlier prevented search engines from indexing or caching delicious web pages.
As a result, Google has 6 million pages from del.icio.us in their index now while this number was around a million sometime back when the ban was in place.
What does this mean for sites / URLs that are popular on del.icio.us - you could see a surge in search engine traffic but getting listed on del.icio.us won't add any link authority or google juice since all links on delicious are marked with nofollow, very similar to Wikipedia.
0 comments:
Post a Comment