Yahoo! Slurp Spider. What is it? - Fly Fishing Forum
Worldwide Flyfishing Discussion Talk flyfishing with members around the world!

Thread Tools Search this Thread Rate Thread Display Modes
Old 08-14-2007, 06:06 PM
Warren's Avatar
Warren Warren is offline
Join Date: Mar 2006
Location: Lake Sunapee, New Hampshire
Posts: 726
Yahoo! Slurp Spider. What is it?

OK all you folks well versed in all things internet. I have a question. I was looking at the who is online section here under the quick links tab in attempt to see what all the guests listed on the front page are all about. At this writing there are 116 members & 2349 guests. The who is online deal shows all the registered users are and what page they are viewing. After the last registered user the guests show up. Some are listed as guests and then there was mostly a thing called .....Yahoo! Slurp Spider.

What is a Yahoo! Slurp Spider. I have never seen the term before? Is it like a web crawler that search engines use? In any event that is where most of the guest hits are coming from. Could someone explain this to me?
Reply With Quote Quick reply to this message
Sponsored Links
Old 08-15-2007, 12:08 AM
juro's Avatar
juro juro is offline
Coast2coast Flyfishaholic
Join Date: Jan 2000
Location: Steelhead country|striper coast|bonefish belt
Posts: 20,594
Its the yahoo crawler, a pain in the sides for most webmasters with forums. They take a high percentage of guest accesses and it depends on time of day etc - but are different from registered users who make up the majority of hits to the forum.

I've put some blockers in place since yahoo insists on being total asses to site owners by eating up resources and bandwidth when the world uses google anyway. Check it in a few days and I will try other means if necessary.

Dear Yahoo Search, give it up. Google won.
IFFF Certified THCI @ 2005
Capeflyfisher Guide Service
Island Hopper, Guitarist, Incurable Dreamer
and Founder, Worldwide Flyfishing Forum
Reply With Quote Quick reply to this message
Old 08-15-2007, 07:14 PM
Warren's Avatar
Warren Warren is offline
Join Date: Mar 2006
Location: Lake Sunapee, New Hampshire
Posts: 726
Thanks Juro, I lernt sumpthin
Reply With Quote Quick reply to this message
Old 09-02-2007, 11:11 PM
bad_angler bad_angler is offline
Registered User
Join Date: Sep 2007
Location: South Carolina
Posts: 4
As a web developer I'd suggest you configure your server to not allow access to parts of your site based on the user-agent string. Robots.txt exclusion is honored by slurp and thats a simple way to limit access to parts of your site. See the slurp help url in your access logs or google for robots.txt format. Spiders are specifically telling your server what manner of client is accessing the pages and you are supposed to handle those clients differently if you don't want them indexing all of your site. If you are running apache with mod_rewrite installed you can add a rewrite rule like this

RewriteCond %{HTTP_USER_AGENT} ^Some Funky User-Agent$
RewriteRule ^index\.html$ /dontindexme.html [L]

to have parts of your site hidden from the spidering of pages. Whenever an agent named 'Some Funky User-Agent' comes to access index.html they get sent to maybe your dead page called dontindexme.html. The user--agent is there for the webmasters benefit..allowing you to control what happens when it visits.

Ok back to flyfishing.
bad_angler - South Carolina's Worst Angler
Reply With Quote Quick reply to this message

Quick Reply

Register Now

In order to be able to post messages on the Fly Fishing Forum forums, you must first register.
Please enter your desired user name, your email address and other required details in the form below.
User Name:
Please enter a password for your user account. Note that passwords are case-sensitive.
Confirm Password:
Email Address
Please enter a valid email address for yourself.
Email Address:
Home Waters
Your home waters
Current Favorite Fly
If you only had one... (change anytime)


Human Verification

In order to verify that you are a human and not a spam bot, please enter the answer into the following box below based on the instructions contained in the graphic.

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Spider! Camoman13 Getting Started 0 11-14-2007 09:49 PM
Spider Crab striblue >> Archive: Striper (etc) Flies 3 11-28-2003 10:20 PM
Caddis spider juro >> Archive: Salmon & Steelhead Flies 0 02-22-2001 09:36 PM

All times are GMT -5. The time now is 02:35 AM.

vBulletin Security provided by vBSecurity v2.2.2 (Pro) - vBulletin Mods & Addons Copyright © 2017 DragonByte Technologies Ltd.
Copyright (All Rights Reserved)