bot_recognizer: class for bot recognizing and dispatching

• Loading "initial" bot definitions from delimited text file (a sample included in distributive);
• Importing initial bot definitions into SQL table, if using DB storage mode planned;
• Importing additional bot definitions from external files (local or internet URL's); at the moment the only supported format is iplists.com bot collections (See links at the bottom of this document)
• detecting bot "id" by the client's IP address and/or user-agent string. In case of using Dispatch() method, respective user procedure can be called. Dispatching process can call unique function for each bot (if desired), or one function for all bots of the same "type": i.e. function 1 for "indexing" bots, function 2 for email harvesters etc. Bots registered as "malicious" can have one common handler function.
  Function Dispatch() can have a limited "working time" interval of a day.
• Debugging site behavior for desired bot, by setting "emulated" bot's IP address and user-agent string.

Installation on the site

• Place php module bot_recognizer.php and "bot-defs.txt" file into your site's folder. If ZEND Framework is not used on your site, as_dbutils.php must be copied too.
• Add manually desired bot definitions to the provided "bot-defs.txt" file, if needed.
• If You're going to use SQL database as a bot definition storage, tune and run script import-botdefs.php once, so "initial" base will be created from "bot-defs.txt" file. This script re-creates table in DB (with dropping previous data), so it can be called each time You changed info in bot-defs.txt

Simple code example

# using dispatch method example
require_once('bot_recognizer.php');

$botrec = new CBotRecognizer();
$botrec->RegisterActionForBots('MicrosoftBots','msn');
$botrec->RegisterActionForBots('GoogleBots','google');
$botrec->Dispatch();

echo "Hello, dear Human !"; // this text is shown only if none of MicrosoftBots and GoogleBots called.
#...

function MicrosoftBots() {
  die("This text is shown for Microsoft spiders (msn) !");
}
function GoogleBots() {
  die("This text is shown for Google bots");
}

# using GetBotId() approach:
require_once('bot_recognizer.php');

$botrec = new CBotRecognizer();
$botid = $botrec->GetBotId();

if($botrec->IsMaliciousBot()) die("You are a malicious bot. Get lost !");

switch($botid) {
  case 'goolge' : case 'msn' :
      echo "This is Microsoft or Google bot. Let them in...";
      break;
  case CBotRecognizer::UNDEFINED_BOT:
      die("some undefined bot detected !");
      break;
  case 0:
     echo "Hello, dear Human !"; // this text is shown only if none of MicrosoftBots and GoogleBots called.
     break;
  default: # all other detected bots...
     echo "Bot $botid detected, no particular action for it !";
     break;
}

Using the class CBotRecognizer

1. short bot name (that will be returned if this bot is recognized), we'll name it "bot id"
2. starting IP address,
3. ending IP address,
4. substring that identifies this bot if found in User-Agent string
5. optional parameter (integer) setting this bot type
6. optional parameter (1) marking this bot as "malicious"

msn|65.55.211.113|65.55.211.119|msnbot
msn|65.55.232.22|65.55.232.22|msnbot
google|66.249.71.22|66.249.71.139|Googlebot

alexa|67.202.54.191|67.202.54.191|ia_archiver
yahoo|72.30.142.240|72.30.142.240|Yahoo!
webalta|76.73.62.242|76.73.62.242|webalta crawler

# some malicious bots from kloth.net, "1" in 5-th field means "malicious bot"
rdprm.gouv.qc.ca|207.96.148.8|207.96.148.8||1
easydl|76.10.155.74|76.10.155.74|EasyDL|1

require_once('as_dbutils.php'); // database access wrapper
require_once('bot_recognizer.php');

$mydb = new CDbEngine(DBTYPE_MYSQL,'localhost','user','password','mydatabase');
$botrec = new CBotRecognizer(array('dbobject'=>$mydb));
#...
$botrec->Dispatch();

require_once('Zend/Db.php');
require_once('Zend/Db/Table.php');
require_once('Zend/Db/Adapter/Pdo/Mysql.php');

require_once('bot_recognizer.php');

# create Zend db adapter object...
$mydb = new Zend_Db_Adapter_Pdo_Mysql(
  array( 'host'=> 'localhost',
         'username' => 'user',
         'password' => 'password',
         'dbname'   => 'mydatabase'));

Zend_Db_Table::setDefaultAdapter($dbAdapter);

# the rest is identical:
$botrec = new CBotRecognizer(array('dbobject'=>$mydb));
#...
$botrec->Dispatch();

1. By calling method CBotRecognizer::CreateBotDefTable(); in that case the empty table [prefix]bot_definitions will be generated.
   (Warning: If the table with the same name exists, it will be dropped, so don't forget to check your existing tables list and use appropriate name prefix !
2. By calling method CBotRecognizer::LoadBotDefinitionsFile('',true) : this function (re)creates empty table (by calling CreateBotDefTable) and loads initial bot definitions from bot-defs.txt file. Probably you'll want to edit bot-defs.txt before running LoadBotDefinitionsFile(), if you have your own bots definitions to be catched and processed.

Method list with descriptions

# fragment from google.txt file, downloaded from www.iplists.com:
# UA "Mediapartners-Google/2.1"
# UA "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
# UA "gsa-crawler (Enterprise; S4-E9LJ2B82FJJAA; me@mycompany.com)"
209.185.108
209.185.253
209.85.238
209.85.238.11
209.85.238.4
...

Additional methods

Links

• iplists.com "IP Addresses of Search Engine Spiders" contains lists of many search spyders, with their IP addresses and user agent strings.
• Nice article on spanishseo.org about bot recognition and blocking malicious ones. It covers some non-trivial methods like reverce and forward DNS lookups.
• "Bad bots" list on kloth.net. A few IP's from this list were placed in bot-defs.txt as "malicious" bots

Change log

1.00.001 (10/02/2009)

• First release