PHP code to block bad bots from crawling your website

Bots are also known as web crawlers or spiders. There are bad bots and there are good bots. How bots are bad and how it is good, it’s a long topic that I am not going to discuss here in this post. Let’s stick to the post title :-). To allow Google and Bing bots, for example, to crawl your website while blocking bad bots, you can utilize a combination of robots.txt and server-side configurations. But besides robots.txt or .htaccess rules, you can also use PHP to block bad bots from accessing your website.

Suppose your website is coded with PHP or using frameworks like CodeIgniter, Laravel, etc then you can use this code snippet. You can use this on WordPress too if you don’t want to use plugins.

Certainly! Here’s an example of a PHP code snippet that can be used to block bad bots while allowing Googlebot and Bingbot to crawl your website:

PHP

<?php
$userAgent = $_SERVER['HTTP_USER_AGENT'];

// Define an array of bad bot user agents to block
$badBots = array(
    'MJ12Bot',     // Block this bot    
    'Baiduspider'  // Block Baidu
);

// Check if the user agent is in the bad bot list
$botDetected = false;
foreach ($badBots as $bot) {
    if (stripos($userAgent, $bot) !== false) {
        $botDetected = true;
        break;
    }
}

// Block bad bots and allow Googlebot and Bingbot
if (!$botDetected) {
    // Good bot, allow access to the website
    // Add your website's code here
} else {
    // Bad bot, block access
    header('HTTP/1.0 403 Forbidden');
    exit;
}
?>

In this code, we retrieve the user agent from the $_SERVER['HTTP_USER_AGENT'] variable, which contains information about the client’s browser or bot. We then check if the user agent matches any of the specified bad bots (You can add more if required). If a bad bot is detected, we send a 403 Forbidden response and exit the script, effectively blocking access.

For good bots like Googlebot and Bingbot, we allow access to the website by placing your website’s code within the corresponding section.

Remember to include this code snippet at the beginning of each PHP file or use a common include file (e.g. header.php) across your website’s pages for consistent bot blocking.

Leave a Reply Cancel reply