[Screenshots] How to Block Bad Bots on SiteGround Tutorial

0
Last Updated - August 2, 2018
SiteGround Block Bad Bots
SiteGround Block Bad Bots

The internet is full of bots. You might have heard that thousands of sites are being hacked on daily basis. Again you might have also heard about the ‘Search Engine’ bots which are used to index the contents of your website. All of these are possible for the bots.

However, the first example is of bad bots and the second one is of the good bots. The bad bots can do damage to your website in many ways. So, if you want to be secure then you need to keep them away from your website.

If you don’t know how to do that. Then this article is for you. Because in this article, we will talk about the methods of blocking the bad bots from any website. If you were searching for this article, you can simply start reading.

So, without further ado, let’s jump in.

Our Top Ranked #1 Web Host

SiteGround Reviews Top Logo

Note: This website ReviewPlan.com is hosted on SiteGround

Latest Uptime Stats (Real Records):

SiteGround Uptime
SiteGround Uptime

Server Response Speed Test Scored A+:

SiteGround Server Speed
SiteGround Server Speed

Visit SiteGround

How to block bad bots on SiteGround

As we have already told you that there are two types of bots. Let us explain further what they do with some examples.

Good Bots:

The bots used by the search engines to crawl your websites are the good ones. They don’t have any type of bad intentions. Some bots are used to check the vulnerability and health of a particular website. The Google bots, Yahoo bots, and Bing bots can be called as good bots. As they don’t want to take advantage of your website.

Bad Bots:

Not all bots are good though. In fact, the number of bad bots is more than the good bots on the internet. There are a lot of different types of bad bots out there. For example, malicious bots are used to do DDoS attacks. It could impact any particular websites or even a computer.

On top of that, there are scrapper bots; which are used to copy any particular websites’ contents for the bot developer. Then they can rewrite or spin the contents for their website. Another example of the bad bot is the spam commenter bots. They will try to comment on any websites to get a ping-back to the owners’ website.

And, the hacker bots are there to hack any websites with low security.

Effects of Bots on a website:

Each type of bots has different purposes. However, they also come with a bad side effect. As we said there a lot of different bots, and they try to access your site without any limitations, it puts a lot of pressure on your website.

Although the good bots are necessary but even they add in the pressure by sending requests after requests to your web host server. On top of that, add the activities of the bad bots. Eventually, it increases the strain and it might slow down your website. For worse, it could even bring down a site.

Now, there’s a solution to this problem though. You can block the bad bots from accessing your website. You will get all the needed instructions below on how to block bad bots on your website hosted in SiteGround or any other hosts.

How to block the bad bots?

There are a number of ways to block bad bots on a website. However, for your convenience, we will share three methods to block bad bots. First one would be using the robots.txt file method, secondly, you can use plugins if you are using WordPress websites and the last one is to tweaking the .htaccess file.

You can follow any one of them. However, if you want an easy way out then use plugins but if you are more of a techy person then go with the third option. Let’s see how to accomplish that step by step.

Method 1. By modifying the ‘robots.txt’:

We have already talked about how to use the robots.txt file to allow or disallow bots on this detailed guide. So, we won’t go into explaining it again. You can read that if you want to know the process.

However, we will just let you know what code to use to block the bots. You will be able to learn the rest from the guide which we mentioned earlier.

The code syntax to use in the robots.txt file:

User-agent: Evilbot
User-agent: FakeAgent
Disallow: /

robots.txt example
robots.txt example

Explanation:

Here, User-agent denotes the name of the bots, for example, Evilbot, FakeAgent. You can use any other name of the bad bots in place of these bots. If you don’t understand the process then please feel free to refer to the ‘robots.txt’ guide.

Also, we should tell you this that the ‘robots.txt’ file is just a guideline. This means that it depends totally on the bots if they abide by the rules or not. Generally, the bad bots are developed in a way that they would not listen to the ‘robots.txt’ file. In that case, you have to follow the other two methods which we will be discussed below.

Method 2. By using plugins:

Now, if you use WordPress CMS on your website then you can use different security plugins to block the bad bots. There are a lot of plugins to block bots. And, if you are not a techy person then you should follow this method. It is one of the easiest methods to block bad bots from your website.

In this section, we will talk about three different plugins which can be used to block the bad bots. You can use any one of them. Even you can use any other plugins if you don’t like the ones listed here.

Plugin 1. Wordfence Security-Firewall & Malware Scan:

This plugin is one of the most popular security plugins among WordPress users. It has over 1 million active installations till date. So, install and activate this plugin on your website and configure it to secure your website.

To make it easier for your, we will walk you through the whole process. So, let’s go.

Steps to follow:

Installing and activating the plugin:

  • Log in to your WordPress dashboard
Log in to WordPress dashboard
Log in to the WordPress dashboard
  • From the left sidebar, hover your mouse pointer over the ‘Plugins’ option and a new sub-menu will appear. Click on the ‘Add New’ button from the sub-menu
‘Add New’ button
‘Add New’ button
  • On the top-right corner of the next page, you will find a search box where you can directly search for your required plugins. So, type in ‘Wordfence’ on the search box and press the ‘Enter’ key on your keyboard
Type in ‘Wordfence’
Type in ‘Wordfence’
  • Now, you will find the ‘Wordfence Security Firewall’ plugin in the search results. It is developed by ‘Wordfence’
‘Wordfence Security Firewall’ plugin
‘Wordfence Security Firewall’ plugin
  • So, click on the ‘Install Now’  button to install it
‘Install Now’  button
‘Install Now’  button
  • And then click on the ‘Activate’ button to activate it
‘Activate’ button
‘Activate’ button

Ok, you have activated the plugin. Now, let’s configure it.

Configuring the ‘Wordfence’ plugin:

  • When the plugin was activated it will take you to a new page and you will see a popup like the image below. You need to enter a valid email address on the blank box. You will receive security alerts to that email address. Then if you want to subscribe to the mailing list of ‘Wordfence’ then you can optionally click on the ‘Yes’ button from the 2nd line (use ‘No’, if you don’t want to subscribe). After that, add a check-mark on the ‘I agree to Terms and Privacy Policy’ option. And, finally, click on the ‘Continue’ button to proceed
Configure the form
Configure the form
  • Next, if you are using the free version then click on the ‘No Thanks’ button. But, if you have a premium key then you can install that
‘No Thanks’ button
‘No Thanks’ button
  • On the next page, you will find a notification on top of the page. Click on the ‘Click here to configure’ button
‘Click here to configure’ button
‘Click here to configure’ button
  • Again, another popup box will appear. First of all, click on the ‘Download .Htaccess’ button and download the file to your PC as a backup. Later, click on the ‘Continue’ button to proceed
‘Continue’ button
‘Continue’ button
  • On the next page, you will find another popup box saying the ‘Installation Successful’. Simply, click on the ‘Close’ button to proceed
‘Close’ button
‘Close’ button
  • Everything will be automatically configured. However, if you want to go ahead then you can manually start a scan. To do that, hover your mouse pointer over the ‘Wordfence’ option from the left sidebar to bring out the sub-menu. From the sub-menu click on the ‘Scan’ button to open the dashboard
‘Scan’ button
‘Scan’ button
  • Now, from the next page, click on the ‘Start Scan’ button to initiate the scanning process
‘Start Scan’ button
‘Start Scan’ button
  • It will take a bit of time to complete the scan. The ‘Wordfence’ plugin will also automatically run a scan daily when it is active on your website
Scan in progress
Scan in progress

That is all for configuring the ‘Wordfence’ plugin. If you don’t want to go with the hassle of configurations then you can use other plugins. In the next section, we will share two more plugins which are very easy to use. In fact, you don’t need to configure them at all. They are like activate and forget. So, let’s see.

Note: As we have already told you how to install and activate a plugin in the previous section, we won’t talk about it further. You can follow the previous section whenever you need to.

Plugin 2. Blackhole for Bad Bots:

  • Install and activate the ‘Blackhole for Bad Bots’ plugin developed by ‘Jeff Starr’ on your WordPress website
‘Blackhole for Bad Bots’ plugin
‘Blackhole for Bad Bots’ plugin
  • That’s it. It will take care of the rest

Plugin 3. BBQ:

It is another plugin developed by ‘Jeff Starr’. The name of the plugin is ‘BBQ: Block Bad Queries’. This is more of a firewall plugin for your website.

  • So, install and activate the plugin (BBQ) on your website
BBQ plugin
BBQ plugin
  • You don’t have to do anything more at all

So, you have learned how to use plugins to block bad bots from accessing your website. Now, we will tell you how to block those bad bots by editing the .htaccess file via your cPanel.

Method 3. By editing the .htaccess file:

Now, if you don’t know what is the ‘.htaccess’ file or even how to access it. Then you should give a read to our detailed guide on how to access the ‘.htaccess’ file. After you have read that, you should come back here and follow along. However, if you know where to find the ‘.htaccess’ the file then you can skip that and step ahead with us now.

From here on, we will assume that you know how to access the .htaccess file. So, we won’t explain it further. Rather, we will tell you how to edit the file. Let’s go.

Note: It is better to take a backup of the ‘.htaccess’ file before tweaking it

How to block bad bots by editing the ‘.htaccess’ file:

  • Log in to your cPanel and access the ‘.htaccess’ file on the root directory (‘public_html’ folder) on your website
.htaccess file
.htaccess file
  • Then click on the ‘.htaccess’ file to select it. Then click on the ‘Code Editor’ button from the top menu
‘Code Editor’ button
‘Code Editor’ button
  • A new popup box will appear. If you don’t want to see this pop up again then click on the ‘Disable Encoding Check’ button. After that, click on the ‘Edit’ button to continue
‘Edit’ button
‘Edit’ button
  • Generally, you will find the next page like the image below
.htaccess file codes
.htaccess file codes
  • Now, you have to paste a block of code below the current codes. It means you should paste the new code on the last line
Paste the new code on the last line
Paste the new code on the last line
  • So, we will share the code below. You have to copy and paste the code where we told you to paste it. Now, remember, you can’t add or remove anything from the code. That way you will mess up the code and your site might be down for that reason. Here is the code:

# 6G FIREWALL/BLACKLIST
# @ https://perishablepress.com/6g/

# 6G:[QUERY STRINGS]
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{QUERY_STRING} (eval\() [NC,OR]
RewriteCond %{QUERY_STRING} (127\.0\.0\.1) [NC,OR]
RewriteCond %{QUERY_STRING} ([a-z0-9]{2000}) [NC,OR]
RewriteCond %{QUERY_STRING} (javascript:)(.*)(;) [NC,OR]
RewriteCond %{QUERY_STRING} (base64_encode)(.*)(\() [NC,OR]
RewriteCond %{QUERY_STRING} (GLOBALS|REQUEST)(=|\[|%) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)(.*)script(.*)(>|%3) [NC,OR]
RewriteCond %{QUERY_STRING} (\\|\.\.\.|\.\./|~|`|<|>|\|) [NC,OR]
RewriteCond %{QUERY_STRING} (boot\.ini|etc/passwd|self/environ) [NC,OR]
RewriteCond %{QUERY_STRING} (thumbs?(_editor|open)?|tim(thumb)?)\.php [NC,OR]
RewriteCond %{QUERY_STRING} (\'|\")(.*)(drop|insert|md5|select|union) [NC]
RewriteRule .* - [F]
</IfModule>

# 6G:[REQUEST METHOD]
<IfModule mod_rewrite.c>
RewriteCond %{REQUEST_METHOD} ^(connect|debug|delete|move|put|trace|track) [NC]
RewriteRule .* - [F]
</IfModule>

# 6G:[REQUEST STRINGS]

<IfModule mod_alias.c>
RedirectMatch 403 (?i)([a-z0-9]{2000,})
RedirectMatch 403 (?i)(https?|ftp|php):/
RedirectMatch 403 (?i)(base64_encode)(.*)(\()
RedirectMatch 403 (?i)(=\\\'|=\\%27|/\\\'/?)\.
RedirectMatch 403 (?i)/(\$(\&)?|\*|\"|\.|,|&|&amp;?)/?$
RedirectMatch 403 (?i)(\{0\}|\(/\(|\.\.\.|\+\+\+|\\\"\\\")
RedirectMatch 403 (?i)(~|`|<|>|:|;|,|%|\\|\s|\{|\}|\[|\]|\|)
RedirectMatch 403 (?i)/(=|\$&|_mm|cgi-|etc/passwd|muieblack)
RedirectMatch 403 (?i)(&pws=0|_vti_|\(null\)|\{\$itemURL\}|echo(.*)kae|etc/passwd|eval\(|self/environ)
RedirectMatch 403 (?i)\.(aspx?|bash|bak?|cfg|cgi|dll|exe|git|hg|ini|jsp|log|mdb|out|sql|svn|swp|tar|rar|rdf)$
RedirectMatch 403 (?i)/(^$|(wp-)?config|mobiquo|phpinfo|shell|sqlpatch|thumb|thumb_editor|thumbopen|timthumb|webshell)\.php
</IfModule>

# 6G:[USER AGENTS]

SetEnvIfNoCase User-Agent ([a-z0-9]{2000}) bad_bot
SetEnvIfNoCase User-Agent (archive.org|binlar|casper|checkpriv|choppy|clshttp|cmsworld|diavol|dotbot|extract|feedfinder|flicky|g00g1e|harvest|heritrix|httrack|kmccrew|loader|miner|nikto|nutch|planetwork|postrank|purebot|pycurl|python|seekerspider|siclab|skygrid|sqlmap|sucker|turnit|vikspider|winhttp|xxxyy|youda|zmeu|zune) bad_bot
Order Allow,Deny
Allow from All
Deny from env=bad_bot

# 6G:[BAD IPS]
Order Allow,Deny
Allow from All
# uncomment/edit/repeat next line to block IPs
# Deny from 123.456.789

Example image after pasting the code
Example image after pasting the code
  • After you have pasted the code, click on the ‘Save’ button from the top -right corner of the screen
‘Save’ button
‘Save’ button
  • You will see a ‘Success’ notification on the bottom left corner of the screen if the saving is completed
‘Success’ notification
‘Success’ notification
  • If you want to add more bots to the block list then you can use more similar types of codes. You can check this link for more code

REMEMBER: You have to check if you can access your site after you have tweaked your .htaccess file. If you can’t then there’s a chance that you have messed up the code. So, recheck the code. Or if you can’t figure it out, clear the codes from the .htaccess file and save it again like it was before. Otherwise, if you have taken a backup of the file then use that instead

Winding up:

So, if you have followed the tutorial, you already know three methods of blocking bad bots from your website. You can use any one of them. But, if you want to be sure enough then you can use all of the methods at once.

And, don’t forget to check if you can access your website after the customization. If you are facing trouble then just revert everything and you will be fine. Hope this will help you secure your website in future. From now on, you will be able to conserve your bandwidth and server resources as well.

LEAVE A REPLY