Battling bad bots – how to escape the great data scrape

The battle over data-scraping bad bots has been rumbling on for some time now, but with LinkedIn dealing with the fallout from a third major incident in four months, there are signs that the situation is rapidly worsening for companies.

Advances in bad bot technology mean operators are now able to get a wealth of personal information on customers – including their home addresses, phone numbers and email addresses – purely from publicly available pages. While these incidents don’t constitute a data breach or hack, for customers whose data ends up being abused in scams, this may be a distinction without a difference.

Data scraping doesn’t just affect social media companies. Bad bots are ripping through virtually every part of the internet – in particular, industries such as telcos, sports and news. In fact, bad bots now account for nearly one-third of all web traffic. For the channel, this is an opportunity to talk to customers about how to deal with this digital scourge – in particular about bad bot threats evolving, what steps need to be taken, and how to achieve this without creating disruption for legitimate customers.

Today, bad bots are operating ceaselessly around the clock, with operators using them for everything from sensitive data scraping to account takeover attacks. Even simple bots can choke server bandwidth and create unnecessary costs for internet service providers (ISPs).

Unfortunately, this breed of bad bots is in the minority, with 57% of bad bot traffic coming from advanced persistent bots (APBs), which are more sophisticated, mimic human behaviour and use a range of methods to evade detection. The latest research indicates that 34% of all account takeover attempts now come from malicious bots, and the average website is under attack 16% of the time.

When it comes to data scraping, although bad bots aren’t generally getting data such as passwords or credit card numbers used in credential stuffing attacks, it is important to highlight that they are still harvesting a goldmine of personal information that can be used for all sorts of scamming and phishing attacks. This can include anything from tricking customers into fraudulent transactions to loss of IP and other sensitive data – not to mention huge reputational damage and regulatory fines.

And data scraping is just the tip of the iceberg. Because bad bots excel at repetitive tasks that would take humans a significantly long time to complete, they can be used for any number of malicious purposes, including competitive data mining, personal and financial data harvesting, brute-force login, digital ad fraud, denial of service, denial of inventory, spam, transaction fraud, and more.