Why reliability metrics in the web scraping industry are broken, and what a meaningful industry standard should look like‍

“99.9%.” “98%.” “Near 100%.” These are all success rates from vendors in the web scraping market. Each may well be accurate. But there’s a catch: none of them mean the same thing. 

Take web unblocking products, the services that get automated requests past websites’ anti-bot defences. In 2026, the success rates on the public record for them, from vendors and independent testers alike, range anywhere from 40% all the way to the maximum of 100%, and most of these figures are defensible for what they’re conveying. 

There’s an even more interesting finding. The variation in numbers doesn’t only exist between competitors, but can appear across a single provider’s own product page, pricing page, and documentation. Most of the numbers are accurate, but there’s a lack of specification – hardly anyone explains what they’re actually measuring.

So, if this is the case, why don’t providers simply post one success rate? While this seems like an obvious solution, it’s also a risky one. Simply altering conditions such as the request type, the domain tested, whether retries are included, and even the time of day can lead to noticeably different success rates from the same service. 

Therefore, rather than a provider trying to form one optimal success rate, what it needs to do is define its success rates. And we need to do this consistently as an industry.  

What is success? The definition conundrum

Before we outline the definition problem that resides in success rates, it’s perhaps useful to define what web scraping is. Web scraping allows companies to automate the collection of publicly available data from websites to gain information for a whole range of purposes. It could be used for market research, to improve product pricing, to understand risk better, or to manage brand reputation and messaging.

To send a request, a company will insert the URL of the website they want information from into the scraping tool. The tool then submits a request for this data to the website’s server.If the server handles the request normally, it sends back the page’s code along with what’s called an HTTP 200 status code. As far as the server is concerned, that means the request succeeded. But that doesn’t necessarily mean the response contains the data the company is after. 

That’s because the server might have returned a 200 for a CAPTCHA page (we’ve all had to click on those image boxes) or a page saying the content no longer exists, sent with a 200 code anyway (a so-called “soft 404”). So, to know if the webpage actually contains the data the company’s looking for, the scraping tool also needs to validate the content on it.

This is where the definition problem emerges. If there are two providers who are running identical infrastructure, but one of them is tallying 200s, and the other is validating content instead, even on an ordinary protected site, they might end up publishing success figures that are tens of points apart. 

This is without considering retries, which complicate the issue even further. A provider might report a 60% success rate that came from one attempt, while another provider might say they had 99% success after five attempts.

Both are correct and could be describing the same infrastructure, but the 99% figure hides how many retries it took, and the extra time and cost that come with them. If a number is quoted without an explanation around retries, then its usefulness for buyers comparing tools when making procurement decisions diminishes. 

A similar problem occurs when accounting for the types of domains that were tested and the timeframe this took place in. The same provider can get excellent results in one independent test and more mediocre ones in another, without any misrepresentation taking place. And if a provider’s success rate is an average across many websites, most of which a given buyer will never scrape, it says very little about how the service will perform on that buyer’s own targets.

The impact on trust in the market 

When there is such disparity between the numbers quoted and little information around the context behind them, trust and credibility suffer. If you’re a genuinely reliable and high-performing provider, the variability between success rates in the market makes it very hard to prove your value. 

From the buyer side, as they can’t accurately compare products, understandably, they lose trust in success rates completely. With no evidence on performance to go off, their decisions can end up being based on price and brand reputation or recognition. But if a buyer is choosing technology or infrastructure for their business, then performance should be the guiding factor. 

So, how can we, as an industry, reestablish trust in performance metrics and ensure they play a key role in decision-making?  

Forming an industry standard

In short, providers need to build the habit of describing what a number really means. 

The minimum amount of context a published success rate should outline is what counts as success; the retry policy behind it, in attempts or time; the domains and request types tested, with which features enabled and at what concurrency; the timeframe the measurement took place in; the sample size and its margin of error; and who took the measurement (the vendor, its customers or an independent tester) and who set up the test. If any of these variables aren’t accounted for, then the headline figure can shift by more than 20 percentage points, despite the service and infrastructure remaining the same. 

We can borrow from an existing discipline. Site reliability engineering, the practice tech companies use to keep their own systems running, starts by defining an “indicator”: a measure that spells out what counts as a good result, what counts as a valid attempt, and how it’s measured. Most reliability claims in web scraping skip that step, which is why buyers can’t check them. The indicator is the part worth borrowing. SRE’s targets and guarantees are designed for systems a company controls, and no scraping provider controls the websites it collects data from.

Retries need to be a special area of focus. The post-retry success rate tells buyers whether they’ll get their data at all. The first-attempt rate tells them what that data costs, in waiting time and in credits. A provider that only posts the post-retry figure is leaving out half the picture. This is why both pre- and post-retry numbers should be clearly signposted – the difference between the two enables buyers to predict what they will need to pay.

If we don’t give buyers the information they need behind the context of success rates, then it becomes more difficult for them to trust the category and, consequently, harder for it to grow. 

Forming an industry standard doesn’t require any complex activity. All it needs is for providers to state what a number means. It’s better to post a lower figure that is backed up by evidence and context than a larger one that will lose its credibility under greater scrutiny.