How do I stop bots and scrapers from copying my listings or price list?

This page may contain affiliate links from which Blur Studio earns commissions.

Scraping complaints usually mean one of two different problems, and the fix is different depending on which one you've actually got. Problem one: a competitor or data aggregator is systematically copying your listings or prices, usually on a schedule, usually not trying very hard to hide it. Problem two: your site is getting hit hard enough by bot traffic (scrapers, AI crawlers, or just noisy automated requests) that it's affecting load times or server costs, regardless of whether anyone's using the copied data for anything. Both are real, and the fixes overlap but aren't identical.

What basic hosting controls actually catch

Netlify's access controls let you put a password in front of a site or preview, limit who sees a project, and block or allow traffic by IP address or country, with firewall traffic rules and basic rate limiting available on its Free plan. A more capable web application firewall exists too, but it's gated much higher than that: it's an Enterprise-only feature that also requires the separate High-Performance Edge add-on, not something that comes with Pro or any lower tier. For most small businesses, that puts the real WAF out of reach, and the basic Free-tier rules are what you're actually working with. This genuinely stops the lazy, high-volume end of the problem: scripts hammering your site from a small number of IPs, obviously automated request patterns, and traffic from regions you have no legitimate customers in. Netlify's own documentation is honest that these tools are rule-based and relatively basic on lower plans, which is the right way to think about them: a real first line, not a complete solution.

Check your specific plan's documentation for exactly which protections are included at your tier before assuming the strongest version is active by default, since the gap between what's on Free and what's on Pro or Enterprise can be meaningful for a business that's being targeted specifically rather than just catching generic bot noise.

When you need something stronger

A determined scraper rotating through many IP addresses, mimicking a real browser's behavior, and pacing requests to look human, will get past basic rate limiting and simple IP blocks without much trouble. This is where a dedicated bot-protection service, Cloudflare being the most commonly reached-for option, earns its cost: it fingerprints traffic in ways a basic rule set doesn't, catching patterns that look automated even when the requests themselves look individually reasonable.

The decision point is really about how much the scraping actually costs you. If a competitor copying your prices costs you real sales because customers are comparing and choosing based on a scraped feed, or if the bot traffic itself is driving up your hosting costs or slowing the site for real visitors, a dedicated service is worth the setup time and ongoing cost. If it's a minor annoyance with no measurable business impact, the basic controls plus occasionally checking your traffic logs might genuinely be enough.

A few practical things worth doing either way

• Check your traffic logs periodically for obvious patterns: a huge spike from one IP range, requests at an inhumanly consistent interval, or traffic hitting every single listing page in alphabetical or sequential order. These are the easy tells before you reach for anything more sophisticated.
• Don't publish your full price list or inventory in a way that's trivially easy to bulk-extract if pricing leakage is your actual concern. A page that renders prices as an image, or that requires a search or filter interaction rather than listing everything on one static page, raises the effort bar for casual scraping without touching your hosting setup at all.
• Know that some scraping is now specifically AI crawlers gathering training or retrieval data rather than a human competitor, and these often identify themselves honestly in their request headers, meaning you can sometimes block them specifically by name through a robots.txt file or a targeted rule, which is a much simpler fix than blocking anonymous scraping traffic generally.

What I'd actually tell a real estate brokerage or ecommerce business

For a Dubai brokerage or any listings-heavy business where the data itself is the product, I'd treat scraping as a real threat worth proper protection, not a minor annoyance, specifically because a competitor undercutting you using your own scraped prices is a direct hit to your actual business, not just a technical nuisance. For a smaller ecommerce store without that specific competitive pressure, basic rate limiting plus occasional log checks is often genuinely proportionate, and I wouldn't push a client to pay for enterprise-grade bot protection against a problem that isn't actually costing them anything measurable yet.

A worked example: a furniture retailer tracking down a price leak

A furniture retailer noticed a competitor's prices suspiciously tracking theirs within hours of every price change, undercutting by a small, consistent margin every single time. Checking the traffic logs revealed a clear pattern: a burst of requests hitting every single product page in exact catalog-ID order, once every six hours, from a narrow range of IP addresses, the kind of mechanically regular pattern no real customer browsing would ever produce.

The fix had two parts, done in order. First, basic rate limiting and an IP range block through the hosting platform's existing controls stopped that specific scraper within a day, since it wasn't sophisticated enough to rotate IPs or vary its timing. Second, and more durable, the retailer changed how prices rendered on the page itself, no longer sitting as plain text in the page's underlying code where a simple script could read every price in one pass, instead loading through a small script call that a basic scraper wouldn't follow without more effort. The second change mattered more over time: IP blocks catch today's scraper, but a motivated competitor can set up a new one; making the data itself harder to bulk-extract raises the cost of scraping regardless of which specific bot or IP range shows up next.

A practical note on separating nuisance traffic from a real competitive threat

Not every scraping hit deserves the same response. Generic, low-volume crawler traffic that isn't targeting your pricing or listings specifically, general search engine bots, research crawlers, the occasional automated tool, usually isn't worth fighting at all and may even help with visibility. The traffic worth real attention is the kind hitting your most commercially sensitive pages specifically and repeatedly, your price list, your full inventory feed, in a pattern that suggests someone's actually using the data competitively rather than just indexing your site generally. Spend your effort there first rather than trying to block every automated request equally.

How I'd approach this

I'd start by figuring out which problem you actually have: is this hurting your business directly (a competitor undercutting you with scraped prices), or is it a technical nuisance (load, server cost) with no real competitive angle? That answer decides whether basic hosting controls are genuinely enough or whether a dedicated service is worth the investment. If your site's architecture itself is part of the problem, prices rendered in a way that's trivially scraped, or a full listings feed sitting on one unprotected page, that's a build-level fix I can make alongside whatever protection layer you add on top, and it's usually the more durable half of the solution.