How to Scrape E-commerce Prices (Without Getting Blocked)

Why Scrape Prices?

Competitors change prices constantly. Amazon updates prices 2.5 million times per day. You cannot track this manually.

Automated price monitoring lets you:

• Match or beat competitor prices instantly
• Spot pricing trends before they hurt you
• Find arbitrage opportunities across marketplaces
• Track MAP (minimum advertised price) violations

The Challenge

E-commerce sites fight scrapers hard. Amazon, eBay, Walmart, and others use:

• CAPTCHA challenges
• IP blocking and rate limits
• Bot detection services
• Dynamic content loading

Simple scripts fail fast. You need infrastructure.

Option 1: Pre-built Actors (Recommended)

For major retailers, someone already solved this. Apify's Store has tested actors for:

These actors handle proxies, retries, and anti-bot measures. You provide product URLs. They return clean data.

Step-by-Step Setup

Step 1: Create Your Product List

Start with competitor product URLs. Organize by category:

Category,Product,URL
Electronics,Sony Headphones,https://amazon.com/dp/B0BT8LG7TY
Electronics,Bose Headphones,https://amazon.com/dp/B0CCZ26B5V
Home,Dyson Vacuum,https://amazon.com/dp/B0CDFPKR9B

Step 2: Pick the Right Actor

Search Apify Store for your retailer. Look for:

• High success rate (95%+)
• Recent updates (within 30 days)
• Good reviews
• Active maintainer

Step 3: Configure the Run

Most e-commerce actors need:

• Product URLs or search queries
• Max items to scrape
• Output format (JSON, CSV, Excel)

Step 4: Schedule Daily Runs

Apify lets you schedule actors to run automatically. Daily price checks work for most use cases. Hourly for high-volume competitive markets.

Step 5: Export and Analyze

Data goes to Apify storage. Export to:

• Google Sheets (built-in integration)
• Your database via API
• Excel download
• Webhook to your systems

Cost Breakdown

Price monitoring costs vary by scale:

Apify's free tier covers small monitoring. The $49/month plan handles most medium businesses.

Common Mistakes

Scraping too fast. Aggressive rates trigger blocks. Let the actor handle timing.

Ignoring failures. Some products fail. Check success rates. Investigate persistent failures.

Not scheduling. Manual runs create gaps. Schedule and forget.

Storing too little history. Keep at least 90 days of price data. Trends matter more than snapshots.

Advanced: Building Your Own

For niche retailers without pre-built actors, you can build custom scrapers. Apify supports:

• Puppeteer (JavaScript)
• Playwright (JavaScript/Python)
• Cheerio (HTML parsing)
• Crawlee (full framework)

But this requires developer time. For one-off sites, often cheaper to hire someone on Apify's marketplace.