TL;DR
Proxies hide your IP to avoid blocks. Use datacenter proxies for simple sites ($0.10/1000 requests). Use residential proxies for protected sites like Instagram or LinkedIn ($10/GB). Most actors auto-configure proxies.
Why Websites Block Scrapers
Websites detect scrapers by looking at request patterns. When hundreds of requests come from one IP address in minutes, that looks suspicious.
Signs that trigger blocks:
- Too many requests from one IP
- Requests faster than human browsing
- Missing browser fingerprints
- Known datacenter IP ranges
How Proxies Help
A proxy routes your request through a different IP address. With proxy rotation, each request can come from a different IP. This makes your scraper look like many different users.
Types of Proxies
| Type | Source | Cost | Best For |
|---|---|---|---|
| Datacenter | Cloud servers | $0.10 per 1,000 requests | Simple sites, public data |
| Residential | Home internet connections | $10 per GB | Protected sites, social media |
| Mobile | Mobile carrier networks | $25+ per GB | Heavily protected sites |
Datacenter Proxies
These come from cloud servers in data centers. They are fast and cheap. The downside: websites can identify datacenter IP ranges and block them.
Use datacenter proxies for:
- Wikipedia, news sites, government data
- Google search results
- Public APIs and open data
- Sites without anti-bot protection
Residential Proxies
These use IP addresses from real home internet connections. Websites cannot easily distinguish them from regular users.
Use residential proxies for:
- Instagram, TikTok, LinkedIn
- Amazon, eBay, retail sites
- Any site with Cloudflare or similar protection
- Sites that block datacenter IPs
Apify Proxy Setup
Apify includes built-in proxy management. Most actors are pre-configured to use the right proxy type.
Default behavior:
- Actors auto-select proxies based on the target site
- Proxy costs are included in actor pricing (usually)
- You do not need to bring your own proxies
Manual Proxy Configuration
Some actors let you customize proxy settings. Look for "Proxy configuration" in the input form.
Options:
- Automatic - Actor picks the best proxy (recommended)
- Datacenter - Force datacenter proxies (cheaper)
- Residential - Force residential proxies (more reliable)
- Custom - Use your own proxy URLs
Troubleshooting Blocks
Problem: Actor gets blocked frequently
Solution: Switch from datacenter to residential proxies. Increase delays between requests.
Problem: Runs are slow
Solution: Residential proxies are slower. If speed matters more than success rate, try datacenter.
Problem: High proxy costs
Solution: Reduce the number of results. Cache data locally. Do not re-scrape unchanged data.
Common Questions
Q: Can I use my own proxies?
A: Yes. Enter your proxy URLs in the "Custom proxy" field. Format: http://user:pass@host:port
Q: Are proxy costs included in actor pricing?
A: Usually yes for pay-per-result actors. Check the actor description. Compute-based actors may charge proxies separately.
Q: How do I know if I need residential proxies?
A: Try datacenter first. If runs fail with "blocked" errors, switch to residential.