RedfinScraper

A Python tool for scraping rental listing data from Redfin.com using the ScrapingAnt web scraping API. Built for educational purposes to demonstrate how to work with real estate APIs, proxy-based web scraping, and structured data extraction.

Disclaimer: This project is intended for educational and research purposes only. Please review Redfin's Terms of Service before use. The authors are not responsible for any misuse of this tool.

How It Works

1. Builds Redfin rental search API URLs with geographic polygon boundaries and filters
2. Routes requests through ScrapingAnt proxy (browser=false, proxy_country=US) to handle access reliably
3. Parses the JSON response directly — no HTML parsing or browser rendering needed
4. The Redfin API returns all matching results in a single response (no pagination required)
5. Deduplicates listings by property_id and exports to CSV/JSON

Setup

pip install -r requirements.txt

Sign up for a ScrapingAnt API key to get started.

Usage

python main.py \
  --polygon "-116.50089 43.30739,-115.96191 43.30739,-115.96191 43.85001,-116.50089 43.85001,-116.50089 43.30739" \
  --market boise \
  --api-key "YOUR_SCRAPINGANT_API_KEY" \
  --verbose

Required Arguments

Argument	Description
--polygon	Search area as closed polygon coordinates: "lon1 lat1,lon2 lat2,...,lon1 lat1"

Optional Arguments

Argument	Default	Description
--market	boise	Redfin market identifier (e.g. boise, seattle, denver)
--num-homes	350	Max number of homes to request
--sort	square-footage-asc	Sort order
--uipt	1,2,3	Property types: 1=House, 2=Condo, 3=Townhouse
--sf	1,2,3,5,6,7	Search filters
--output / -o	output/redfin_rentals.csv	Output CSV path
--json	false	Also export as JSON
--api-key	env SCRAPINGANT_API_KEY	ScrapingAnt API key
--verbose / -v	false	Enable verbose logging

Output Fields

Field	Description
property_id	Redfin property identifier
rental_id	Redfin rental listing UUID
property_type	House, Condo, Townhouse, Apartment, etc.
address	Full formatted address
city, state, zip_code	Address components
price_min, price_max	Rent price range (USD)
bedrooms_min, bedrooms_max	Bedroom count range
bathrooms_min, bathrooms_max	Bathroom count range
sqft_min, sqft_max	Square footage range
lat, lon	Geographic coordinates
url	Full Redfin listing URL
description	Listing description (truncated to 500 chars)
feed_source	Data source (e.g. APPFOLIO_FREE, RENTEC)
last_updated	When the listing was last updated
date_available	Move-in availability date
num_available_units	Number of available units
date_scraped	Timestamp of scrape

Example

Scrape rental listings in the Boise, ID area:

python main.py \
  --polygon "-116.50089 43.30739,-115.96191 43.30739,-115.96191 43.85001,-116.50089 43.85001,-116.50089 43.30739" \
  --market boise \
  --uipt "1,2,3" \
  --api-key "YOUR_KEY" \
  --json \
  --verbose

Learning Objectives

This project demonstrates several key concepts for educational purposes:

• API Reverse Engineering: How to identify and interact with undocumented REST APIs used by modern web applications
• Proxy-Based Scraping: Using ScrapingAnt to route requests through residential proxies for reliable data access
• Geospatial Queries: Working with polygon-based geographic search boundaries
• Data Normalization: Transforming nested JSON API responses into flat, structured datasets
• Robust Error Handling: Implementing retry logic with exponential backoff for production-grade scrapers