Add fetchPage() with axios, timeout, and retry logic
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 12s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m36s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m42s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped

Implement fetchPage() function for HTTP scraping:
- Uses axios for HTTP GET requests
- Configurable timeout (default 30s) and User-Agent header
- Retry with exponential backoff (1s, 2s, 4s) on 5xx and network errors
- Does not retry on 4xx client errors
- Logs each attempt with attempt number and error details
- Returns HTML string on success, throws after retries exhausted
This commit is contained in:
2026-01-30 11:22:32 -07:00
parent 3af5a90c93
commit 19baf43493
3 changed files with 683 additions and 0 deletions

34
config/scraper.js Normal file
View File

@ -0,0 +1,34 @@
/**
* Scraper configuration
*
* All settings can be overridden via environment variables
*/
module.exports = {
// Target URL
TARGET_URL: 'https://countryclubtowersandgardens.com/property/country-club-towers/apartments/?spaces_tab=unit',
// Scheduling
SCRAPER_SCHEDULE: process.env.SCRAPER_SCHEDULE || '0 6 * * *',
SCRAPER_TIMEZONE: process.env.SCRAPER_TIMEZONE || 'UTC',
SCRAPER_ENABLED: process.env.SCRAPER_ENABLED !== 'false',
// HTTP settings
SCRAPER_TIMEOUT: parseInt(process.env.SCRAPER_TIMEOUT, 10) || 30000,
USER_AGENT: 'Mozilla/5.0 (compatible; ApartmentScraper/1.0)',
// Retry configuration
RETRY_CONFIG: {
maxRetries: 3,
baseDelay: 1000, // 1 second, will be: 1s, 2s, 4s
timeout: parseInt(process.env.SCRAPER_TIMEOUT, 10) || 30000
},
// Collections (environment variable overrides for development isolation)
COLLECTIONS: {
UNITS: process.env.SCRAPER_UNITS_COLLECTION || 'units_migration_test',
PRICES: process.env.SCRAPER_PRICES_COLLECTION || 'unit_prices_migration_test',
DAILY_SUMMARIES: process.env.SCRAPER_SUMMARIES_COLLECTION || 'daily_summaries',
SCRAPER_RUNS: process.env.SCRAPER_RUNS_COLLECTION || 'scraper_runs'
}
};