Commit Graph

2 Commits

Author SHA1 Message Date
7114aee239 Add scraper configuration module with environment variable support
Some checks failed
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Failing after 0s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m35s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m45s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
- Create config/scraper.js with all scraper constants
- Add TARGET_URL for apartment listing page
- Add SCRAPER_SCHEDULE, SCRAPER_TIMEZONE, SCRAPER_ENABLED settings
- Add SCRAPER_TIMEOUT and USER_AGENT settings
- Add RETRY_CONFIG with maxRetries, baseDelay, timeout
- Add COLLECTIONS object with env var overrides for all 4 collections
- Create comprehensive test suite with 25 test cases
2026-01-29 16:17:20 -07:00
9dba679221 Add scraperLogger.js with credential redaction (#6)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-01-28 10:52:00 -07:00