Stephen Minakian
5ef5d5af73
Some checks failed
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m56s
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add the top-level runScrape() function that coordinates the full scraper pipeline: fetch HTML (with retry), parse units, convert data types, upsert units, insert prices, mark stale units, and update daily summary. Features: - dryRun mode skips all database writes while still parsing/validating - htmlContent parameter allows injecting HTML directly (bypasses fetch) - Calculates newUnitsCount and rentedUnitsCount by diffing against prior state - Records every run to scraper_runs history (success or failure) - Structured logging with jobId correlation throughout the pipeline - Graceful error handling at each pipeline stage Includes 15 tests covering full workflow, dry run, error handling, trigger types, new/rented unit calculation, and empty HTML edge case.