SCRAPE-11: Create main runScraper() orchestration function (#16)

## Summary

Implements the top-level runScrape() orchestration function that coordinates the entire scraper pipeline end-to-end.

### What it does

- Full pipeline orchestration: Calls fetchPage, parseUnits, convertDataTypes, upsertUnits, insertPrices, markStaleUnits, updateDailySummary in sequence
- dryRun mode: When enabled, parses and validates HTML but skips all database writes
- htmlContent injection: Accepts raw HTML directly, bypassing the fetch step
- New/rented unit calculation: Diffs currently scraped units against previously active units to determine newUnitsCount and rentedUnitsCount for the daily summary
- Run history recording: Every scrape (success or failure) is recorded to the scraper_runs collection via recordScraperRun()
- Structured logging: All pipeline stages log with jobId correlation for traceability
- Error resilience: Catches and handles errors at each stage, ensuring partial failures are logged and recorded

### Test coverage (15 tests)

- Full workflow with mocked dependencies
- Result structure validation and jobId generation
- dryRun mode skips DB writes
- htmlContent bypasses fetch
- Success and failure history recording
- Fetch error handling with retry exhaustion
- Database operation error handling
- New/rented unit count calculation
- Default and scheduled trigger types
- Empty HTML (no units) edge case

Reviewed-on: #16
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
This commit is contained in:
2026-02-06 09:45:32 -07:00
committed by stephen
parent d1f717891a
commit 0ad4c98abe
3 changed files with 629 additions and 4 deletions

View File

@ -133,8 +133,8 @@ jobs:
fi
# Truncate outputs if too long (max 10000 chars each)
LINT_OUTPUT=$(echo "$RAW_LINT_OUTPUT" | head -c 10000)
TEST_OUTPUT=$(echo "$RAW_TEST_OUTPUT" | head -c 10000)
LINT_OUTPUT="${RAW_LINT_OUTPUT:0:10000}"
TEST_OUTPUT="${RAW_TEST_OUTPUT:0:10000}"
# Build JSON payload
PAYLOAD=$(jq -n \