SCRAPE-11: Create main runScraper() orchestration function (#16)
## Summary Implements the top-level runScrape() orchestration function that coordinates the entire scraper pipeline end-to-end. ### What it does - Full pipeline orchestration: Calls fetchPage, parseUnits, convertDataTypes, upsertUnits, insertPrices, markStaleUnits, updateDailySummary in sequence - dryRun mode: When enabled, parses and validates HTML but skips all database writes - htmlContent injection: Accepts raw HTML directly, bypassing the fetch step - New/rented unit calculation: Diffs currently scraped units against previously active units to determine newUnitsCount and rentedUnitsCount for the daily summary - Run history recording: Every scrape (success or failure) is recorded to the scraper_runs collection via recordScraperRun() - Structured logging: All pipeline stages log with jobId correlation for traceability - Error resilience: Catches and handles errors at each stage, ensuring partial failures are logged and recorded ### Test coverage (15 tests) - Full workflow with mocked dependencies - Result structure validation and jobId generation - dryRun mode skips DB writes - htmlContent bypasses fetch - Success and failure history recording - Fetch error handling with retry exhaustion - Database operation error handling - New/rented unit count calculation - Default and scheduled trigger types - Empty HTML (no units) edge case Reviewed-on: #16 Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
This commit is contained in:
4
.github/workflows/deploy.yml
vendored
4
.github/workflows/deploy.yml
vendored
@ -133,8 +133,8 @@ jobs:
|
||||
fi
|
||||
|
||||
# Truncate outputs if too long (max 10000 chars each)
|
||||
LINT_OUTPUT=$(echo "$RAW_LINT_OUTPUT" | head -c 10000)
|
||||
TEST_OUTPUT=$(echo "$RAW_TEST_OUTPUT" | head -c 10000)
|
||||
LINT_OUTPUT="${RAW_LINT_OUTPUT:0:10000}"
|
||||
TEST_OUTPUT="${RAW_TEST_OUTPUT:0:10000}"
|
||||
|
||||
# Build JSON payload
|
||||
PAYLOAD=$(jq -n \
|
||||
|
||||
Reference in New Issue
Block a user