SCRAPE-11: Create main runScraper() orchestration function #16

Merged

stephen merged 2 commits from scraper/run-scrape into dev 2026-02-06 09:45:32 -07:00

Conversation 1 Commits 2 Files Changed 3 +629 -4

stephen commented 2026-02-06 01:37:55 -07:00

Owner

Copy Link

Summary

Implements the top-level runScrape() orchestration function that coordinates the entire scraper pipeline end-to-end.

What it does

• Full pipeline orchestration: Calls fetchPage, parseUnits, convertDataTypes, upsertUnits, insertPrices, markStaleUnits, updateDailySummary in sequence
• dryRun mode: When enabled, parses and validates HTML but skips all database writes
• htmlContent injection: Accepts raw HTML directly, bypassing the fetch step
• New/rented unit calculation: Diffs currently scraped units against previously active units to determine newUnitsCount and rentedUnitsCount for the daily summary
• Run history recording: Every scrape (success or failure) is recorded to the scraper_runs collection via recordScraperRun()
• Structured logging: All pipeline stages log with jobId correlation for traceability
• Error resilience: Catches and handles errors at each stage, ensuring partial failures are logged and recorded

Test coverage (15 tests)

• Full workflow with mocked dependencies
• Result structure validation and jobId generation
• dryRun mode skips DB writes
• htmlContent bypasses fetch
• Success and failure history recording
• Fetch error handling with retry exhaustion
• Database operation error handling
• New/rented unit count calculation
• Default and scheduled trigger types
• Empty HTML (no units) edge case

## Summary Implements the top-level runScrape() orchestration function that coordinates the entire scraper pipeline end-to-end. ### What it does - Full pipeline orchestration: Calls fetchPage, parseUnits, convertDataTypes, upsertUnits, insertPrices, markStaleUnits, updateDailySummary in sequence - dryRun mode: When enabled, parses and validates HTML but skips all database writes - htmlContent injection: Accepts raw HTML directly, bypassing the fetch step - New/rented unit calculation: Diffs currently scraped units against previously active units to determine newUnitsCount and rentedUnitsCount for the daily summary - Run history recording: Every scrape (success or failure) is recorded to the scraper_runs collection via recordScraperRun() - Structured logging: All pipeline stages log with jobId correlation for traceability - Error resilience: Catches and handles errors at each stage, ensuring partial failures are logged and recorded ### Test coverage (15 tests) - Full workflow with mocked dependencies - Result structure validation and jobId generation - dryRun mode skips DB writes - htmlContent bypasses fetch - Success and failure history recording - Fetch error handling with retry exhaustion - Database operation error handling - New/rented unit count calculation - Default and scheduled trigger types - Empty HTML (no units) edge case

stephen added 1 commit 2026-02-06 01:37:55 -07:00

SCRAPE-11: Implement runScrape() main orchestration function ... 5ef5d5af73

Add the top-level runScrape() function that coordinates the full scraper
pipeline: fetch HTML (with retry), parse units, convert data types, upsert
units, insert prices, mark stale units, and update daily summary.

Features:
- dryRun mode skips all database writes while still parsing/validating
- htmlContent parameter allows injecting HTML directly (bypasses fetch)
- Calculates newUnitsCount and rentedUnitsCount by diffing against prior state
- Records every run to scraper_runs history (success or failure)
- Structured logging with jobId correlation throughout the pipeline
- Graceful error handling at each pipeline stage

Includes 15 tests covering full workflow, dry run, error handling,
trigger types, new/rented unit calculation, and empty HTML edge case.

stephen force-pushed scraper/run-scrape from 5ef5d5af73 to 5e1f1f8041 2026-02-06 09:16:29 -07:00 Compare

stephen added 1 commit 2026-02-06 09:29:14 -07:00

Fix SIGPIPE in CI webhook step by using bash string slicing ... 08d7a4b0b0

Replace echo | head -c pipe with ${VAR:0:10000} bash substring
to avoid exit code 141 when output exceeds truncation limit.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

stephen commented 2026-02-06 09:40:22 -07:00

Author

Owner

Copy Link

❌ Changes Requested

Code quality, patterns, security, and tests are all excellent. However, commit 08d7a4b has a 'Co-Authored-By: Claude Opus 4.6' trailer which violates the project's no-AI-mention policy for commits.

Issues:

.github/workflows/deploy.yml:0
Commit 08d7a4b ('Fix SIGPIPE in CI webhook step') includes a 'Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com' trailer in the commit message. Per project policy (CLAUDE.md), AI/Claude must never be mentioned in commits, PRs, or code comments. Please amend or squash this commit to remove the Co-Authored-By trailer before merging.

## ❌ Changes Requested Code quality, patterns, security, and tests are all excellent. However, commit 08d7a4b has a 'Co-Authored-By: Claude Opus 4.6' trailer which violates the project's no-AI-mention policy for commits. ### Issues: **.github/workflows/deploy.yml:0** Commit 08d7a4b ('Fix SIGPIPE in CI webhook step') includes a 'Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>' trailer in the commit message. Per project policy (CLAUDE.md), AI/Claude must never be mentioned in commits, PRs, or code comments. Please amend or squash this commit to remove the Co-Authored-By trailer before merging.

stephen merged commit 0ad4c98abe into dev 2026-02-06 09:45:32 -07:00

stephen deleted branch scraper/run-scrape 2026-02-06 09:45:32 -07:00

stephen referenced this issue from a commit 2026-02-06 09:45:34 -07:00

SCRAPE-11: Create main runScraper() orchestration function (#16)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

stephen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stephen/apartment-dashboard-api#16

No description provided.