SCRAPE-10: Implement recordScraperRun() for scraper_runs #15

Merged

stephen merged 3 commits from scraper/record-run into dev 2026-02-06 02:00:20 -07:00

Conversation 1 Commits 3 Files Changed 3 +364 -16

stephen commented 2026-02-06 00:31:37 -07:00

Owner

Copy Link

Summary

• Adds recordScraperRun(db, runData) to scraperService.js that persists scraper execution history to the scraper_runs collection
• Inserts a document containing all run metadata (jobId, trigger, status, duration, unit processing metrics, timing fields, errors array) along with a recordedAt timestamp
• Uses non-blocking error handling: failures during recording are caught and logged but never thrown, making it safe for use in finally blocks without disrupting the scraper pipeline
• Returns the insertOne result on success or null on failure

Test Coverage

17 tests covering:

• Insertion behavior and recordedAt field injection
• Preservation of all runData fields (jobId, trigger, status, duration, metrics, timing, dryRun, errors)
• Graceful error handling (no throw on DB failure, error logging)
• Return value semantics (result object on success, null on failure)
• recordedAt field validation (Date instance, proximity to current time, no field overwrites)

Test Plan

• All 17 unit tests pass
• No regressions in existing test suites

## Summary - Adds recordScraperRun(db, runData) to scraperService.js that persists scraper execution history to the scraper_runs collection - Inserts a document containing all run metadata (jobId, trigger, status, duration, unit processing metrics, timing fields, errors array) along with a recordedAt timestamp - Uses non-blocking error handling: failures during recording are caught and logged but never thrown, making it safe for use in finally blocks without disrupting the scraper pipeline - Returns the insertOne result on success or null on failure ## Test Coverage 17 tests covering: - Insertion behavior and recordedAt field injection - Preservation of all runData fields (jobId, trigger, status, duration, metrics, timing, dryRun, errors) - Graceful error handling (no throw on DB failure, error logging) - Return value semantics (result object on success, null on failure) - recordedAt field validation (Date instance, proximity to current time, no field overwrites) ## Test Plan - [x] All 17 unit tests pass - [x] No regressions in existing test suites

stephen added 1 commit 2026-02-06 00:31:38 -07:00

Implement recordScraperRun() for scraper_runs collection ... d4bb1088a2

Add recordScraperRun(db, runData) to scraperService that persists
scraper execution history to the scraper_runs collection. The function
inserts a document containing all run metadata (jobId, trigger, status,
duration, unit metrics, timing, errors) along with a recordedAt
timestamp.

Designed for use in finally blocks: errors during recording are caught,
logged, and swallowed so they never interrupt the scraper pipeline.
Returns the insertOne result on success or null on failure.

Includes 17 tests covering insertion behavior, field preservation,
graceful error handling, return values, and recordedAt semantics.

stephen added 1 commit 2026-02-06 01:09:00 -07:00

Fix shell quoting in CI webhook notification step ... 74dc62c53f

Move all GitHub Actions expressions into env: block instead of
interpolating directly into shell script. Test output containing
single quotes was breaking the shell parser, causing EOF errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

stephen commented 2026-02-06 01:20:04 -07:00

Author

Owner

Copy Link

❌ Changes Requested

recordScraperRun() uses console.error instead of the structured logger pattern used by every other function in scraperService.js. It should accept a logger parameter to match upsertUnits, insertPrices, markStaleUnits, and updateDailySummary.

Issues:

services/scraperService.js:634
Pattern inconsistency: every other db-accessing function in this file accepts a logger parameter and uses logger.error()/logger.info() for output. This function should follow the same pattern: async function recordScraperRun(db, runData, logger) and use logger.error('Failed to record scraper run', { errorMessage: error.message }) instead of console.error.

tests/scraper/recordScraperRun.test.js:186
Update test to pass a mock logger and assert logger.error() is called instead of spying on console.error. This aligns with how other test suites in this project verify logging behavior.

## ❌ Changes Requested recordScraperRun() uses console.error instead of the structured logger pattern used by every other function in scraperService.js. It should accept a logger parameter to match upsertUnits, insertPrices, markStaleUnits, and updateDailySummary. ### Issues: **services/scraperService.js:634** Pattern inconsistency: every other db-accessing function in this file accepts a `logger` parameter and uses `logger.error()`/`logger.info()` for output. This function should follow the same pattern: `async function recordScraperRun(db, runData, logger)` and use `logger.error('Failed to record scraper run', { errorMessage: error.message })` instead of `console.error`. **__tests__/scraper/recordScraperRun.test.js:186** Update test to pass a mock logger and assert `logger.error()` is called instead of spying on `console.error`. This aligns with how other test suites in this project verify logging behavior.

stephen added 1 commit 2026-02-06 01:47:14 -07:00

Replace console.error with structured logger in recordScraperRun ... 48512e01ba

Switch error logging from console.error to the injected logger.error
pattern for consistency with the rest of the scraper service layer.
Update tests to verify logger.error is called instead of console.error.

stephen merged commit d1f717891a into dev 2026-02-06 02:00:20 -07:00

stephen deleted branch scraper/record-run 2026-02-06 02:00:20 -07:00

stephen referenced this issue from a commit 2026-02-06 02:00:21 -07:00

SCRAPE-10: Implement recordScraperRun() for scraper_runs (#15)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

stephen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stephen/apartment-dashboard-api#15

No description provided.