SCRAPE-10: Implement recordScraperRun() for scraper_runs #15

Merged
stephen merged 3 commits from scraper/record-run into dev 2026-02-06 02:00:20 -07:00

3 Commits

Author SHA1 Message Date
48512e01ba Replace console.error with structured logger in recordScraperRun
All checks were successful
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 3s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m36s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m51s
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Switch error logging from console.error to the injected logger.error
pattern for consistency with the rest of the scraper service layer.
Update tests to verify logger.error is called instead of console.error.
2026-02-06 01:41:50 -07:00
74dc62c53f Fix shell quoting in CI webhook notification step
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 12s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m51s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Move all GitHub Actions expressions into env: block instead of
interpolating directly into shell script. Test output containing
single quotes was breaking the shell parser, causing EOF errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 01:08:55 -07:00
d4bb1088a2 Implement recordScraperRun() for scraper_runs collection
Some checks failed
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m36s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m51s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add recordScraperRun(db, runData) to scraperService that persists
scraper execution history to the scraper_runs collection. The function
inserts a document containing all run metadata (jobId, trigger, status,
duration, unit metrics, timing, errors) along with a recordedAt
timestamp.

Designed for use in finally blocks: errors during recording are caught,
logged, and swallowed so they never interrupt the scraper pipeline.
Returns the insertOne result on success or null on failure.

Includes 17 tests covering insertion behavior, field preservation,
graceful error handling, return values, and recordedAt semantics.
2026-02-06 00:31:07 -07:00