Commit Graph

10 Commits

Author SHA1 Message Date
d4bb1088a2 Implement recordScraperRun() for scraper_runs collection
Some checks failed
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m36s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m51s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add recordScraperRun(db, runData) to scraperService that persists
scraper execution history to the scraper_runs collection. The function
inserts a document containing all run metadata (jobId, trigger, status,
duration, unit metrics, timing, errors) along with a recordedAt
timestamp.

Designed for use in finally blocks: errors during recording are caught,
logged, and swallowed so they never interrupt the scraper pipeline.
Returns the insertOne result on success or null on failure.

Includes 17 tests covering insertion behavior, field preservation,
graceful error handling, return values, and recordedAt semantics.
2026-02-06 00:31:07 -07:00
b4978caf31 SCRAPE-9: Implement updateDailySummary() aggregation (#14)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-06 00:15:30 -07:00
d8adfbf90c SCRAPE-8: Implement markStaleUnits() (#13)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 23:48:57 -07:00
2b53288fbe SCRAPE-7: Implement insertPrices() bulk operation (#12)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 22:32:02 -07:00
4863dc1824 SCRAPE-6: Implement upsertUnits() bulk operation (#11)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 22:10:54 -07:00
0cbeaaf9d5 SCRAPE-5: Implement convertDataTypes() with type parsing helpers (#10)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 21:33:58 -07:00
072e80d069 SCRAPE-4: Implement parseUnits() with cheerio selectors (#9)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 17:14:01 -07:00
c6beaf8333 SCRAPE-3: Implement fetchPage() with axios, timeout, retry (#8)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 15:58:07 -07:00
2993d019c5 SCRAPE-2: Add scraper config constants (#7) 2026-01-31 20:54:43 -07:00
9dba679221 Add scraperLogger.js with credential redaction (#6)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-01-28 10:52:00 -07:00