d4bb1088a2
Implement recordScraperRun() for scraper_runs collection
...
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m36s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m51s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add recordScraperRun(db, runData) to scraperService that persists
scraper execution history to the scraper_runs collection. The function
inserts a document containing all run metadata (jobId, trigger, status,
duration, unit metrics, timing, errors) along with a recordedAt
timestamp.
Designed for use in finally blocks: errors during recording are caught,
logged, and swallowed so they never interrupt the scraper pipeline.
Returns the insertOne result on success or null on failure.
Includes 17 tests covering insertion behavior, field preservation,
graceful error handling, return values, and recordedAt semantics.
2026-02-06 00:31:07 -07:00
b4978caf31
SCRAPE-9: Implement updateDailySummary() aggregation ( #14 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-06 00:15:30 -07:00
d8adfbf90c
SCRAPE-8: Implement markStaleUnits() ( #13 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 23:48:57 -07:00
2b53288fbe
SCRAPE-7: Implement insertPrices() bulk operation ( #12 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 22:32:02 -07:00
4863dc1824
SCRAPE-6: Implement upsertUnits() bulk operation ( #11 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 22:10:54 -07:00
0cbeaaf9d5
SCRAPE-5: Implement convertDataTypes() with type parsing helpers ( #10 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 21:33:58 -07:00
072e80d069
SCRAPE-4: Implement parseUnits() with cheerio selectors ( #9 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 17:14:01 -07:00
c6beaf8333
SCRAPE-3: Implement fetchPage() with axios, timeout, retry ( #8 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 15:58:07 -07:00
2993d019c5
SCRAPE-2: Add scraper config constants ( #7 )
2026-01-31 20:54:43 -07:00
9dba679221
Add scraperLogger.js with credential redaction ( #6 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-01-28 10:52:00 -07:00
ccda2be551
Update tests to expect wrapped response format
...
- Update all test files to use response.body.data.* instead of
response.body.* to match the API response convention
- Skip SEC-4.3 rate limiting tests (moved to Phase 5)
- All 202 tests now pass
2026-01-22 14:33:01 -07:00
81e5682ab1
Admin Dashboard Backend (Phases 1-4) ( #5 )
...
Deploy Apartment API / deploy (push) Failing after 5m1s
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-01-22 14:11:43 -07:00
