apartment-dashboard-api

Author	SHA1	Message	Date
Stephen Minakian	5e1f1f8041	SCRAPE-11: Implement runScrape() main orchestration function Some checks failed CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s Details CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s Details CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 10m0s Details CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s Details CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped Details CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped Details Add the top-level runScrape() function that coordinates the full scraper pipeline: fetch HTML (with retry), parse units, convert data types, upsert units, insert prices, mark stale units, and update daily summary. Features: - dryRun mode skips all database writes while still parsing/validating - htmlContent parameter allows injecting HTML directly (bypasses fetch) - Calculates newUnitsCount and rentedUnitsCount by diffing against prior state - Records every run to scraper_runs history (success or failure) - Structured logging with jobId correlation throughout the pipeline - Graceful error handling at each pipeline stage Includes 15 tests covering full workflow, dry run, error handling, trigger types, new/rented unit calculation, and empty HTML edge case.	2026-02-06 09:16:23 -07:00
Stephen Minakian	d1f717891a	SCRAPE-10: Implement recordScraperRun() for scraper_runs (#15 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-06 02:00:19 -07:00
Stephen Minakian	b4978caf31	SCRAPE-9: Implement updateDailySummary() aggregation (#14 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-06 00:15:30 -07:00
Stephen Minakian	d8adfbf90c	SCRAPE-8: Implement markStaleUnits() (#13 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 23:48:57 -07:00
Stephen Minakian	2b53288fbe	SCRAPE-7: Implement insertPrices() bulk operation (#12 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 22:32:02 -07:00
Stephen Minakian	4863dc1824	SCRAPE-6: Implement upsertUnits() bulk operation (#11 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 22:10:54 -07:00
Stephen Minakian	0cbeaaf9d5	SCRAPE-5: Implement convertDataTypes() with type parsing helpers (#10 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 21:33:58 -07:00
Stephen Minakian	072e80d069	SCRAPE-4: Implement parseUnits() with cheerio selectors (#9 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 17:14:01 -07:00
Stephen Minakian	c6beaf8333	SCRAPE-3: Implement fetchPage() with axios, timeout, retry (#8 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 15:58:07 -07:00
stephen	2993d019c5	SCRAPE-2: Add scraper config constants (#7 )	2026-01-31 20:54:43 -07:00
Stephen Minakian	9dba679221	Add scraperLogger.js with credential redaction (#6 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-01-28 10:52:00 -07:00

11 Commits