apartment-dashboard-api

Author	SHA1	Message	Date
Stephen Minakian	5ef5d5af73	SCRAPE-11: Implement runScrape() main orchestration function Some checks failed CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s Details CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped Details CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s Details CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s Details CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m56s Details CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped Details Add the top-level runScrape() function that coordinates the full scraper pipeline: fetch HTML (with retry), parse units, convert data types, upsert units, insert prices, mark stale units, and update daily summary. Features: - dryRun mode skips all database writes while still parsing/validating - htmlContent parameter allows injecting HTML directly (bypasses fetch) - Calculates newUnitsCount and rentedUnitsCount by diffing against prior state - Records every run to scraper_runs history (success or failure) - Structured logging with jobId correlation throughout the pipeline - Graceful error handling at each pipeline stage Includes 15 tests covering full workflow, dry run, error handling, trigger types, new/rented unit calculation, and empty HTML edge case.	2026-02-06 01:35:15 -07:00
Stephen Minakian	b4978caf31	SCRAPE-9: Implement updateDailySummary() aggregation (#14 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-06 00:15:30 -07:00
Stephen Minakian	d8adfbf90c	SCRAPE-8: Implement markStaleUnits() (#13 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 23:48:57 -07:00
Stephen Minakian	2b53288fbe	SCRAPE-7: Implement insertPrices() bulk operation (#12 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 22:32:02 -07:00
Stephen Minakian	4863dc1824	SCRAPE-6: Implement upsertUnits() bulk operation (#11 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 22:10:54 -07:00
Stephen Minakian	0cbeaaf9d5	SCRAPE-5: Implement convertDataTypes() with type parsing helpers (#10 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 21:33:58 -07:00
Stephen Minakian	072e80d069	SCRAPE-4: Implement parseUnits() with cheerio selectors (#9 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 17:14:01 -07:00
Stephen Minakian	c6beaf8333	SCRAPE-3: Implement fetchPage() with axios, timeout, retry (#8 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-02-05 15:58:07 -07:00
Stephen Minakian	9dba679221	Add scraperLogger.js with credential redaction (#6 ) Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-01-28 10:52:00 -07:00
Stephen Minakian	81e5682ab1	Admin Dashboard Backend (Phases 1-4) (#5 ) Some checks failed Deploy Apartment API / deploy (push) Failing after 5m1s Details Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-01-22 14:11:43 -07:00
Stephen Minakian	ef4ddc0de0	Add Google OAuth authentication system (#4 ) Some checks failed Deploy Apartment API / deploy (push) Failing after 1s Details Co-authored-by: Stephen Minakian <stephenminakian@gmail.com> Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>	2026-01-21 19:15:22 -07:00

11 Commits