stephen/apartment-dashboard-api
1
0
Fork 0
Code Issues 1 Pull Requests Actions Releases Wiki Activity
47 Commits 3 Branches 0 Tags
2b20a59512d4af84117023d0f12e18963b5b64f1
Commit Graph

21 Commits

Author SHA1 Message Date
Stephen Minakian
2b20a59512 Add comprehensive auth middleware test coverage for scraper routes
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
Details
CI/CD Pipeline - Apartment API / Lint & Test (pull_request) Successful in 41s
Details
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
Details
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
Details
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Details 
Add 29 new tests verifying requireAuth and requireAdmin middleware
behavior across all 3 scraper endpoints (POST /run, GET /status,
GET /history). Test coverage includes:

- Missing JWT token returns 401
- Invalid JWT token returns 401
- Expired JWT token returns 401
- Token signed with wrong secret returns 401
- Disabled user returns 401
- Non-existent user returns 401
- Non-admin user returns 403
- Valid admin user receives expected 200/202 responses
- Middleware ordering: auth failures return 401 before admin check

Total scraperRoutes tests: 81 (52 existing + 29 new)
 2026-02-06 23:06:35 -07:00
Stephen Minakian
d0538334a5 SCRAPE-18: Implement GET /admin/scraper/history endpoint (#23) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 22:34:46 -07:00
Stephen Minakian
9d0b9debbb SCRAPE-17: Implement GET /admin/scraper/status endpoint (#22) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 22:20:06 -07:00
Stephen Minakian
daa234428e SCRAPE-16: Implement POST /admin/scraper/run endpoint (#21) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 22:07:10 -07:00
Stephen Minakian
c6d480a870 SCRAPE-15: Add getNextScheduledRun() test coverage (#20) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 18:21:29 -07:00
Stephen Minakian
0c07baa977 SCRAPE-14: Add graceful shutdown handling (#19) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 18:15:24 -07:00
Stephen Minakian
9d789d38fd SCRAPE-13: Create node-cron job initialization (#18) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 17:28:53 -07:00
Stephen Minakian
dd0269eb30 SCRAPE-12: Implement in-process mutex (isScraperRunning flag) (#17) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 16:27:29 -07:00
Stephen Minakian
0ad4c98abe SCRAPE-11: Create main runScraper() orchestration function (#16) 
## Summary

Implements the top-level runScrape() orchestration function that coordinates the entire scraper pipeline end-to-end.

### What it does

- Full pipeline orchestration: Calls fetchPage, parseUnits, convertDataTypes, upsertUnits, insertPrices, markStaleUnits, updateDailySummary in sequence
- dryRun mode: When enabled, parses and validates HTML but skips all database writes
- htmlContent injection: Accepts raw HTML directly, bypassing the fetch step
- New/rented unit calculation: Diffs currently scraped units against previously active units to determine newUnitsCount and rentedUnitsCount for the daily summary
- Run history recording: Every scrape (success or failure) is recorded to the scraper_runs collection via recordScraperRun()
- Structured logging: All pipeline stages log with jobId correlation for traceability
- Error resilience: Catches and handles errors at each stage, ensuring partial failures are logged and recorded

### Test coverage (15 tests)

- Full workflow with mocked dependencies
- Result structure validation and jobId generation
- dryRun mode skips DB writes
- htmlContent bypasses fetch
- Success and failure history recording
- Fetch error handling with retry exhaustion
- Database operation error handling
- New/rented unit count calculation
- Default and scheduled trigger types
- Empty HTML (no units) edge case

Reviewed-on: #16
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 09:45:32 -07:00
Stephen Minakian
d1f717891a SCRAPE-10: Implement recordScraperRun() for scraper_runs (#15) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 02:00:19 -07:00
Stephen Minakian
b4978caf31 SCRAPE-9: Implement updateDailySummary() aggregation (#14) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-06 00:15:30 -07:00
Stephen Minakian
d8adfbf90c SCRAPE-8: Implement markStaleUnits() (#13) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-05 23:48:57 -07:00
Stephen Minakian
2b53288fbe SCRAPE-7: Implement insertPrices() bulk operation (#12) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-05 22:32:02 -07:00
Stephen Minakian
4863dc1824 SCRAPE-6: Implement upsertUnits() bulk operation (#11) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-05 22:10:54 -07:00
Stephen Minakian
0cbeaaf9d5 SCRAPE-5: Implement convertDataTypes() with type parsing helpers (#10) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-05 21:33:58 -07:00
Stephen Minakian
072e80d069 SCRAPE-4: Implement parseUnits() with cheerio selectors (#9) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-05 17:14:01 -07:00
Stephen Minakian
c6beaf8333 SCRAPE-3: Implement fetchPage() with axios, timeout, retry (#8) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-02-05 15:58:07 -07:00
stephen
2993d019c5 SCRAPE-2: Add scraper config constants (#7) 2026-01-31 20:54:43 -07:00
Stephen Minakian
9dba679221 Add scraperLogger.js with credential redaction (#6) 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-01-28 10:52:00 -07:00
Stephen Minakian
ccda2be551 Update tests to expect wrapped response format 
- Update all test files to use response.body.data.* instead of
  response.body.* to match the API response convention
- Skip SEC-4.3 rate limiting tests (moved to Phase 5)
- All 202 tests now pass
 2026-01-22 14:33:01 -07:00
Stephen Minakian
81e5682ab1 Admin Dashboard Backend (Phases 1-4) (#5)
Some checks failed
Deploy Apartment API / deploy (push) Failing after 5m1s
Details 
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
 2026-01-22 14:11:43 -07:00