Commit Graph

8 Commits

Author SHA1 Message Date
31c3c0c8ee Implement upsertUnits() bulk operation for apartment units
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 12s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m44s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add upsertUnits() to scraperService that performs bulk upserts of
apartment unit data using MongoDB bulkWrite with unordered operations.

Key behaviors:
- Uses unit_code as the unique key for matching existing units
- Sets last_scraped timestamp and data_source on every upsert
- Uses $setOnInsert for first_seen to preserve original insert time
- Handles empty arrays gracefully with early return
- Uses ordered:false for parallel execution and partial failure tolerance
- Logs result counts (matched, modified, upserted) on success
- Propagates errors with detailed logging on failure

Includes 23 tests covering bulk operations, upsert behavior, field
handling, empty input, error scenarios, and return values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-05 21:55:01 -07:00
0cbeaaf9d5 SCRAPE-5: Implement convertDataTypes() with type parsing helpers (#10)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 21:33:58 -07:00
072e80d069 SCRAPE-4: Implement parseUnits() with cheerio selectors (#9)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 17:14:01 -07:00
c6beaf8333 SCRAPE-3: Implement fetchPage() with axios, timeout, retry (#8)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-02-05 15:58:07 -07:00
2993d019c5 SCRAPE-2: Add scraper config constants (#7) 2026-01-31 20:54:43 -07:00
9dba679221 Add scraperLogger.js with credential redaction (#6)
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-01-28 10:52:00 -07:00
ccda2be551 Update tests to expect wrapped response format
- Update all test files to use response.body.data.* instead of
  response.body.* to match the API response convention
- Skip SEC-4.3 rate limiting tests (moved to Phase 5)
- All 202 tests now pass
2026-01-22 14:33:01 -07:00
81e5682ab1 Admin Dashboard Backend (Phases 1-4) (#5)
Some checks failed
Deploy Apartment API / deploy (push) Failing after 5m1s
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
2026-01-22 14:11:43 -07:00