stephen/apartment-dashboard-api
1
0
Fork 0
Code Issues 1 Pull Requests Actions Releases Wiki Activity

SCRAPE-6: Implement upsertUnits() bulk operation #11

Merged
stephen merged 1 commits from scraper/upsert-units into dev 2026-02-05 22:10:54 -07:00
Conversation 0 Commits 1 Files Changed 2 +560
stephen commented 2026-02-05 22:00:16 -07:00
Owner
Copy Link

Implement upsertUnits() in scraperService.js using MongoDB bulkWrite with unordered operations. Uses unit_code as unique key, sets last_scraped and data_source on every upsert, preserves first_seen via setOnInsert. Handles empty arrays, logs results, and re-throws errors. Includes 23 tests covering all behaviors.

Implement upsertUnits() in scraperService.js using MongoDB bulkWrite with unordered operations. Uses unit_code as unique key, sets last_scraped and data_source on every upsert, preserves first_seen via setOnInsert. Handles empty arrays, logs results, and re-throws errors. Includes 23 tests covering all behaviors.
stephen added 1 commit 2026-02-05 22:00:17 -07:00
Implement upsertUnits() bulk operation for apartment units
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 12s
Details
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
Details
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m44s
Details
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
Details
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
Details
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Details
31c3c0c8ee 
Add upsertUnits() to scraperService that performs bulk upserts of
apartment unit data using MongoDB bulkWrite with unordered operations.

Key behaviors:
- Uses unit_code as the unique key for matching existing units
- Sets last_scraped timestamp and data_source on every upsert
- Uses $setOnInsert for first_seen to preserve original insert time
- Handles empty arrays gracefully with early return
- Uses ordered:false for parallel execution and partial failure tolerance
- Logs result counts (matched, modified, upserted) on success
- Propagates errors with detailed logging on failure

Includes 23 tests covering bulk operations, upsert behavior, field
handling, empty input, error scenarios, and return values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
stephen merged commit 4863dc1824 into dev 2026-02-05 22:10:54 -07:00
stephen deleted branch scraper/upsert-units 2026-02-05 22:10:54 -07:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
stephen
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: stephen/apartment-dashboard-api#11
No description provided.
Delete Branch "scraper/upsert-units"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?