stephen/apartment-dashboard-api
1
0
Fork 0
Code Issues 1 Pull Requests Actions Releases Wiki Activity

SCRAPE-7: Implement insertPrices() bulk operation #12

Merged
stephen merged 1 commits from scraper/insert-prices into dev 2026-02-05 22:32:03 -07:00
Conversation 0 Commits 1 Files Changed 2 +646
stephen commented 2026-02-05 22:21:23 -07:00
Owner
Copy Link

Implements insertPrices() for bulk upserting daily unit price records. Uses bulkWrite with compound key for idempotent re-runs, filters null prices, includes 30 tests.

Implements insertPrices() for bulk upserting daily unit price records. Uses bulkWrite with compound key for idempotent re-runs, filters null prices, includes 30 tests.
stephen added 1 commit 2026-02-05 22:21:24 -07:00
Implement insertPrices() bulk operation for daily price records
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
Details
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m47s
Details
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
Details
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
Details
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
Details
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Details
406672c16b 
Add insertPrices() to scraperService that performs bulk upsert of
daily unit price records to the prices collection. Key behaviors:

- Uses bulkWrite with updateOne/upsert ops for atomic batch writes
- Compound key (unit_code + date_checked) ensures idempotent re-runs
- Filters out units with null prices before writing
- Stores price, last_updated, and data_source fields per record
- Uses ordered:false for parallel write performance
- Logs warnings for empty/all-null input without hitting the database
- Propagates bulkWrite errors with detailed logging

Includes 30 tests covering filtering, upsert behavior, composite key
logic, field validation, idempotency, error handling, and logging.
stephen merged commit 2b53288fbe into dev 2026-02-05 22:32:03 -07:00
stephen deleted branch scraper/insert-prices 2026-02-05 22:32:03 -07:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
stephen
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: stephen/apartment-dashboard-api#12
No description provided.
Delete Branch "scraper/insert-prices"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?