SCRAPE-8: Implement markStaleUnits() #13

Merged

stephen merged 1 commits from scraper/mark-stale into dev 2026-02-05 23:48:57 -07:00

Conversation 0 Commits 1 Files Changed 2 +492

stephen commented 2026-02-05 23:38:08 -07:00

Owner

Copy Link

Summary

• Implement markStaleUnits() in scraperService.js that marks units as unavailable when they are not found in the current scrape results
• Uses MongoDB updateMany with a $nin filter on unit_code to efficiently target only units that were previously available but no longer appear in the scraped data
• Sets available to false and records marked_stale_date for affected units
• Handles edge cases: empty currentUnitCodes set, already unavailable units, and database errors with proper logging

Test Coverage

18 tests covering filter criteria, $set fields, units in currentCodes not affected, already unavailable units, error handling, logging, empty currentUnitCodes edge case, and return value verification.

Test plan

• All 18 unit tests pass
• Verify $nin filter correctly excludes current unit codes
• Verify only available: true units are affected
• Verify error propagation from MongoDB failures

## Summary - Implement markStaleUnits() in scraperService.js that marks units as unavailable when they are not found in the current scrape results - Uses MongoDB updateMany with a $nin filter on unit_code to efficiently target only units that were previously available but no longer appear in the scraped data - Sets available to false and records marked_stale_date for affected units - Handles edge cases: empty currentUnitCodes set, already unavailable units, and database errors with proper logging ## Test Coverage 18 tests covering filter criteria, $set fields, units in currentCodes not affected, already unavailable units, error handling, logging, empty currentUnitCodes edge case, and return value verification. ## Test plan - [x] All 18 unit tests pass - [x] Verify $nin filter correctly excludes current unit codes - [x] Verify only available: true units are affected - [x] Verify error propagation from MongoDB failures

stephen added 1 commit 2026-02-05 23:38:08 -07:00

SCRAPE-8: Implement markStaleUnits() to flag unavailable units ... 6eaf79f662

Add markStaleUnits() that marks units as unavailable when they are not
present in the current scrape results. Uses MongoDB updateMany with a
$nin filter on unit_code to efficiently update only units that were
previously available but no longer appear in the scraped data. Sets
available to false and records marked_stale_date.

Includes 18 tests covering filter criteria, $set fields, already
unavailable units, empty currentUnitCodes, error handling, logging,
and return value verification.

stephen merged commit d8adfbf90c into dev 2026-02-05 23:48:57 -07:00

stephen deleted branch scraper/mark-stale 2026-02-05 23:48:57 -07:00

stephen referenced this issue from a commit 2026-02-05 23:48:57 -07:00

SCRAPE-8: Implement markStaleUnits() (#13)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

stephen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stephen/apartment-dashboard-api#13

No description provided.