SCRAPE-8: Implement markStaleUnits() to flag unavailable units
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m47s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped

Add markStaleUnits() that marks units as unavailable when they are not
present in the current scrape results. Uses MongoDB updateMany with a
$nin filter on unit_code to efficiently update only units that were
previously available but no longer appear in the scraped data. Sets
available to false and records marked_stale_date.

Includes 18 tests covering filter criteria, $set fields, already
unavailable units, empty currentUnitCodes, error handling, logging,
and return value verification.
This commit is contained in:
2026-02-05 23:37:41 -07:00
parent 2b53288fbe
commit 6eaf79f662
2 changed files with 492 additions and 0 deletions

View File

@ -493,12 +493,55 @@ async function insertPrices(db, units, date, logger) {
}
}
/**
* Mark units not in current scrape as stale/unavailable.
* Uses updateMany to set available: false and marked_stale_date
* for all units whose unit_code is NOT in the current scrape
* and that are currently available.
*
* @param {Db} db - MongoDB database instance
* @param {Set<string>} currentUnitCodes - Unit codes from current scrape
* @param {string} date - Current date in YYYY-MM-DD format
* @param {Object} logger - Logger instance
* @returns {Promise<Object>} Update result
*/
async function markStaleUnits(db, currentUnitCodes, date, logger) {
const collection = db.collection(config.COLLECTIONS.UNITS);
try {
const result = await collection.updateMany(
{
unit_code: { $nin: [...currentUnitCodes] },
available: true
},
{
$set: {
available: false,
marked_stale_date: date
}
}
);
logger.info('Stale units marked', { count: result.modifiedCount });
return result;
} catch (error) {
logger.error('Failed to mark stale units', {
errorType: error.name,
errorMessage: error.message
});
throw error;
}
}
module.exports = {
fetchPage,
parseUnits,
convertDataTypes,
upsertUnits,
insertPrices,
markStaleUnits,
// Export helpers for testing
parseInteger,
parsePositiveInteger,