Implement upsertUnits() bulk operation for apartment units
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 12s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m44s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped

Add upsertUnits() to scraperService that performs bulk upserts of
apartment unit data using MongoDB bulkWrite with unordered operations.

Key behaviors:
- Uses unit_code as the unique key for matching existing units
- Sets last_scraped timestamp and data_source on every upsert
- Uses $setOnInsert for first_seen to preserve original insert time
- Handles empty arrays gracefully with early return
- Uses ordered:false for parallel execution and partial failure tolerance
- Logs result counts (matched, modified, upserted) on success
- Propagates errors with detailed logging on failure

Includes 23 tests covering bulk operations, upsert behavior, field
handling, empty input, error scenarios, and return values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-05 21:55:01 -07:00
parent 0cbeaaf9d5
commit 31c3c0c8ee
2 changed files with 560 additions and 0 deletions

View File

@ -369,10 +369,72 @@ function convertDataTypes(unit) {
};
}
// ============================================================
// Database Operations
// ============================================================
/**
* Upsert unit records to database using bulkWrite.
* Each unit is matched by unit_code as the unique key.
* Sets last_scraped and data_source on every update.
* Sets first_seen only on initial insert via $setOnInsert.
*
* @param {Db} db - MongoDB database instance
* @param {Array<Object>} units - Array of unit objects
* @param {Object} logger - Logger instance
* @returns {Promise<Object>} Bulk write result
*/
async function upsertUnits(db, units, logger) {
const collection = db.collection(config.COLLECTIONS.UNITS);
const now = new Date().toISOString();
const operations = units.map(unit => ({
updateOne: {
filter: { unit_code: unit.unit_code },
update: {
$set: {
...unit,
last_scraped: now,
data_source: 'web_scraper'
},
$setOnInsert: {
first_seen: now
}
},
upsert: true
}
}));
if (operations.length === 0) {
logger.warn('No units to upsert');
return { modifiedCount: 0, upsertedCount: 0 };
}
try {
const result = await collection.bulkWrite(operations, { ordered: false });
logger.info('Units upserted', {
matched: result.matchedCount,
modified: result.modifiedCount,
upserted: result.upsertedCount
});
return result;
} catch (error) {
logger.error('Failed to upsert units', {
errorType: error.name,
errorMessage: error.message
});
throw error;
}
}
module.exports = {
fetchPage,
parseUnits,
convertDataTypes,
upsertUnits,
// Export helpers for testing
parseInteger,
parsePositiveInteger,