Implement upsertUnits() bulk operation for apartment units
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 12s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m44s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 12s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m44s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add upsertUnits() to scraperService that performs bulk upserts of apartment unit data using MongoDB bulkWrite with unordered operations. Key behaviors: - Uses unit_code as the unique key for matching existing units - Sets last_scraped timestamp and data_source on every upsert - Uses $setOnInsert for first_seen to preserve original insert time - Handles empty arrays gracefully with early return - Uses ordered:false for parallel execution and partial failure tolerance - Logs result counts (matched, modified, upserted) on success - Propagates errors with detailed logging on failure Includes 23 tests covering bulk operations, upsert behavior, field handling, empty input, error scenarios, and return values. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@ -369,10 +369,72 @@ function convertDataTypes(unit) {
|
||||
};
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// Database Operations
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Upsert unit records to database using bulkWrite.
|
||||
* Each unit is matched by unit_code as the unique key.
|
||||
* Sets last_scraped and data_source on every update.
|
||||
* Sets first_seen only on initial insert via $setOnInsert.
|
||||
*
|
||||
* @param {Db} db - MongoDB database instance
|
||||
* @param {Array<Object>} units - Array of unit objects
|
||||
* @param {Object} logger - Logger instance
|
||||
* @returns {Promise<Object>} Bulk write result
|
||||
*/
|
||||
async function upsertUnits(db, units, logger) {
|
||||
const collection = db.collection(config.COLLECTIONS.UNITS);
|
||||
const now = new Date().toISOString();
|
||||
|
||||
const operations = units.map(unit => ({
|
||||
updateOne: {
|
||||
filter: { unit_code: unit.unit_code },
|
||||
update: {
|
||||
$set: {
|
||||
...unit,
|
||||
last_scraped: now,
|
||||
data_source: 'web_scraper'
|
||||
},
|
||||
$setOnInsert: {
|
||||
first_seen: now
|
||||
}
|
||||
},
|
||||
upsert: true
|
||||
}
|
||||
}));
|
||||
|
||||
if (operations.length === 0) {
|
||||
logger.warn('No units to upsert');
|
||||
return { modifiedCount: 0, upsertedCount: 0 };
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await collection.bulkWrite(operations, { ordered: false });
|
||||
|
||||
logger.info('Units upserted', {
|
||||
matched: result.matchedCount,
|
||||
modified: result.modifiedCount,
|
||||
upserted: result.upsertedCount
|
||||
});
|
||||
|
||||
return result;
|
||||
|
||||
} catch (error) {
|
||||
logger.error('Failed to upsert units', {
|
||||
errorType: error.name,
|
||||
errorMessage: error.message
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
fetchPage,
|
||||
parseUnits,
|
||||
convertDataTypes,
|
||||
upsertUnits,
|
||||
// Export helpers for testing
|
||||
parseInteger,
|
||||
parsePositiveInteger,
|
||||
|
||||
Reference in New Issue
Block a user