Add scraper indexes and switch to validation collection names
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Lint & Test (pull_request) Successful in 43s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Lint & Test (pull_request) Successful in 43s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add createScraperIndexes() to create indexes on the scraper_runs collection: a compound index on status+startedAt for active job queries and a descending index on startedAt for recent run lookups. Change default collection names from production tables (units_migration_test, unit_prices_migration_test) to dedicated validation collections (units_scraper, unit_prices_scraper). This enables the Node.js scraper to run in parallel with the existing Python scraper during validation without interfering with production data. Update all scraper test files to reference the new default collection names.
This commit is contained in:
@ -730,6 +730,36 @@ async function recordScraperRun(db, runData, logger) {
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// Index Management
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Create indexes on the scraper_runs collection.
|
||||
* Should be called once during application startup.
|
||||
* Idempotent - safe to call multiple times.
|
||||
*
|
||||
* @param {Db} db - MongoDB database instance
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
async function createScraperIndexes(db) {
|
||||
const collection = db.collection(config.COLLECTIONS.SCRAPER_RUNS);
|
||||
|
||||
// Compound index for querying runs by status sorted by most recent
|
||||
await collection.createIndex(
|
||||
{ status: 1, startedAt: -1 },
|
||||
{ name: 'status_startedAt' }
|
||||
);
|
||||
|
||||
// Index for sorting all runs by start time (most recent first)
|
||||
await collection.createIndex(
|
||||
{ startedAt: -1 },
|
||||
{ name: 'startedAt_desc' }
|
||||
);
|
||||
|
||||
console.log('Scraper indexes created successfully');
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// Main Orchestration Function
|
||||
// ============================================================
|
||||
@ -871,6 +901,7 @@ module.exports = {
|
||||
markStaleUnits,
|
||||
updateDailySummary,
|
||||
recordScraperRun,
|
||||
createScraperIndexes,
|
||||
// Export helpers for testing
|
||||
getTodayUTC,
|
||||
getYesterdayUnitCodes,
|
||||
|
||||
Reference in New Issue
Block a user