Add scraper indexes and switch to validation collection names
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Lint & Test (pull_request) Successful in 43s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped

Add createScraperIndexes() to create indexes on the scraper_runs
collection: a compound index on status+startedAt for active job queries
and a descending index on startedAt for recent run lookups.

Change default collection names from production tables
(units_migration_test, unit_prices_migration_test) to dedicated
validation collections (units_scraper, unit_prices_scraper). This
enables the Node.js scraper to run in parallel with the existing Python
scraper during validation without interfering with production data.

Update all scraper test files to reference the new default collection
names.
This commit is contained in:
2026-02-07 12:04:37 -07:00
parent 80d44bca04
commit 2e3ef0580c
8 changed files with 101 additions and 20 deletions

View File

@ -730,6 +730,36 @@ async function recordScraperRun(db, runData, logger) {
}
}
// ============================================================
// Index Management
// ============================================================
/**
* Create indexes on the scraper_runs collection.
* Should be called once during application startup.
* Idempotent - safe to call multiple times.
*
* @param {Db} db - MongoDB database instance
* @returns {Promise<void>}
*/
async function createScraperIndexes(db) {
const collection = db.collection(config.COLLECTIONS.SCRAPER_RUNS);
// Compound index for querying runs by status sorted by most recent
await collection.createIndex(
{ status: 1, startedAt: -1 },
{ name: 'status_startedAt' }
);
// Index for sorting all runs by start time (most recent first)
await collection.createIndex(
{ startedAt: -1 },
{ name: 'startedAt_desc' }
);
console.log('Scraper indexes created successfully');
}
// ============================================================
// Main Orchestration Function
// ============================================================
@ -871,6 +901,7 @@ module.exports = {
markStaleUnits,
updateDailySummary,
recordScraperRun,
createScraperIndexes,
// Export helpers for testing
getTodayUTC,
getYesterdayUnitCodes,