SCRAPE-31: Add scraper indexes and validation collections #30

Merged
stephen merged 2 commits from scraper/indexes into dev 2026-02-07 16:50:51 -07:00

2 Commits

Author SHA1 Message Date
fe24154518 Use logger parameter instead of console.log in createScraperIndexes
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Lint & Test (pull_request) Successful in 44s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Accept logger as second parameter to match the pattern used by all other
functions in scraperService.js (recordScraperRun, upsertUnits, etc.).
Replaces console.log with logger.info for production code consistency.
2026-02-07 16:49:20 -07:00
2e3ef0580c Add scraper indexes and switch to validation collection names
All checks were successful
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Lint & Test (pull_request) Successful in 43s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add createScraperIndexes() to create indexes on the scraper_runs
collection: a compound index on status+startedAt for active job queries
and a descending index on startedAt for recent run lookups.

Change default collection names from production tables
(units_migration_test, unit_prices_migration_test) to dedicated
validation collections (units_scraper, unit_prices_scraper). This
enables the Node.js scraper to run in parallel with the existing Python
scraper during validation without interfering with production data.

Update all scraper test files to reference the new default collection
names.
2026-02-07 12:04:37 -07:00