SCRAPE-31: Add scraper indexes and validation collections #30

Merged

stephen merged 2 commits from scraper/indexes into dev 2026-02-07 16:50:51 -07:00

Conversation 1 Commits 2 Files Changed 8 +116 -20

stephen commented 2026-02-07 12:05:39 -07:00

Owner

Copy Link

Added createScraperIndexes() for scraper_runs collection with compound index on status+startedAt and descending index on startedAt. Changed default collection names from production (units_migration_test, unit_prices_migration_test) to validation (units_scraper, unit_prices_scraper) for parallel validation. Updated all scraper test files.

Added createScraperIndexes() for scraper_runs collection with compound index on status+startedAt and descending index on startedAt. Changed default collection names from production (units_migration_test, unit_prices_migration_test) to validation (units_scraper, unit_prices_scraper) for parallel validation. Updated all scraper test files.

stephen added 1 commit 2026-02-07 12:05:39 -07:00

Add scraper indexes and switch to validation collection names ... 2e3ef0580c

Add createScraperIndexes() to create indexes on the scraper_runs
collection: a compound index on status+startedAt for active job queries
and a descending index on startedAt for recent run lookups.

Change default collection names from production tables
(units_migration_test, unit_prices_migration_test) to dedicated
validation collections (units_scraper, unit_prices_scraper). This
enables the Node.js scraper to run in parallel with the existing Python
scraper during validation without interfering with production data.

Update all scraper test files to reference the new default collection
names.

stephen commented 2026-02-07 12:07:11 -07:00

Author

Owner

Copy Link

❌ Changes Requested

The createScraperIndexes function contains a console.log statement in production code. It should use the logger pattern consistent with the rest of the service.

Issues:

services/scraperService.js:762
Remove console.log('Scraper indexes created successfully'). Production code must not use console.log. Either accept a logger parameter (consistent with all other functions in this service like recordScraperRun(db, runData, logger)) and call logger.info(...), or remove the log statement entirely. The function signature should be createScraperIndexes(db, logger) to match the existing pattern.

## ❌ Changes Requested The `createScraperIndexes` function contains a `console.log` statement in production code. It should use the logger pattern consistent with the rest of the service. ### Issues: **services/scraperService.js:762** Remove `console.log('Scraper indexes created successfully')`. Production code must not use `console.log`. Either accept a `logger` parameter (consistent with all other functions in this service like `recordScraperRun(db, runData, logger)`) and call `logger.info(...)`, or remove the log statement entirely. The function signature should be `createScraperIndexes(db, logger)` to match the existing pattern.

stephen added 1 commit 2026-02-07 16:49:24 -07:00

Use logger parameter instead of console.log in createScraperIndexes ... fe24154518

Accept logger as second parameter to match the pattern used by all other
functions in scraperService.js (recordScraperRun, upsertUnits, etc.).
Replaces console.log with logger.info for production code consistency.

stephen merged commit 4ca202068f into dev 2026-02-07 16:50:51 -07:00

stephen deleted branch scraper/indexes 2026-02-07 16:50:51 -07:00

stephen referenced this issue from a commit 2026-02-07 16:50:51 -07:00

SCRAPE-31: Add scraper indexes and validation collections (#30)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

stephen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stephen/apartment-dashboard-api#30

No description provided.