Implement recordScraperRun() for scraper_runs collection
Some checks failed
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m36s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m51s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped

Add recordScraperRun(db, runData) to scraperService that persists
scraper execution history to the scraper_runs collection. The function
inserts a document containing all run metadata (jobId, trigger, status,
duration, unit metrics, timing, errors) along with a recordedAt
timestamp.

Designed for use in finally blocks: errors during recording are caught,
logged, and swallowed so they never interrupt the scraper pipeline.
Returns the insertOne result on success or null on failure.

Includes 17 tests covering insertion behavior, field preservation,
graceful error handling, return values, and recordedAt semantics.
This commit is contained in:
2026-02-06 00:31:07 -07:00
parent b4978caf31
commit d4bb1088a2
2 changed files with 341 additions and 0 deletions

View File

@ -619,6 +619,33 @@ async function updateDailySummary(db, summaryData, logger) {
}
}
/**
* Record scraper run to history collection.
* Called in the finally block of runScrape() to persist run metadata.
* This function must NOT throw errors - history recording should never break the scraper.
*
* @param {Db} db - MongoDB database instance
* @param {Object} runData - Run data to record (jobId, trigger, status, duration, etc.)
* @returns {Promise<Object|null>} Insert result, or null on failure
*/
async function recordScraperRun(db, runData) {
const collection = db.collection(config.COLLECTIONS.SCRAPER_RUNS);
try {
const result = await collection.insertOne({
...runData,
recordedAt: new Date()
});
return result;
} catch (error) {
// Log but don't throw - recording history should not break scraper
console.error('Failed to record scraper run:', error.message);
return null;
}
}
module.exports = {
fetchPage,
parseUnits,
@ -627,6 +654,7 @@ module.exports = {
insertPrices,
markStaleUnits,
updateDailySummary,
recordScraperRun,
// Export helpers for testing
getYesterday,
parseInteger,