Implement recordScraperRun() for scraper_runs collection
Some checks failed
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m36s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m51s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Some checks failed
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m36s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m51s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add recordScraperRun(db, runData) to scraperService that persists scraper execution history to the scraper_runs collection. The function inserts a document containing all run metadata (jobId, trigger, status, duration, unit metrics, timing, errors) along with a recordedAt timestamp. Designed for use in finally blocks: errors during recording are caught, logged, and swallowed so they never interrupt the scraper pipeline. Returns the insertOne result on success or null on failure. Includes 17 tests covering insertion behavior, field preservation, graceful error handling, return values, and recordedAt semantics.
This commit is contained in:
@ -619,6 +619,33 @@ async function updateDailySummary(db, summaryData, logger) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Record scraper run to history collection.
|
||||
* Called in the finally block of runScrape() to persist run metadata.
|
||||
* This function must NOT throw errors - history recording should never break the scraper.
|
||||
*
|
||||
* @param {Db} db - MongoDB database instance
|
||||
* @param {Object} runData - Run data to record (jobId, trigger, status, duration, etc.)
|
||||
* @returns {Promise<Object|null>} Insert result, or null on failure
|
||||
*/
|
||||
async function recordScraperRun(db, runData) {
|
||||
const collection = db.collection(config.COLLECTIONS.SCRAPER_RUNS);
|
||||
|
||||
try {
|
||||
const result = await collection.insertOne({
|
||||
...runData,
|
||||
recordedAt: new Date()
|
||||
});
|
||||
|
||||
return result;
|
||||
|
||||
} catch (error) {
|
||||
// Log but don't throw - recording history should not break scraper
|
||||
console.error('Failed to record scraper run:', error.message);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
fetchPage,
|
||||
parseUnits,
|
||||
@ -627,6 +654,7 @@ module.exports = {
|
||||
insertPrices,
|
||||
markStaleUnits,
|
||||
updateDailySummary,
|
||||
recordScraperRun,
|
||||
// Export helpers for testing
|
||||
getYesterday,
|
||||
parseInteger,
|
||||
|
||||
Reference in New Issue
Block a user