stephen/apartment-dashboard-api
1
0
Fork 0
Code Issues 1 Pull Requests Actions Releases Wiki Activity

SCRAPE-23: Sanitize error messages in scraper_runs #27

Merged
stephen merged 1 commits from scraper/sanitize-errors into dev 2026-02-06 23:44:02 -07:00
Conversation 0 Commits 1 Files Changed 2 +273 -4
stephen commented 2026-02-06 23:42:32 -07:00
Owner
Copy Link

Summary

  • Added sanitizeError() and sanitizeMessage() functions to strip sensitive information from error messages before they are stored in scraper_runs history
  • Strips Unix and Windows file paths, MongoDB connection strings, and credential patterns
  • Preserves error type/name and useful debugging descriptions
  • Applied in runScrape() catch block before calling recordScraperRun()
  • 18 new tests, 33 total scraperService tests passing

Files Changed

  • services/scraperService.js - Added sanitizeMessage(), sanitizeError(), and integrated into error handling
  • tests/scraper/scraperService.test.js - Added 18 tests for sanitization

Test Plan

  • All 33 scraperService tests pass
  • All scraper test suites pass
  • Rebased on latest dev
## Summary - Added sanitizeError() and sanitizeMessage() functions to strip sensitive information from error messages before they are stored in scraper_runs history - Strips Unix and Windows file paths, MongoDB connection strings, and credential patterns - Preserves error type/name and useful debugging descriptions - Applied in runScrape() catch block before calling recordScraperRun() - 18 new tests, 33 total scraperService tests passing ## Files Changed - services/scraperService.js - Added sanitizeMessage(), sanitizeError(), and integrated into error handling - __tests__/scraper/scraperService.test.js - Added 18 tests for sanitization ## Test Plan - All 33 scraperService tests pass - All scraper test suites pass - Rebased on latest dev
stephen added 1 commit 2026-02-06 23:42:32 -07:00
feat: sanitize error messages before recording to scraper_runs
All checks were successful
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 2s
Details
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
Details
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Details
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
Details
CI/CD Pipeline - Apartment API / Lint & Test (pull_request) Successful in 44s
Details
41e252410e 
Add sanitizeError() and sanitizeMessage() functions that strip sensitive
information from error messages before they are stored in scraper_runs
history. This prevents accidental exposure of infrastructure details in
the database.

Sanitization covers:
- Unix and Windows file paths (e.g., /home/deploy/app/..., C:\Users\...)
- MongoDB connection strings (mongodb:// and mongodb+srv://)
- Credential patterns (API_KEY=, password=, secret=, token=)
- Stack trace file path references

The error type/name (e.g., MongoServerError, TypeError) and general
debugging description are preserved to maintain diagnostic usefulness.

Applied in runScrape() catch block before calling recordScraperRun(),
ensuring only sanitized messages reach the database.

Added 18 new tests covering all sanitization categories: file paths,
connection strings, credentials, error type preservation, description
preservation, stack trace removal, and integration with recordScraperRun.
stephen merged commit e70f7429ea into dev 2026-02-06 23:44:02 -07:00
stephen deleted branch scraper/sanitize-errors 2026-02-06 23:44:02 -07:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
stephen
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: stephen/apartment-dashboard-api#27
No description provided.
Delete Branch "scraper/sanitize-errors"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?