The anchore/sbom-action GitHub Action uses upload-artifact@v4 internally,
which is not supported on GHES. Install Syft directly via CLI and run it
as a shell command to generate the SBOM without the artifact upload.
Truncate lint/test output to 10000 chars at the source (CI job) before
writing to GITHUB_OUTPUT, instead of in the downstream notify job.
Previously the full output was passed as an env var between jobs, which
could exceed Linux's ARG_MAX limit and prevent bash from launching.
Phase tests share a single MongoMemoryServer database and conflict
when run in parallel. Sequential execution is needed until tests
use isolated databases per file.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Combine lint and test into a single 'ci' job (eliminates duplicate
checkout + npm ci, saving ~60-90s)
- Remove --runInBand flag so Jest parallelizes across worker pools
- Remove unused mongo:7 service container (tests use MongoMemoryServer)
- Fix failure detection: check step outcomes instead of job result,
which was always 'success' due to continue-on-error
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Implements the top-level runScrape() orchestration function that coordinates the entire scraper pipeline end-to-end.
### What it does
- Full pipeline orchestration: Calls fetchPage, parseUnits, convertDataTypes, upsertUnits, insertPrices, markStaleUnits, updateDailySummary in sequence
- dryRun mode: When enabled, parses and validates HTML but skips all database writes
- htmlContent injection: Accepts raw HTML directly, bypassing the fetch step
- New/rented unit calculation: Diffs currently scraped units against previously active units to determine newUnitsCount and rentedUnitsCount for the daily summary
- Run history recording: Every scrape (success or failure) is recorded to the scraper_runs collection via recordScraperRun()
- Structured logging: All pipeline stages log with jobId correlation for traceability
- Error resilience: Catches and handles errors at each stage, ensuring partial failures are logged and recorded
### Test coverage (15 tests)
- Full workflow with mocked dependencies
- Result structure validation and jobId generation
- dryRun mode skips DB writes
- htmlContent bypasses fetch
- Success and failure history recording
- Fetch error handling with retry exhaustion
- Database operation error handling
- New/rented unit count calculation
- Default and scheduled trigger types
- Empty HTML (no units) edge case
Reviewed-on: #16
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com>
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com>
- Add ESLint 9 with flat config for Node.js linting
- Add lint and lint:fix npm scripts
- Add lint job to CI/CD pipeline
- Add notify job to send test/lint results to n8n webhook
- Webhook reports pass/fail status with failure details
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
