08d7a4b0b0
Fix SIGPIPE in CI webhook step by using bash string slicing
...
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m35s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 9m58s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Successful in 3s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Replace echo | head -c pipe with ${VAR:0:10000} bash substring
to avoid exit code 141 when output exceeds truncation limit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-06 09:29:11 -07:00
5e1f1f8041
SCRAPE-11: Implement runScrape() main orchestration function
...
CI/CD Pipeline - Apartment API / Scan Dependencies (pull_request) Successful in 13s
CI/CD Pipeline - Apartment API / Run Linting (pull_request) Successful in 9m37s
CI/CD Pipeline - Apartment API / Run Tests (pull_request) Successful in 10m0s
CI/CD Pipeline - Apartment API / Send Webhook Notification (pull_request) Failing after 2s
CI/CD Pipeline - Apartment API / Build & Push Image (pull_request) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (pull_request) Has been skipped
Add the top-level runScrape() function that coordinates the full scraper
pipeline: fetch HTML (with retry), parse units, convert data types, upsert
units, insert prices, mark stale units, and update daily summary.
Features:
- dryRun mode skips all database writes while still parsing/validating
- htmlContent parameter allows injecting HTML directly (bypasses fetch)
- Calculates newUnitsCount and rentedUnitsCount by diffing against prior state
- Records every run to scraper_runs history (success or failure)
- Structured logging with jobId correlation throughout the pipeline
- Graceful error handling at each pipeline stage
Includes 15 tests covering full workflow, dry run, error handling,
trigger types, new/rented unit calculation, and empty HTML edge case.
2026-02-06 09:16:23 -07:00
d1f717891a
SCRAPE-10: Implement recordScraperRun() for scraper_runs ( #15 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-06 02:00:19 -07:00
b4978caf31
SCRAPE-9: Implement updateDailySummary() aggregation ( #14 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-06 00:15:30 -07:00
d8adfbf90c
SCRAPE-8: Implement markStaleUnits() ( #13 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 23:48:57 -07:00
2b53288fbe
SCRAPE-7: Implement insertPrices() bulk operation ( #12 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 22:32:02 -07:00
4863dc1824
SCRAPE-6: Implement upsertUnits() bulk operation ( #11 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 22:10:54 -07:00
0cbeaaf9d5
SCRAPE-5: Implement convertDataTypes() with type parsing helpers ( #10 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 21:33:58 -07:00
072e80d069
SCRAPE-4: Implement parseUnits() with cheerio selectors ( #9 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 17:14:01 -07:00
c6beaf8333
SCRAPE-3: Implement fetchPage() with axios, timeout, retry ( #8 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-02-05 15:58:07 -07:00
2993d019c5
SCRAPE-2: Add scraper config constants ( #7 )
2026-01-31 20:54:43 -07:00
3af5a90c93
Add PR number to webhook payload
...
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-28 10:54:12 -07:00
9dba679221
Add scraperLogger.js with credential redaction ( #6 )
...
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-01-28 10:52:00 -07:00
6daacf4d12
Add ESLint and n8n webhook notifications to CI/CD
...
- Add ESLint 9 with flat config for Node.js linting
- Add lint and lint:fix npm scripts
- Add lint job to CI/CD pipeline
- Add notify job to send test/lint results to n8n webhook
- Webhook reports pass/fail status with failure details
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-28 09:39:55 -07:00
8dfdfdc8bb
Fix security vulnerabilities and add early dependency scanning
...
CI/CD Pipeline - Apartment API / Scan Dependencies (push) Successful in 18s
CI/CD Pipeline - Apartment API / Run Tests (push) Successful in 10m20s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Failing after 3m8s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been skipped
- Update to node:20-alpine base image
- Add apk upgrade to fix OS-level vulnerabilities
- Update npm to latest to fix bundled package vulnerabilities
- Add scan-deps job that runs in parallel with tests
- Replace deprecated --only=production with --omit=dev
2026-01-23 16:10:45 -07:00
3518a8344a
Fix security vulnerabilities in dependencies
...
CI/CD Pipeline - Apartment API / Run Tests (push) Successful in 9m41s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Failing after 35s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been skipped
- Update qs to 6.14.1 (CVE-2025-15284)
- Update express to 4.22.1
- Update body-parser to 1.20.4
Fixes HIGH severity npm audit findings
2026-01-23 15:41:35 -07:00
fa0377f5e3
Fix Trivy action compatibility with Gitea runner
...
CI/CD Pipeline - Apartment API / Run Tests (push) Successful in 9m40s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Failing after 38s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been skipped
Replace aquasecurity/trivy-action with direct Trivy installation
to avoid node24 compatibility issues with act-based runners
2026-01-23 15:22:17 -07:00
2d7b412c1a
Add image signing, SBOM generation, and vulnerability scanning to CI/CD
...
CI/CD Pipeline - Apartment API / Run Tests (push) Successful in 9m42s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been cancelled
CI/CD Pipeline - Apartment API / Build & Push Image (push) Has been cancelled
- Add Trivy vulnerability scanning (fails on CRITICAL/HIGH)
- Add Syft SBOM generation in SPDX format
- Add Cosign image signing using digest
- Attach SBOM attestation to image in Harbor
- Add cosign.pub for signature verification
2026-01-23 14:55:01 -07:00
ad5555d759
Add daily price change to /available-units endpoint
...
CI/CD Pipeline - Apartment API / Run Tests (push) Successful in 9m42s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Successful in 23s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Successful in 16s
Add yesterdayPrice and priceChangeToday fields to show price delta
between today and yesterday instead of high-low spread.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 12:19:31 -07:00
b584dc94d4
Remove inline docker login from deploy script
...
CI/CD Pipeline - Apartment API / Run Tests (push) Successful in 9m41s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Successful in 24s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Successful in 14s
Use pre-configured Docker credentials on server instead of passing
Harbor credentials through SSH script, avoiding shell interpolation
issues with special characters in robot account username.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-22 16:43:58 -07:00
62daabc1c3
Remove GitHub Actions cache - not supported by Gitea
...
CI/CD Pipeline - Apartment API / Build & Push Image (push) Successful in 24s
CI/CD Pipeline - Apartment API / Run Tests (push) Successful in 9m41s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Failing after 4s
Gitea Actions doesn't support type=gha caching, causing timeout errors.
Removing cache config to get builds working. Can add registry-based
caching later if needed.
2026-01-22 16:10:09 -07:00
cbb5c1ce48
Fix port conflict: use port 27018 for test MongoDB
...
CI/CD Pipeline - Apartment API / Run Tests (push) Successful in 9m47s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Failing after 1m24s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been skipped
Host machine already has MongoDB on 27017, so map the test
container to 27018 instead to avoid port allocation conflict.
2026-01-22 15:43:04 -07:00
47743656e2
Remove git pull - using dedicated deployment directory
CI/CD Pipeline - Apartment API / Run Tests (push) Failing after 1s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been skipped
2026-01-22 15:33:15 -07:00
f0c674d877
Add git pull to deployment to sync docker-compose.yml
2026-01-22 15:29:36 -07:00
d3733dbebe
Fix CI/CD best practice violations
...
CI/CD Pipeline - Apartment API / Run Tests (push) Failing after 1s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been skipped
- Use SHA-based image tags instead of 'latest' for deployments
- Pass exact image tag from build job to deploy job
- Add default for SSH_PORT
- Health check now fails deploy if not passing
- Added script_stop for proper error handling
2026-01-22 15:27:21 -07:00
0c387b1bcc
Update CI/CD to build once and push to Harbor
...
CI/CD Pipeline - Apartment API / Run Tests (push) Failing after 1s
CI/CD Pipeline - Apartment API / Build & Push Image (push) Has been skipped
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been skipped
- Add image variable substitution to docker-compose.yml
- Build image in CI, push to Harbor registry
- Deploy pulls from Harbor instead of rebuilding
- Supports both local dev (build) and prod (pull from registry)
2026-01-22 14:46:57 -07:00
94bbacb3a2
Add proper CI/CD pipeline with testing
...
CI/CD Pipeline - Apartment API / Run Tests (push) Failing after 12s
CI/CD Pipeline - Apartment API / Deploy to Production (push) Has been skipped
- Separate test and deploy jobs
- Tests run first and block deployment on failure
- Uses MongoDB service container for tests
- SSH-based deployment for security and flexibility
- Health check verification after deployment
- Requires secrets: SSH_HOST, SSH_USER, SSH_PRIVATE_KEY, DEPLOY_PATH
2026-01-22 14:33:59 -07:00
ccda2be551
Update tests to expect wrapped response format
...
- Update all test files to use response.body.data.* instead of
response.body.* to match the API response convention
- Skip SEC-4.3 rate limiting tests (moved to Phase 5)
- All 202 tests now pass
2026-01-22 14:33:01 -07:00
81e5682ab1
Admin Dashboard Backend (Phases 1-4) ( #5 )
...
Deploy Apartment API / deploy (push) Failing after 5m1s
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-01-22 14:11:43 -07:00
d6e56a6e40
Add AUTH.md documenting authentication implementation
...
Deploy Apartment API / deploy (push) Failing after 1s
- Document completed Google OAuth authentication system
- Include implementation details, security measures, and troubleshooting
- Track future enhancements (admin dashboard, Apple Sign-In, partial public access)
2026-01-22 08:26:17 -07:00
ef4ddc0de0
Add Google OAuth authentication system ( #4 )
...
Deploy Apartment API / deploy (push) Failing after 1s
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-01-21 19:15:22 -07:00
04a78a21cc
Add soonest field to available-units endpoint ( #3 )
...
Deploy Apartment API / deploy (push) Failing after 1s
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-01-21 12:04:26 -07:00
34ebace4de
expanded-api ( #2 )
...
Deploy Apartment API / deploy (push) Failing after 1s
Reviewed-on: #2
Co-authored-by: Stephen Minakian <stephenminakian@gmail.com >
Co-committed-by: Stephen Minakian <stephenminakian@gmail.com >
2026-01-21 10:18:46 -07:00
49bd77bec6
Try to rebuild and deploy
Deploy Apartment API / deploy (push) Successful in 9m49s
2025-07-15 22:02:16 -06:00
4833f55b29
Remove health check
Deploy Apartment API / deploy (push) Successful in 9m41s
2025-07-15 20:47:34 -06:00
4c3b63c4bf
Added priority to traefik tag
Deploy Apartment API / deploy (push) Successful in 9m44s
2025-07-15 20:36:12 -06:00
9153a2faf8
Initial commit
Deploy Apartment API / deploy (push) Successful in 9m48s
2025-07-15 17:51:26 -06:00
