SCRAPE-4: Implement parseUnits() with cheerio selectors #9

Merged

stephen merged 1 commits from scraper/parse-units into dev 2026-02-05 17:14:02 -07:00

Conversation 1 Commits 1 Files Changed 3 +597

stephen commented 2026-02-05 16:50:01 -07:00

Owner

Copy Link

Summary

• Implement parseUnits(html, logger) function in scraperService.js
• Uses cheerio to extract apartment unit data from HTML
• Finds section.spaces__tab-unit container and extracts all data-spaces-* attributes from article elements
• Deduplicates units by unit_code with warning logging
• Handles missing container gracefully (logs error, returns empty array)
• Extracts image URLs from nested img elements

Test plan

• 22 unit tests covering all acceptance criteria
• Valid HTML fixture with 5 varied apartment units
• Edge cases: empty HTML, missing container, missing attributes
• Deduplication logic with duplicate detection logging
• Image URL extraction from nested elements

## Summary - Implement `parseUnits(html, logger)` function in `scraperService.js` - Uses cheerio to extract apartment unit data from HTML - Finds `section.spaces__tab-unit` container and extracts all `data-spaces-*` attributes from article elements - Deduplicates units by `unit_code` with warning logging - Handles missing container gracefully (logs error, returns empty array) - Extracts image URLs from nested img elements ## Test plan - [x] 22 unit tests covering all acceptance criteria - [x] Valid HTML fixture with 5 varied apartment units - [x] Edge cases: empty HTML, missing container, missing attributes - [x] Deduplication logic with duplicate detection logging - [x] Image URL extraction from nested elements

stephen added 1 commit 2026-02-05 16:50:01 -07:00

Implement parseUnits() HTML parsing with cheerio ... 39b513cbba

- Extract apartment unit data from section.spaces__tab-unit container
- Parse all data-spaces-* attributes from article elements
- Deduplicate units by unit_code with warning logging
- Handle missing container gracefully (returns empty array)
- Add comprehensive test suite with HTML fixture (22 tests)

stephen merged commit 072e80d069 into dev 2026-02-05 17:14:02 -07:00

stephen deleted branch scraper/parse-units 2026-02-05 17:14:02 -07:00

stephen referenced this issue from a commit 2026-02-05 17:14:02 -07:00

SCRAPE-4: Implement parseUnits() with cheerio selectors (#9)

stephen commented 2026-02-06 17:26:03 -07:00

Author

Owner

Copy Link

❌ Changes Requested

PR deletes ~4,200 lines of working code (all DB operations, data conversion, orchestration, mutex lock, and 8 test suites) while adding no new functionality — parseUnits() already existed on dev. The CI pipeline changes are the only meaningful addition.

Issues:

services/scraperService.js:198
All functions after parseUnits() have been deleted: convertDataTypes, upsertUnits, insertPrices, markStaleUnits, updateDailySummary, recordScraperRun, runScrape, and all parse helpers. These are working, tested functions on dev. If this is intentional (rollback to only parse layer), it needs explicit justification.

services/scraperService.js:7
The crypto and createLogger imports were removed. The createLogger import is needed by runScrape (now deleted). If downstream functions are expected to be re-added later in separate PRs, this removal is premature.

jobs/scraperJob.js:1
Entire mutex lock module deleted. This was a complete, working implementation with passing tests. Deletion not explained by the PR scope (parseUnits).

tests/scraper/dataConversion.test.js:1
8 test suites deleted (dataConversion, insertPrices, markStaleUnits, recordScraperRun, scraperJob, scraperService, updateDailySummary, upsertUnits). This removes ~3,500 lines of meaningful test coverage. These tests were passing on dev.

## ❌ Changes Requested PR deletes ~4,200 lines of working code (all DB operations, data conversion, orchestration, mutex lock, and 8 test suites) while adding no new functionality — parseUnits() already existed on dev. The CI pipeline changes are the only meaningful addition. ### Issues: **services/scraperService.js:198** All functions after parseUnits() have been deleted: convertDataTypes, upsertUnits, insertPrices, markStaleUnits, updateDailySummary, recordScraperRun, runScrape, and all parse helpers. These are working, tested functions on dev. If this is intentional (rollback to only parse layer), it needs explicit justification. **services/scraperService.js:7** The `crypto` and `createLogger` imports were removed. The createLogger import is needed by runScrape (now deleted). If downstream functions are expected to be re-added later in separate PRs, this removal is premature. **jobs/scraperJob.js:1** Entire mutex lock module deleted. This was a complete, working implementation with passing tests. Deletion not explained by the PR scope (parseUnits). **__tests__/scraper/dataConversion.test.js:1** 8 test suites deleted (dataConversion, insertPrices, markStaleUnits, recordScraperRun, scraperJob, scraperService, updateDailySummary, upsertUnits). This removes ~3,500 lines of meaningful test coverage. These tests were passing on dev.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

stephen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stephen/apartment-dashboard-api#9

No description provided.