Omair Saleh ab875cd4d9 fix: JV scraper broken by brotli encoding + improved robustness ...

ROOT CAUSE: _browser_headers() included 'Accept-Encoding: gzip, deflate, br'
but the container has no brotli decoder. Server sent compressed response
that requests couldn't decode → garbled HTML → empty title → 'Could not
find product' error on Demo A and Demo C.

FIXES:
- Remove 'br' from Accept-Encoding (use 'gzip, deflate' only)
- Price extraction: try itemprop on any element, then .pricec class, then regex
- Image extraction: multi-strategy (itemprop, gallery links, CDN pattern, OG)
- Detect homepage redirect (product removed/renamed) → clear error message
- Increase timeout from 15s to 20s for JV product scraping

TESTED:
- D3+K2: Title ✓, Price £12.95 ✓, 10 benefits ✓, 3 images ✓
- Vitamin D3 4000iu: Title ✓, £8.95 ✓, 6 benefits ✓, 7 images ✓
- B12: Title ✓, £11.95 ✓, 10 benefits ✓, 7 images ✓
- Removed product: clean error 'redirected to homepage'

2026-03-02 22:43:38 +08:00

22 KiB

Raw Blame History

View Raw