Omair Saleh
ccfc9ceeb1
fix: bulletproof competitor scraper — 4-tier fallback chain
Tier 1-3: HTTP with Chrome/Firefox/Safari UAs + full browser headers
Tier 4: Gemini + Google Search grounding (bypasses everything)
- Dead URLs (404): skips straight to Gemini, finds product via Google
- Cloudflare/CAPTCHA: detected and routed to Gemini
- JS-rendered pages: Gemini reads them via Google's infrastructure
- Updated default competitor URL to Vitabiotics (works direct)
Tested against:
- H&B dead URL (404) → Gemini found full product data
- Boots (Cloudflare) → Gemini returned £4.00, 4.6★, 8 bullets
- Vitabiotics → direct Chrome scrape, 9 bullets
- Amazon (CAPTCHA) → Gemini grounding fallback