Omair Saleh ccfc9ceeb1 fix: bulletproof competitor scraper — 4-tier fallback chain
Tier 1-3: HTTP with Chrome/Firefox/Safari UAs + full browser headers
Tier 4: Gemini + Google Search grounding (bypasses everything)

- Dead URLs (404): skips straight to Gemini, finds product via Google
- Cloudflare/CAPTCHA: detected and routed to Gemini
- JS-rendered pages: Gemini reads them via Google's infrastructure
- Updated default competitor URL to Vitabiotics (works direct)

Tested against:
- H&B dead URL (404) → Gemini found full product data
- Boots (Cloudflare) → Gemini returned £4.00, 4.6★, 8 bullets
- Vitabiotics → direct Chrome scrape, 9 bullets
- Amazon (CAPTCHA) → Gemini grounding fallback
2026-03-02 21:12:55 +08:00
Description
JustVitamin � QuikCue � AI Content Engine Proposal Site
1.1 MiB
Languages
HTML 55.3%
Python 30.2%
CSS 7.5%
JavaScript 6.9%
Dockerfile 0.1%