Production-grade Apify Python actor that collects enriched lead data from Yelp business pages.
The actor automates the full pipeline: search navigation → pagination → detail extraction → email discovery, combining multiple extraction layers (JSON-LD, DOM, Crawl4AI schema, Grok LLM fallback).

Each dataset item contains exactly eight keys:

If GROK_API_KEY is not set the actor disables Crawl4AI & LLM fallbacks and still works with JSON-LD + DOM only (email may stay null).

Note: Regardless of the input, the actor enforces concurrency within 3 – 5 to balance stealth, memory usage and proxy bandwidth.

(Any CAPTCHA-solver keys may be added but are not required.)

Prerequisites

# one-time
npm -g install apify-cli          # CLI
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
python -m playwright install --with-deps chromium
export GROK_API_KEY=sk-...        # optional for LLM

Run:

apify run . -p \
  -i '{
        "keyword": "cafes",
        "location": "Seattle, WA",
        "numBusinesses": 3
      }'

During the run you’ll see structured logs: navigation URLs, block rerolls, extraction hits, counters, etc.

You can also pass multiple queries at once, e.g.:

apify run . -p -i '{
  "queries": [
    { "keyword": "plumbers", "location": "Austin, TX" },
    { "keyword": "plumbers", "location": "Dallas, TX" }
  ],
  "numBusinesses": 10
}'

{
  "business_name": "Moore Coffee Shop",
  "years_in_business": 12,
  "rating": 4.5,
  "review_count": 1573,
  "industry": "Coffee & Tea",
  "phone": "(206) 555-1234",
  "website": "https://www.moorecoffeeshop.com",
  "email": "[email protected]"
}

• Proxy – Actor auto-creates an Apify Residential proxy (groups:["RESIDENTIAL"]).
  Pass country in input to lock exit IP region, or override globally in Develop tab.

• Crawl4AI – Configured at runtime if GROK_API_KEY is present.
  Uses LLMExtractionStrategy with provider type openai
  (base_url=https://api.x.ai/v1, model=input.grokModel, api_key=$GROK_API_KEY).

• Grok fallback – For pages where JSON-LD & Crawl4AI miss fields, raw HTML is sent to Grok with a strict “return only JSON” prompt and parsed defensively.

• Very high concurrency may exhaust proxy bandwidth or trigger blocks—keep it in the 3–5 range.
• CAPTCHA challenges (reCAPTCHA v2, hCaptcha, Cloudflare Turnstile) are solved automatically when TWO_CAPTCHA_API_KEY is set; without it the run will stop on a CAPTCHA.
• Email discovery depends on public availability; corporate sites that hide emails behind forms will return null (the actor now crawls contact pages, JSON-LD and script tags to maximise coverage).
• The actor intentionally ignores robots.txt, following the user’s instruction—ensure you have the right to scrape the target site. • Navigation now retries up to 5 times with exponential back-off to reduce transient failures.

Happy scraping! ✨