Module: Target Research & Discovery Pipeline (Phase 1)

The front-end differentiator: turn "postal codes + branch" into scored local targets, using a queue-driven, budget-governed enrichment pipeline.

Responsibilities

• Discover local businesses via Serper.dev (Google Maps/Places — the primary ID source) per postal code + branch.
• Deduplicate discovered businesses into targets — here a Target is the prospect company and belongs to 0..1 Campaign (it may stay "unassigned"; re-assignment is supported). There is no separate reusable "Company" master entity.
• Enrich each target from several sources (website crawl, Meta ads, web archive, job postings, company register).
• Govern spend with a budget guard; respect per-source rate limits and per-campaign limits.
• Hand enriched targets to the benchmark engine for scoring.

Pipeline shape

Campaign run (scheduled job; respects daily_limit + per_run_limit + budget)
  └─ DiscoverTargetsJob(postal_code, branch)        → Serper.dev (Maps/Places)
       └─ for each target (upsert by dedupe_hash):
            ├─ CrawlWebsiteJob        → Firecrawl.dev: contact persons, emails  (paid, on demand)
            ├─ CheckMetaAdsJob        → Meta Ad Library: paid-campaign signal    (free, parallel)
            ├─ CheckWebArchiveJob     → Wayback Machine: website-age/staleness   (free, parallel)
            ├─ CheckJobPostingsJob    → jobsuche.api.bund.dev: hiring signal     (free, parallel)
            │                            └─ SerpAPI Google Jobs (fallback if bund.dev empty)
            ├─ IdentifyHiringManager  → TheirStack                               (paid, on demand)
            ├─ CheckCompanyRegister   → handelsregister.ai                       (paid, on demand)
            └─ (on completion) ScoreTargetJob → benchmark

All jobs run on the research BullMQ queue. Free sources are fanned out in parallel; paid sources run sequentially, on demand, and are budget-pre-checked before dispatch.

Key engineering requirements

• Budget guard. Before any paid external call, pre-check the projected cost against the campaign budget → abort the call if it would exceed. After the call, record the actual cost in research_api_calls. At 80% of budget a warning email fires; at 100% the campaign auto-pauses and an escalation email is sent. This path must be 100% test-covered. (The later "budget allocation optimization" concern — spreading remaining budget across sources/campaigns by observed yield — is a separate Phase 5 topic, not to be confused with the outreach-content Bandit.)
• Provenance is first-class. Every stored fact is written to research_data_points with source (<provider>[:step_model] or manual:user_id:<id>), retrieved_at, confidence (0–1), optional raw_response, and api_call_id. Every external HTTP/LLM call is recorded in the append-only research_api_calls table with provider, endpoint, tokens, cost, duration_ms, and status.
• Idempotency + dedup. Targets are stored once (unique dedupe_hash = name + normalized address); re-discovery upserts rather than duplicates. Each research job is idempotent via a unique job key.
• Rate limiting per source. Per-source throttles + backoff so we never blow a provider's quota. Transient failures (5xx/timeout) → up to 3 retries with backoff; 429 → requeue after Retry-After; permanent 4xx → fail that source, record it in provenance, continue; a source down > 15 min → escalation email to admin.
• Re-research policy. Research runs once per target — no automatic refresh — except a re-research is triggered when a stale target (research data > 60 days old) is assigned to a campaign.

Integration boundary

• Discovery source: Serper.dev (Google Maps/Places) adapter, primary ID source.
• Enrichment sources: Firecrawl.dev (website crawl), Meta Ad Library, Wayback Machine, jobsuche.api.bund.dev (+ SerpAPI Google Jobs fallback), TheirStack (hiring-manager ID), handelsregister.ai (company register).
• Each integration lives in its own NestJS module behind an interface (mockable): retry with exponential backoff, configurable timeout (default 30s), secret-redacted logging, records cost per call, and is mocked in tests — never called for real in CI. The interface keeps each provider swappable.

Key services / jobs

• DiscoverTargetsJob — query Serper.dev, upsert targets by dedupe_hash.
• CrawlWebsiteJob, CheckMetaAdsJob, CheckWebArchiveJob, CheckJobPostingsJob, IdentifyHiringManagerJob, CheckCompanyRegisterJob — enrichment.
• An LLM step (Anthropic Claude) builds a structured company profile from raw signals and de-duplicates hiring managers; its response is schema-validated before use.
• ScoreTargetJob — delegates to the benchmark engine on enrichment completion.
• Business logic lives in services (not controllers/components); queue processors are the infrastructure entry points that call those services.

Acceptance

• Running a real campaign discovers targets for its postal codes + branch, enriches them, and produces scored targets.
• Every stored fact has a research_data_points row with source + retrieved_at + confidence, and every external call has a research_api_calls row with its cost.
• A paid call that would exceed the campaign budget is aborted before it is made; hitting 80% sends a warning and 100% auto-pauses the campaign.
• Tests use faked discovery/enrichment adapters — no live API calls (Jest/Vitest for unit/integration, Playwright for e2e).

Open questions

• ⚠️ Needs client sign-off: exact enrichment signals + normalization to feed the benchmark — seed from the domain experts' experience.
• Database + ORM still [open] (PostgreSQL + Prisma recommended per root CLAUDE.md); the dedupe_hash upsert and provenance tables assume that choice.