Combining scraping and AI can reduce the time spent sorting company listings, directories or marketplace records. The result should be a prioritised work queue, not an opaque list of people to contact. Good classification begins with a clear definition of a qualified lead and a defensible data source.
Use official APIs where available and collect only public or authorised business data. Respect access controls, applicable terms, robots guidance, request limits and data-protection obligations. Do not infer sensitive traits. Document the purpose, lawful basis, retention period and suppression process before scaling collection.
Define the labels before choosing a model
Start with labels a salesperson can act on: qualified, review, not relevant and insufficient data. Write examples and exclusion rules for each label. If two people cannot classify a sample consistently, an AI system will only automate that ambiguity.
Separate hard rules from semantic judgment. Country, category, minimum company size and duplicate status are deterministic filters. A model can then assess descriptions or service fit, returning a label, confidence score and short reason in a fixed schema.
A production pipeline
- collect approved fields with source URL and retrieval time
- normalise names, domains, locations and categories
- deduplicate with exact keys and cautious fuzzy matching
- apply exclusions and deterministic qualification rules
- classify remaining text against a versioned rubric
- validate structured output and confidence
- route uncertain or high-value cases to human review
- export only approved records to the CRM
Queues and idempotent workers keep these stages independent. See the scraping architecture with queues, workers and retries and the guide to building a data pipeline from scraping to CRM.
Do not let AI hide poor data
A model cannot repair missing provenance, stale listings or the wrong target market. Validate required fields before classification and assign insufficient data instead of inventing an answer. Keep raw source facts separate from generated labels so every decision can be audited or recomputed.
Measure precision and recall on a representative, manually labelled test set. In sales workflows, false positives waste time and can create compliance risk; false negatives hide opportunities. Choose thresholds based on that cost rather than using an arbitrary score such as 0.7.
Common mistakes
- collecting data before defining a legitimate purpose
- sending every record to an expensive model
- mixing deterministic filters with model opinions
- accepting free-form output instead of a schema
- treating confidence as calibrated probability
- training or prompting with unnecessary personal data
- exporting duplicates and suppressed contacts again
- changing prompts without versioning and evaluation
Practical checklist
- approve sources, fields, purpose and retention
- prefer APIs and public business information
- define labels with positive and negative examples
- normalise and deduplicate before AI processing
- use deterministic rules for objective criteria
- require structured output with a reason code
- set human-review and rejection thresholds
- record source, parser, rubric and model versions
- test precision, recall, cost and latency
- monitor drift and honour deletion or suppression requests
Traceability and stability
Store a decision envelope for each lead: source facts, collection timestamp, transformation version, rubric version, output and reviewer correction. Avoid logging secrets or unnecessary personal data. Corrections should feed an evaluation set, not silently overwrite history.
Monitor extraction completeness, duplicate rate, label distribution, review rate, cost per accepted lead and downstream conversion. A sudden improvement in qualification may indicate a broken parser feeding identical descriptions. The approach in detecting duplicates, spam and junk leads is a useful earlier quality gate.
When hiring a technical person makes sense
Bring in a specialist when several sources need different parsers, classifications affect customer outreach, auditability matters or model cost is growing unpredictably. A technical lead can define the data contract, evaluation set, review workflow and compliance controls before unreliable scores enter the CRM.
Final takeaway
Scraping supplies permitted facts; rules establish objective fit; AI helps interpret ambiguous text; humans handle uncertainty. If you need a stable, measurable lead-classification pipeline, review my technical services or contact me with your sources, qualification rules and expected volume.