An AI pipeline reconciling hundreds of partner restaurants across four sources and 10+ languages
Hundreds of European partner restaurants. Four data sources that disagree with each other. Ten-plus languages. We built the pipeline that reconciled all of it automatically.
The challenge
A global beverage company's on-premise presence is its restaurant network. These are the venues where consumers encounter the product in context, where the brand experience happens in person. The company wanted to make that network navigable: a directory where someone could find partner restaurants by city, cuisine, or dining style.
The data situation made this harder than it sounds. Restaurant information for hundreds of venues sat scattered across TripAdvisor, TheFork, Google Reviews, and individual restaurant websites. No two sources agreed on anything. Names, opening hours, cuisine categories, addresses — all conflicted, often across more than ten European languages.
"Ristorante Da Mario" on TripAdvisor. "Da Mario Restaurant" on Google. "da mario" on the restaurant's own site. Are those the same place? Sometimes yes, sometimes no. You have to look at the address, the phone number, the cuisine description, even review patterns to be sure.
Manually verifying each restaurant would have required a team of researchers working for months on just the first market. By the time they finished, a meaningful portion of the data would already be out of date. The brand needed automation, and the automation needed to handle judgment calls, not just data copying.
What we learned
| Five sources, no consensus | When data lives across five platforms, none agree on names, hours, or categories — manual reconciliation runs months. |
| Same-or-not is judgment | Deciding two records describe the same restaurant requires reading context, not field comparison. |
| Rules break on the 10% | Deterministic logic handles the patternable cases — the edge cases are exactly what AI is for. |
The solution
Twistag's most important architectural decision was where to use AI and where not to. The approach was surgical: conventional ETL for normalisation and straightforward deduplication, AI only for the cases that genuinely require contextual judgment. This keeps AI costs proportional to complexity rather than volume and keeps the system's behaviour predictable for the 80% of cases that don't need a language model.
The pipeline runs on GCP with Cloud Run for serverless scraping workers and Pub/Sub managing the event queue between stages. Each scraper extracts a standardised set of fields — name, address, cuisine type, price range, opening hours, description, ratings, source URLs. Puppeteer handles JavaScript-rendered platforms like TripAdvisor and TheFork. Lighter HTTP scrapers handle static restaurant websites. Google Search API serves as the discovery layer for restaurants that don't appear prominently on the major review sites. Adding a new regional source means deploying a new scraper container without touching the rest of the pipeline.
Raw scraped data passes through an ETL layer before AI sees it. Address standardisation, cuisine taxonomy mapping, language detection, obvious duplicate removal — all deterministic, all cheap, all predictable. Each field also receives a confidence score based on the source's historical accuracy for that field type, the recency of the data, and consistency across sources. These scores feed into the AI step as context.
The reconciliation engine runs on OpenAI for three tasks. Entity matching: determining whether listings from different sources refer to the same physical restaurant — this goes beyond name similarity, considering address proximity, cuisine alignment, and review content patterns. Conflict resolution: when sources disagree on a specific field, the AI evaluates which to trust field by field rather than source by source, because Google has fresher hours, TripAdvisor has more granular cuisine classifications, the restaurant's own website carries the most accurate name. Content generation: synthesising restaurant descriptions from multiple sources into a single coherent profile in the brand's voice. Every reconciliation task is structured as explicit input, context, and expected output, which makes the AI's decisions auditable and the prompts iteratively improvable.
What this shaped
| AI for judgment, not volume | Deterministic ETL handles the bulk cheaply. Reserve LLM calls for decisions that genuinely need context. |
| Confidence scores enable triage | Attach certainty to every AI decision so manual review focuses on cases where it actually matters. |
| Field-level trust beats source-level | One source has the best hours, another the best cuisine — trust per field, not per source. |
The impact
The first phase covered hundreds of partner restaurants across multiple European markets in three months, indexed across more than ten languages, reconciled across at least four major data sources. Each profile represents the output of AI-driven reconciliation across at least two independent sources, with confidence scores on each field giving the team a clear signal of where to focus any manual review.
The more durable result is the architecture. Adding a new European market requires deploying scrapers for any region-specific data sources and running the pipeline. ETL, reconciliation, and profile generation work unchanged regardless of country or language. Phase two and beyond are operational expansions, not engineering projects.
What this proved
| Architecture beats market replication | Once the pipeline works for one market, adding languages and regions is operational work, not engineering. |
| Reconciliation outlasts source updates | Combining sources beats any single source — data stays right when one source moves. |
| Review built in, not bolted on | Manual review surfaces from day one prevent bad data propagating before scale makes it expensive to fix. |
Technologies used
- OpenAI
- GCP
- Puppeteer
