MaDI-Bench: An End-to-End Data Integration Benchmark

Data integration is the process of combining data from multiple, heterogeneous sources into a consistent, unified representation. Data integration involves a sequence of interdependent tasks including schema matching, value normalization, blocking, entity matching, and data fusion. Existing table-based benchmarks either evaluate these steps in isolation or cover only incomplete versions of the data integration pipeline, omitting specific steps. The lack of public end-to-end data integration benchmarks hinders research on data integration methods that address the integration process as a whole and account for the interdependencies among the different tasks.

The Mannheim Data Integration Benchmark (MaDI-Bench) fills this gap. It is the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process. MaDI-Bench contributes (i) a set of end-to-end data integration tasks spanning several application domains, each requiring the full schema matching, value normalization, entity matching, and data fusion pipeline, and (ii) a generic method for deriving task variants that mitigates rapid benchmark saturation as data integration systems advance. We validate the benchmark using human-engineered pipelines, a best-of-breed pipeline, an LLM workflow, and a pipeline written by a coding agent. The validation demonstrates the utility of the benchmark for measuring the step-wise as well as the end-to-end performance of data integration pipelines. All benchmark artifacts are available for public download.

MaDI-Bench consists of five base end-to-end data integration tasks. These base tasks require the integration of data about Games, Companies, Music, Products, and Scientific Papers. Each task takes several heterogeneous tables as input and asks the system under test (SUT) to integrate these tables into a single output table conforming to a given target schema. The integration requires the SUT to perform schema matching, value normalization, blocking, entity matching, and data fusion. The benchmark provides ground truth to score every step on its own, not only the final table: a gold schema mapping for schema matching, labeled normalization sets for value normalization, labeled record pairs for blocking and entity matching, and human-verified records for data fusion, together with metrics for the end-to-end result. MaDI-Bench offers an easy, a medium, and a hard variant of each base task, 20 tasks in total.

Schema matching  →  Value normalization  →  Entity blocking & matching  →  Data fusion

MaDI-Bench provides the following artifacts for each of the base tasks:

• Source tables and table metadata. One table per source, each with a metadata file recording its origin, the columns it contains, and the date it was published. Example: forbes.csv, forbes_metadata.json
• Target schema. The columns of the fused table, with a value constraint per attribute (for example a valid founding-year range) and categorical attributes tied to standard taxonomies such as GICS industries and CLDR country names. Example: target_schema.json
• Gold schema mapping. The correspondences from the source columns to the attributes of the target schema. Example: sm_mapping_gold.json
• Normalization validation and test sets. A validation and a test set of source cells, each paired with its correctly normalized value; 1,745 cells across the five base tasks. Most cells need a formatting rule or a mapping onto a vocabulary that the task provides; some already hold the correct value and test that a normalizer leaves it intact. Example: validation.csv
• Entity matching sets. Labeled train, validation, and test pairs for two pairs of sources per task (three for Products); more than 93,000 pairs across the five base tasks. The test pairs also evaluate blocking. Example: forbes_2_dbpedia_train.csv
• Fusion validation and test sets. 100 validation and 100 test entities per task with human-verified attribute values; close to 8,000 values across the five base tasks. All tasks use version 2 of the fusion gold, which leaves out the values that cannot be obtained from any source record of the entity, even after normalization. Example: validation_set.xml
• Reference outputs. The output of the human-engineered pipeline (P1), whose clusters with at least two source records form the silver reference for the end-to-end metrics. Example: companies/base/output/

The easy, medium, and hard variants come with their own source tables, gold schema mapping, and entity matching and normalization sets. They keep the fusion entities and gold values of their base task.

The sources describe the same kind of entity but rarely agree on column names, value formats, or even the facts themselves, which is what makes the merge hard.

A system under test (SUT) needs to handle the following steps of the overall integration task. MaDI-Bench provides ground truth for each step as well as for the output of the overall integration (fused dataset). Thus, a SUT can be evaluated one step at a time or in an end-to-end fashion.

Sources rarely share column names, so the source columns have to be matched to the attributes of the target schema before values can be combined. Source columns without a matching target attribute stay unmapped. Names may be descriptive, abbreviated, or meaningless, and where a name gives nothing away, a matcher has to read the values instead.

Matched columns still store values in different formats and vocabularies, so they are rewritten into the forms the target schema requires. The schema sets a constraint for every attribute: a data type, and where it applies a numeric range, a string pattern, or a date format. A founding date, for instance, must be an ISO YYYY-MM-DD inside an allowed range, and assets must be a non-negative integer in US dollars below an upper bound. For categorical attributes, the task provides taxonomies that specify which values are allowed and how detailed the normalized values should be. The taxonomies ship with the task: some follow a standard, for example CLDR for countries and GICS for industries; others cover domain vocabularies such as the platforms, genres, and ESRB ratings of the Games task. Normalization is scored as accuracy on the normalization test set: each test cell pairs a source value with its correctly normalized value. The test cells include values that are already correct, so a normalizer that changes correct values loses accuracy.

The SUT must identify records referring to the same real-world entity across sources. Since comparing all record pairs requires scoring the full Cartesian product and is therefore quadratic in the input size, this quickly becomes computationally infeasible. Blocking addresses this issue by reducing the search space to a smaller set of candidate pairs, after which matching classifies each candidate as either a match or a non-match. The MaDI-Bench entity matching test sets are designed to contain many hard-to-decide corner-case record pairs.

After matching, records referring to the same real-world entities form clusters. Data fusion consolidates these clusters into a single record per cluster. When records provide conflicting values, the SUT must resolve the conflict and decide which value to retain, for example with a conflict resolution heuristic or a learned truth-discovery model. The values in the MaDI-Bench data fusion test sets are verified by humans who checked them against data from relevant websites.

Beyond the per-step evaluation, MaDI-Bench assesses the final output table along three quality dimensions: coverage (did the right entities and values make it into the output table?), consistency (does the table fit the target schema?), and correctness (are the fused values correct?). The metrics use three types of reference data: reference-free (no ground truth, structural and schema checks of the output, some of them relative to the input tables), silver (comparison with the clusters of the human-engineered pipeline P1 that combine at least two source records), and ground truth (comparison with the human-verified fusion test sets). The silver reference covers a much larger part of the output table, while the ground truth covers only the test entities but gives the stronger guarantee of correctness.

We validate MaDI-Bench using four pipelines that span the design space from human-engineered to fully LLM-driven. The human-engineered pipelines (P1) and the LLM workflow (P3) build on the PyDI integration framework. The best-of-breed pipeline (P2) chooses, for each step, between specialized methods from the literature and methods of PyDI. The coding agent Claude Code (P4) writes and runs its own integration code without PyDI.

Each pipeline is run once per task. The tables report every step on the five base tasks, with the human-engineered pipeline (Human, P1), the best-of-breed pipeline (BoB, P2), the LLM workflow (LLM, P3), and the Claude Code pipeline (CC, P4). Where the pipelines differ, the best score per task is in green. The values are those of the paper. The outputs and scores of every run are in the results/ folder of the repository, and the cells of each paper table with their sources in results/paper_tables/.

All four pipelines reach at least 92.68 F1 on every task, and 15 of the 20 pipeline scores are 100. The label-based and instance-based matchers of PyDI are shown for reference. They fail in complementary places. Label-based matching falls to 26.1 on Companies, where FullContact uses anonymized column names such as Attribute_2. Instance-based matching falls to 7.7 on Products, possibly because several numeric attributes, such as read and write speeds, draw values from similar ranges. Neither reference matcher reaches more than 74 F1 on any task. Values are F1 (%).

Normalization accuracy is measured on the normalization test sets. It covers cells that need a transformation and cells whose value is already correct, so a normalizer that changes correct values loses accuracy. Averaged over the tasks, normalization is the lowest step score of every pipeline, between 28.45 (P2) and 60.12 (P4), and every pipeline scores below 40 on at least one task. Games is the hardest task for P1, P2, and P4: 171 of its 182 test cells need a transformation, mostly a mapping of genres and platform aliases onto the target taxonomies. Values are accuracy (%).

Pair completeness lies between 93.33 and 100 for all pipelines and tasks. Reduction ratios exceed 97% on every task except Products, where P1 removes 71.89% of the possible record pairs. On Scientific Papers, the largest task, every pipeline removes at least 99.97%. Entity matching F1 ranges from 64.98 to 99.83. Games and Products separate the pipelines most. On Games, identical titles recur across platforms, and the same game on another platform is a different entity. Values are percentages, averaged over the source pairs. The best entity matching F1 per task is in green.

Fusion accuracy is the share of the verified cells of the fusion test set for which the fused output holds a correct value. A fused value can be wrong because matching kept the record with the correct value out of the cluster, because matching added records of other entities, or because conflict resolution chose a wrong value. P4 reaches the highest accuracy on Games, Companies, Music, and Products, and P1 on Scientific Papers. The best score per task still leaves 8.89 (Scientific Papers) to 22.19 (Games) points open. Values are accuracy (%).

MaDI-Bench also scores the final fused table as a whole, along three dimensions: coverage (are the right entities and values present), consistency (does the output fit the target schema), and correctness (are the merged values right). The reference-free metrics need no reference. The silver reference consists of the clusters of the P1 output that combine at least two source records, so P1 itself is not listed. The ground-truth reference consists of the human-verified fusion test sets. The table shows the means over the five tasks. For metrics with a preferred direction, the best pipeline is in green.

The runtimes measure the execution of the final pipeline in wall-clock seconds. Model training is part of the runtime, while the configuration search and the human design effort are not. The mean runtimes differ by a factor of 15, from 358 seconds for P4 to 5,205 seconds for P2. Scientific Papers, the largest task, is the most expensive task for every pipeline. The last two columns give the estimated API cost of the two LLM-driven pipelines in US dollars. The fastest pipeline per task is in green.

To keep the benchmark hard as systems improve, each base task also ships in easy, medium, and hard versions. The knobs keep the identity of every retained record and its true matches. Two knobs add near-duplicate entities or remove records from some sources. Eight difficulty knobs perturb the data, each set to its own target for easy, medium, and hard. A harder level turns up the kinds of heterogeneity that real integration runs into, while the easy level supports the evaluation of simpler and cheaper methods. The easy level makes schema and entity matching easier, but not fusion.

Here is the DBpedia record for the Taisei Corporation, base against hard. Schema naming divergence (Knob 7) renames every column to a cryptic code: org_name→nm, established→ey, nation→cn, headquarters→hq, sector→sg, keypeople_name→kpn, total_assets_val→ta, annual_income→ai. On top of that, the values change:

Each knob targets one or more pipeline stages and is set to an easy, medium, or hard target. MaDI-Bench uses eight:

The difficulty of each variant is measured with a committee of methods per step. The committee mean is the average over all members, the ceiling the score of the strongest member. From medium to hard, the committee mean declines for schema matching, entity matching, and fusion in every domain. The signal is clearest for fusion, whose mean falls by 10.9 to 24.0 points and whose ceiling by 10.6 to 28.1 points from base to hard. The easy level lowers the difficulty of schema and entity matching, while its fusion mean lies below the base score in four domains. Rows are in thousands, density and scores in %.

The validation pipelines P2, P3, and P4 were also run on every variant. Fusion shows the clearest difficulty gradient. For every pipeline and domain, fusion accuracy on the hard variant lies below the base task, with losses of 16.7 to 30.0 points for P2, 13.1 to 29.7 for P3, and 2.0 to 14.6 for P4. On the hard variants, no pipeline exceeds 75.8 fusion accuracy in any domain. Schema matching stays at or above 91.3 F1 on every level. Values are percentages.

This paper introduced the Mannheim Data Integration Benchmark (MaDI-Bench). The benchmark covers all steps of the data integration process, from schema matching and value normalization through blocking and entity matching to data fusion, and evaluates each step as well as the integrated output, taking into account the dependencies between the steps. To prevent a quick saturation of the benchmark as agentic systems progress, we introduced a generic variant-generation method for deriving harder as well as easier variants from the base tasks. We validated MaDI-Bench and the generated variants using human-designed pipelines, an LLM workflow, a best-of-breed pipeline, and a Claude Code integration agent. The validation showed the benchmark's utility for measuring the step-wise as well as end-to-end performance of data integration pipelines. We hope that the benchmark will prove useful for the community and will support the development of fully automatic as well as human-in-the-loop data integration systems.

Tasks are released as plain files (CSV, JSON, JSON Lines, and XML), so any system can read them. The evaluation package of the repository scores value normalization and data fusion; schema matching, blocking, entity matching, and the end-to-end metrics are scored with the evaluators of PyDI at the commit that the repository pins. The repository README describes the submission formats and the commands.

• Benchmark repository: all 20 tasks with their ground truth, the evaluation code, the variant generator, the code of P1, P2, and P3, the skills and prompts of P4, and the outputs and scores of the validation runs.
• PyDI: the integration framework and the per-step evaluator classes.
• The tasks: source tables, target schemas and taxonomies, gold schema mappings, and the labeled entity matching, normalization, and fusion sets.