MaDI-Bench: An End-to-End Data Integration Benchmark

Data integration combines heterogeneous data sets into a single, coherent representation. It involves a sequence of interdependent tasks: schema matching, value normalization, entity blocking, entity matching, and data fusion. Existing benchmarks either evaluate these steps in isolation or cover only incomplete versions of the data integration pipeline, omitting specific steps. The lack of public end-to-end data integration benchmarks hinders research on data integration methods that address the integration process as a whole.

The Mannheim Data Integration Benchmark (MaDI-Bench) fills this gap. MaDI-Bench is the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process. MaDI-Bench contributes (i) a set of base end-to-end data integration tasks spanning several application domains, each requiring the full schema matching, value normalization, entity matching, and conflict resolution pipeline; and (ii) a generic method for deriving task variants that mitigates rapid benchmark saturation as data integration systems advance. We validate the benchmark using human-engineered pipelines, a best-of-breed pipeline, and an LLM-based pipeline. The validation demonstrates the utility of the benchmark for measuring the step-wise as well as the end-to-end performance of data integration pipelines. All benchmark artefacts are available for public download.

MaDI-Bench consists of five end-to-end data integration tasks: Games, Companies, Music, Products, and Scientific Papers. Each task takes several heterogeneous source tables in a domain and asks a system to return one fused target table, exercising schema matching, value normalization, entity matching, and data fusion together. The benchmark provides ground truth to score every step on its own, not only the final table: a gold schema mapping for schema matching, the target schema's constraints and taxonomies for value normalization (measured as consistency with the schema), labeled record pairs for blocking and entity matching, and hand-verified records for data fusion, together with metrics for the end-to-end result. Each base task also ships in easy, medium, and hard variants, for 20 integration tasks in total.

Schema matching  →  Value normalization  →  Entity blocking & matching  →  Data fusion

MaDI provides the following artefacts for each of the base tasks:

• Source tables and table metadata. One table per source, each with a metadata file recording its origin, the columns it contains, and the date it was published. Example: forbes.csv, forbes_metadata.json
• Target schema. The columns of the fused table, with a value constraint per attribute (for example a valid founding-year range) and categorical attributes tied to standard taxonomies such as GICS industries and CLDR country names. Example: target_schema.json
• Gold schema mapping. A correspondence from every source column to the target schema. Example: sm_mapping_gold.json
• Entity matching sets. Labeled train, validation, and test pairs for each pair of sources. Example: forbes_2_dbpedia_train.csv
• Fusion validation and test sets. 100 validation and 100 test entities per task, hand-annotated with the verified value for every attribute. Example: validation_set.xml
• Reference outputs. The output of the human-engineered pipeline, used as a silver standard. Example: companies/base/output/

The sources describe the same kind of entity but rarely agree on column names, value formats, or even the facts themselves, which is what makes the merge hard.

A system under test (SUT) needs to handle the following four subtasks of the overall integration task. MaDI-Bench provides ground truth for each subtask as well as for the output of the overall integration (fused dataset). Thus, a SUT can be evaluated one task at a time or in an end-to-end fashion.

Sources rarely share column names, so every source column has to be matched to a column in the target schema before values can be combined. Names may be descriptive, abbreviated, or meaningless, and where a name gives nothing away, a matcher has to read the values instead.

Matched columns still store values in different formats and vocabularies, so they are rewritten into the forms the target schema requires. The schema sets a constraint for every attribute: a data type, and where it applies a numeric range, a string pattern, or a date format. A founding date, for instance, must be an ISO YYYY-MM-DD inside an allowed range, and assets must be a non-negative integer in US dollars below an upper bound. Categorical attributes are pinned to taxonomies that ship with the task and fix both the allowed values and the level to normalize to: countries follow CLDR, industries follow GICS, and each task adds its own, such as platform, genre, and rating for Games. Normalization is scored as consistency, that is, whether the output values satisfy these schema constraints.

Records that refer to the same entity have to be found across sources. Scoring the full cartesian product of record pairs is quadratic in the input size and computationally infeasible, so blocking first reduces it to a smaller set of candidate pairs, and matching then classifies each candidate as a match or a non-match. The hardest cases are look-alikes that fall into the same block but refer to different entities.

After matching groups the records of an entity into a cluster, fusion resolves the cluster into a single value for each attribute. Where the sources report conflicting values, fusion has to decide which value to keep, and this conflict resolution is the step where the benchmark leaves the most room for improvement. Each gold value was set by annotators who verified it against independent sources.

Beyond the per-step scores, MaDI-Bench rates the final fused table along three quality dimensions: coverage (did the right entities and values make it in?), consistency (does the table fit the target schema?), and correctness (are the merged values right?). Each is reported at three reference levels: reference-free (structure only, no labels needed), silver (against the human pipeline output), and ground truth (against the annotated test sets).

We validate MaDI-Bench with three pipelines that span the design space from human-engineered to fully automated. All three build on the PyDI integration framework, and their per-step and end-to-end outputs are released in the repository under results/ and the per-task output/ folders.

The three pipelines behave very differently across the steps. Schema matching is close to perfect for all three, entity matching covers a wide range across tasks, and data fusion is the hardest step for every pipeline. Every number below is per task, and the strongest pipeline in each row is shown in green. The columns are the human-engineered pipeline (Human, P1), the best-of-breed pipeline (BoB, P2), and the LLM-based pipeline (LLM, P3).

The three pipelines reach close to optimal F1 by using the LLM Schema matcher provided in PyDI, which reads column names and values together. The label-based and instance-based matchers are shown for reference: their much lower scores show that schema matching is genuinely difficult in the benchmark, while state-of-the-art LLMs reach close to perfect performance once they are given both the headers and the values. Values are F1 (%).

Pair completeness stays between 94.4 and 100% for every pipeline, so the candidate sets keep almost all of the labeled test matches. Reduction ratio is above 99% on every task except Products, where the LLM pipeline's embedding blocker prunes 97.5% of the pairs against 71.9% for the other two. Entity matching itself spans a wide range, from near the ceiling on Papers and Music to clearly more open on Games and Products. Values are percentages. The best entity-matching score per task is in green.

Fusion is the hardest step for every pipeline. The LLM pipeline produces the most accurate fused values on four of the five tasks. Best-of-breed leads on Products. Values are accuracy (%). the best per task is in green.

All three pipelines are preconfigured, so the times below cover only the final integration run, not any configuration search or design effort. The main difference comes from entity matching: best-of-breed mostly selects Ditto, a pre-trained language model whose per-task fine-tuning and inference dominate its runtime, while the human and LLM pipelines use lighter matchers. On Companies, where best-of-breed instead picks the cheaper Magellan, it runs about as fast as the others. Values are end-to-end wall-clock seconds.

MaDI-Bench also scores the final fused table as a whole, along three dimensions: coverage (are the right entities and values present), consistency (does the output fit the target schema), and correctness (are the merged values right). Each is reported at three reference levels: reference-free, against the human pipeline (silver), and against the annotated test sets (ground truth). The table gives the mean over the five tasks under the ground-truth reference, with the better pipeline per metric in green.

To keep the benchmark hard as systems improve, each base task also ships in easy, medium, and hard versions. The records and the correct answers stay the same. Eight difficulty knobs perturb the data, each set to its own target for easy, medium, and hard. A harder level turns up the kinds of heterogeneity that real integration runs into.

Here is the DBpedia record for the Taisei Corporation, base against hard. Schema naming divergence (Knob 8) first renames every column to a cryptic code: org_name→nm, established→ey, nation→cn, headquarters→hq, sector→sg, keypeople_name→kpn, total_assets_val→ta, annual_income→ai. On top of that, the values change:

Each knob targets one or two pipeline stages and is set to an easy, medium, or hard target. MaDI-Bench uses eight:

This paper introduced the MaDI-Bench integration benchmark. The benchmark models the full complexity of the data integration process, including schema matching, value normalization, entity matching, and conflict resolution, while at the same time taking into account the dependencies between the subtasks. To prevent a quick saturation of the benchmark as agentic systems progress, we introduce a generic variant-generation method for deriving harder variants from the base tasks. We validated MaDI-Bench and the generated variants using human-designed pipelines, an LLM-based pipeline, and a best-of-breed pipeline. The validation showed the benchmark's utility for measuring the step-wise as well as end-to-end performance of data integration systems. We hope that the benchmark will prove useful for the community and will support the development of fully automatic as well as human-in-the-loop data integration systems.

Tasks are released as plain files (CSV, JSON, and XML), so any system can read them. PyDI adds the integration steps and an evaluator for every stage, so a run can be scored without labeling any data.

• Benchmark repository: all 20 tasks, ground truth, and the validation runs of the three pipelines.
• PyDI: the integration framework and the per-step evaluator classes.
• The tasks: inputs, target schemas, labeled splits, and fused gold records.