MaDI-Bench: An End-to-End Data Integration Benchmark

Data integration combines heterogeneous data from multiple sources into a single, coherent data set. Data integration involves a sequence of interdependent tasks: schema matching, value normalization, entity blocking, entity matching, and data fusion. Existing data integration benchmarks either evaluate these steps in isolation or cover only incomplete versions of the data integration pipeline, omitting specific steps. The lack of public end-to-end data integration benchmarks hinders research on data integration methods that address the integration process as a whole.

The Mannheim Data Integration Benchmark (MaDI-Bench) fills this gap. MaDI-Bench is the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process. MaDI-Bench contributes (i) a set of base end-to-end data integration tasks spanning several application domains, each requiring the full schema matching, value normalization, entity matching, and conflict resolution pipeline; and (ii) a generic method for deriving variants of the base tasks that mitigates the rapid saturation of the benchmark as data integration systems advance. We validate MaDI-Bench using human-engineered pipelines, a best-of-breed pipeline, and an LLM-based pipeline. The validation demonstrates the utility of the benchmark for measuring the step-wise as well as the end-to-end performance of data integration pipelines. All benchmark artefacts are available for public download.

MaDI-Bench consists of five end-to-end data integration tasks. These base task require the integration of data about Games, Companies, Music, Products, and Scientific Papers. Each task takes several heterogeneous tables as input and asks the system under test (SUT) to integrate these tables into a single output table conforming to a given target schema. The integration requires the SUT to perform schema matching, value normalization, entity matching, and data fusion. The benchmark provides ground truth to score every step on its own, not only the final table: a gold schema mapping for schema matching, the target schema's constraints and taxonomies for value normalization (measured as consistency with the schema), labeled record pairs for blocking and entity matching, and hand-verified records for data fusion, together with metrics for the end-to-end result. MaDI-Bench offers easy, medium, and hard variants of each base task.

Schema matching  →  Value normalization  →  Entity blocking & matching  →  Data fusion

MaDI-Bench provides the following artifacts for each of the base tasks:

• Source tables and table metadata. One table per source, each with a metadata file recording its origin, the columns it contains, and the date it was published. Example: forbes.csv, forbes_metadata.json
• Target schema. The columns of the fused table, with a value constraint per attribute (for example a valid founding-year range) and categorical attributes tied to standard taxonomies such as GICS industries and CLDR country names. Example: target_schema.json
• Gold schema mapping. A correspondence from every source column to the target schema. Example: sm_mapping_gold.json
• Entity matching sets. Labeled train, validation, and test pairs for each pair of sources. Example: forbes_2_dbpedia_train.csv
• Fusion validation and test sets. 100 validation and 100 test entities per task, hand-annotated with the verified value for every attribute. Example: validation_set.xml
• Reference outputs. The output of the human-engineered pipeline, used as a silver standard. Example: companies/base/output/

The sources describe the same kind of entity but rarely agree on column names, value formats, or even the facts themselves, which is what makes the merge hard.

A system under test (SUT) needs to handle the following four steps of the overall integration task. MaDI-Bench provides ground truth or a reference signal for each step as well as for the output of the overall integration (fused dataset). Thus, a SUT can be evaluated one step at a time or in an end-to-end fashion.

Sources rarely share column names, so every source column has to be matched to a column in the target schema before values can be combined. Names may be descriptive, abbreviated, or meaningless, and where a name gives nothing away, a matcher has to read the values instead.

Matched columns still store values in different formats and vocabularies, so they are rewritten into the forms the target schema requires. The schema sets a constraint for every attribute: a data type, and where it applies a numeric range, a string pattern, or a date format. A founding date, for instance, must be an ISO YYYY-MM-DD inside an allowed range, and assets must be a non-negative integer in US dollars below an upper bound. For categorical attributes, the task provides taxonomies that specify which values are allowed and how detailed the normalized values should be. Some attributes use standard taxonomies, e.g. countries follow CLDR, industries uses GICS. In addition, some tasks also use propretary taxonomies, for instance for platform, genre, and rating in the Games task. Normalization is scored as consistency, that is, whether the output values satisfy the given schema constraints.

The SUT must identify records referring to the same real-world entity across sources. Since comparing all record pairs requires scoring the full Cartesian product and is therefore quadratic in the input size, this quickly becomes computationally infeasible. Blocking addresses this issue by reducing the search space to a smaller set of candidate pairs, after which matching classifies each candidate as either a match or a non-match. The MaDI-Bench entity matching test sets are designed to contain a decent amount of hard-to-decide corner case record pairs.

After matching, records referring to the same real-world entities form clusters. Data fusion consolidates these clusters into a single record per cluster. When records provide conflicting values, the SUT must apply a conflict resolution heuristic to decide which value to retain. The values in the MaDI-Bench data fusion test sets are verified by humans who checked them against data from relevant websites.

Beyond the per-step evaluation, MaDI-Bench also provides code for assessing the final output table along three quality dimensions: coverage (did the right entities and values make it into the output table?), consistency (does the table fit the target schema?), and correctness (are the fused values correct?). MaDi-Bench provides for assessing these quality dimension using different types of reference data: reference-free (no ground truth used, only structural comparison to imput), silver (comparison to the output of the human-designed pipelines), and ground truth (evaluation against the human-verified test sets). While the silver evaluation checks all values of the output table, the round truth-based evaluation covers only a subset of all records.

We validate MaDI-Bench using three pipelines that span the design space from human-engineered to fully automated. The human-engineered and LLM-based pipelines build on the PyDI integration framework, while the best-of-breed pipeline chains specialized systems from the literature. The per-step and end-to-end outputs of all three pipelines are found in the repository under results/ and the per-task output/ folders.

The three pipelines behave very differently across the steps. Schema matching is close to perfect for all three, entity matching covers a wide range across tasks, and data fusion is the hardest step for every pipeline. Every number below is per task, and where the pipelines differ, the best score per task is shown in green. The columns are the human-engineered pipeline (Human, P1), the best-of-breed pipeline (BoB, P2), and the LLM-based pipeline (LLM, P3).

The three pipelines reach close to optimal F1 by using the PyDI LLM schema matcher, which reads column names and values together. The label-based and instance-based matchers are shown for reference: their much lower scores show that schema matching is genuinely difficult in the benchmark, while state-of-the-art LLMs reach close to perfect performance once they are given both the headers and the values. Values are F1 (%).

Pair completeness stays between 94.4 and 100% for every pipeline, so the candidate sets keep almost all of the labeled test matches. Reduction ratio is above 99% on every task except Products, where the LLM pipeline's embedding blocker prunes 97.5% of the pairs against 71.9% for the other two. Entity matching itself spans a wide range, from near the ceiling on Papers and Music to clearly more open on Games and Products. Values are percentages. The best entity-matching score per task is in green.

Fusion is the hardest step for every pipeline. The LLM pipeline produces the most accurate fused values on four of the five tasks. Best-of-breed leads on Products. Values are accuracy (%). The best score per task is in green. The strong results of the LLM-based pipeline are due not only to its choice of conflict resolution method, but also to its more comprehensive value normalization.

All three pipelines are preconfigured, so the times below cover only the final integration run, not any configuration search or design effort. The main difference comes from entity matching: best-of-breed mostly selects Ditto, a pre-trained language model whose per-task fine-tuning and inference dominate its runtime, while the human and LLM pipelines use lighter matchers. On Companies, where best-of-breed instead picks the cheaper Magellan, it runs about as fast as the others. Values are end-to-end wall-clock seconds.

MaDI-Bench also scores the final fused table as a whole, along three dimensions: coverage (are the right entities and values present), consistency (does the output fit the target schema), and correctness (are the merged values right). The table shows mean values over the five tasks calculated using ground-truth test sets. The results of the better pipeline per metric are set in green.

To keep the benchmark hard as systems improve, each base task also ships in easy, medium, and hard versions. The underlying entities and their true matches stay the same. Eight difficulty knobs perturb the data, each set to its own target for easy, medium, and hard. A harder level turns up the kinds of heterogeneity that real integration runs into.

Here is the DBpedia record for the Taisei Corporation, base against hard. Schema naming divergence (Knob 7) first renames every column to a cryptic code: org_name→nm, established→ey, nation→cn, headquarters→hq, sector→sg, keypeople_name→kpn, total_assets_val→ta, annual_income→ai. On top of that, the values change:

Each knob targets one or two pipeline stages and is set to an easy, medium, or hard target. MaDI-Bench uses eight:

This paper introduced the MaDI-Bench integration benchmark. The benchmark models the full complexity of the data integration process, including schema matching, value normalization, entity matching, and conflict resolution, while at the same time taking into account the dependencies between the subtasks. To prevent a quick saturation of the benchmark as agentic systems progress, we introduce a generic variant-generation method for deriving harder variants from the base tasks. We validated MaDI-Bench and the generated variants using human-designed pipelines, an LLM-based pipeline, and a best-of-breed pipeline. The validation showed the benchmark's utility for measuring the step-wise as well as end-to-end performance of data integration systems. We hope that the benchmark will prove useful for the community and will support the development of fully automatic as well as human-in-the-loop data integration systems.

Tasks are released as plain files (CSV, JSON, and XML), so any system can read them. PyDI adds the integration steps and an evaluator for every stage, so a run can be scored without labeling any data.

• Benchmark repository: all 20 tasks, ground truth, and the validation runs of the three pipelines.
• PyDI: the integration framework and the per-step evaluator classes.
• The tasks: inputs, target schemas, labeled splits, and fused gold records.