Billiger.de Products: A Bilingual Entity Matching Benchmark

Billiger.de Products is an entity matching benchmark requiring matchers to identify product offers that refer to the same real-world product. It contains 13,730 offers describing 2,168 distinct products, collected from the German price-comparison platform billiger.de. The benchmark provides both the original German offers and machine-translated English versions of all training, validation, and test sets, enabling controlled analyses of how language affects entity matching performance. Following the multidimensional design of the WDC Products benchmark, Billiger.de Products provides dataset variants that systematically vary the size of the training set, the proportion of difficult-to-match corner cases, and the proportion of products that are not observed during training. These variants support fine-grained evaluations of data efficiency, robustness to challenging examples, and generalization to previously unseen products. We validate the benchmark using six supervised entity matching methods and GPT-5.2 in a zero-shot setting.

Entity matching is the task of determining whether two records refer to the same real-world entity, for example whether two product offers describe the same product. Widely used entity matching benchmarks, such as Abt-Buy, Amazon-Google, or WDC Products, are monolingual: each contains records in only one language, typically English. However, many real-world applications must process and integrate data in multiple languages. For example, a European shopper searching for the lowest price of a product may not care whether the corresponding offer is published by a German, French, or Italian online retailer. A price-comparison platform serving such users must therefore be able to match product offers both within and across languages. Despite the practical importance of this setting, benchmarks for systematically evaluating entity matching systems on multilingual and cross-lingual data remain scarce.

Billiger.de Products is a step in this direction. The benchmark uses product data from the German price comparison platform billiger.de and is, to the best of our knowledge, the largest entity matching benchmark for German-language product data. In addition to the German original, we provide a machine-translated English version of every record. Splits, labels, and identifiers are identical in both versions, so the benchmark can be used as a German benchmark, as an English benchmark, or as a pair of parallel benchmarks for measuring how the language affects matching performance. Since every German record can also be paired with the English translation of its counterpart, the benchmark is also a starting point for cross-lingual matching experiments.

The benchmark follows the multi-dimensional design of WDC Products and controls task difficulty along three dimensions: the size of the training set, the fraction of hard-to-match corner cases, and the fraction of entities unseen during training. The sections below describe the benchmark in detail and report baseline results for six supervised matchers and for GPT-5.2 in a zero-shot setting on both language versions, as well as a cross-language experiment that evaluates matchers trained on German pairs on mixed German-English record pairs.

The benchmark use product offers originating from the German price comparison portal billiger.de. The benchmark consists of 13,730 product offers describing 2,168 distinct products. Every offer carries the attributes name, description, brand, and price, with the attributes name and price having a value density of 100%, while brand and description have a density above 99 % . Identifiers such as EAN or manufacturer part numbers are deleted from the offers so that matchers need to rely on the textual description of a product rather than identifier lookup.

Following the design of WDC Products, task difficulty is controlled along three dimensions, resulting in 9 training sets, 27 validation sets, and 9 test sets per language:

• Training set size: before leakage cleaning, the small, medium, and large training sets contained 2,500, 6,000, and 26,888–27,393 pairs. The released sets now contain 2,461–2,475, 5,897–5,913, and 26,571–27,112 pairs, respectively. The smaller sets are subsets of the larger ones, and all three are built from the same 500 product groups.
• Corner cases: groups of four highly similar but distinct products generate hard positives and hard negatives. Datasets contain 20 %, 50 %, or 80 % corner-case product groups.
• Unseen entities: in the Seen, Half-Seen, and Unseen test sets, 0 %, 50 %, or 100 % of the to-be-matched products do not occur in the training data. The Half-Seen and Unseen test sets evaluate the ability of a matcher to generalize to new and thus unseen products.
• Two languages: every record exists in the German original and in an English machine translation. Splits, labels, and identifiers are identical, so performance differences are attributable to the language of the records and the translation process.

The products belong to thirteen different product categories. The figure below shows the amount of product offers that belong to each category. Clothing, furniture, and electronics account for the largest shares.

The record pairs in the training, validation, and test sets are either labeled as matches or non-matches. The labels are derived from a grouping (clustering) of offers that refer to the same product which was provided to us by billiger.de. Positive pairs (matches) are directly taken from these groups, negative pairs (non-matches) are generated by combining offers from one group with similar offers from other groups, resulting in hard negatives, with one randomly choosen negative per group. The amount of hard negatives per product grows with the training set size.

The test sets contain roughly 4,440 pairs each with a class imbalance of 1:10 to 1:12 between matches and non-matches. All nine test sets went through a two-round manual verification process involving two human annotators, who checked the pairs using a strict definition of what constitutes a match: product variants that differ in size, color, configuration, capacity, or package quantity were labeled as non-matches. The audit corrected pairs in which similar but distinct product variants had been grouped together.

The following matching and non-matching pairs illustrate the variety of product categories and comparison difficulties contained in the benchmark.

A real corner case from the benchmark: two wardrobe offers that agree in brand, model name, color, and almost the entire description, yet refer to different product variants (168 cm vs. 85 cm width). Under the strict identity definition of the benchmark, this pair is a non-match. Matchers that rely on surface similarity are reliably fooled by such pairs.

Two offers for the same De'Longhi coffee machine use different spelling, word order, and levels of detail, but share the identifying model number ECAM 650.55 MS. This pair is a match.

These two METABO power tools share a brand and technical vocabulary, but the model numbers and product types conflict: one is a mitre saw, the other a rotary hammer. The hard-negative pair is therefore a non-match.

Six supervised matchers spanning four methodological generations (word co-occurrence, feature-based machine learning, fine-tuned transformers, contrastive and graph-based architectures) were evaluated across all 27 configurations of both language versions, plus GPT-5.2 in a strict zero-shot setting. The table shows F1 scores on the German benchmark for all three training sizes. GPT-5.2 uses no training data, and results are shown for both the Simple and the Rule-Guided prompt.

The baseline results underline the difficulty of the benchmark. The traditional feature-based approaches WordCooc and Magellan average only 43.89 and 42.85 F1 across the 27 German variants, which is too low for practical use on this benchmark. Neural matchers such as Ditto reach up to 91.76 F1 in favorable configurations, but, as reported for WDC Products, they remain strongly affected by unseen entities. Averaged across all corner-case ratios and training sizes, Ditto loses 19.53 F1 from Seen to Unseen test sets. Even zero-shot GPT-5.2 does not solve the benchmark: with the stronger Rule-Guided prompt, its scores remain between 77.78 and 84.91 F1. The benchmark therefore remains challenging for traditional matchers, supervised neural models, and the evaluated LLM.

The three difficulty dimensions separate the benchmark variants as intended. F1 generally increases with the training set size, decreases as the corner-case ratio grows from 20 % to 80 %, and decreases from the Seen to the Unseen test sets. Individual medium-to-large results can deviate from the overall training-size trend. Comparing the Large, 20 % corner-case, Seen row with the Small, 80 % corner-case, Unseen row, the losses range from 22.94 F1 for Magellan (55.73 to 32.79) to 61.01 F1 for RoBERTa (84.95 to 23.94). In the Large, 20 % corner-case rows, the Seen-to-Unseen drops for R-SupCon, Ditto, and RoBERTa are 35.68, 26.52, and 25.71 F1, compared with 26.59, 9.60, and 9.16 F1 reported for WDC Products. The broader range of product categories may contribute to the larger drops, but this comparison does not establish their cause.

Ditto and HierGAT are the strongest supervised systems and reach 91.76 and 89.41 F1 on the easiest German variant. Zero-shot GPT-5.2 behaves differently. Under the Rule-Guided prompt its F1 varies by 7.13 points across all difficulty settings (19.15 under the Simple prompt), and it leads on every Unseen test set, where supervised models lose the training signal they depend on. Supervised matchers overtake GPT-5.2 only on Seen test sets with medium or large training data. A caveat applies to interpreting the GPT-5.2 results on the Seen/Unseen dimension. GPT-5.2 was trained on a large, undisclosed corpus, and all products in the benchmark were publicly available online. Some or all individual product records may therefore have appeared in its pre-training data. Here, Seen and Unseen only indicate whether the products occur in the benchmark's supervised training sets, not whether GPT-5.2 encountered them during pre-training.

Performance by Product Category

The difficulty of matching decisions depends on the product category. To avoid selecting a different representative split for each model, the table now averages same-category F1 over all eligible German benchmark configurations. Electronics & Computers ranks among the two easiest categories for all seven models. Clothing & Accessories is the most difficult category for Magellan, RoBERTa, R-SupCon, and GPT-5.2 and the second most difficult for HierGAT and Ditto. Furniture & Living ranks among the two most difficult categories for five models. Distinctive model numbers and technical specifications may contribute to the stronger results for electronics, while variant-heavy, near-duplicate descriptions may contribute to the lower results for clothing and furniture.

Mean same-category F1 across all eligible German benchmark configurations. A category contributes only where its same-category pair has at least 45 test pairs. Supervised scores are averaged over available runs within each configuration. WordCooc and Magellan use the selected classifier for that configuration. GPT-5.2 uses the Rule-Guided prompt. Automotive, Electronics, Clothing, Furniture, and Toys cover all 9 test sets. Sports and Tools cover 8, and Cosmetics covers 7.

In addition to the German version of the benchmark, we also provide a Englisch version of the benchmark. The Englisch version was created by translating the German product offers to English using GPT-5.2. The English version offers exactly the same variants and the same splits as the German version. The table below compares the F1 results of the eight matchers on the English version to the results on the original German version. The table reports the difference of the F1 score on the English version minus the F1 score on the German version for all 27 benchmark variants.

The main result is that changing the language rarely changes the practical assessment of a matcher. WordCooc and Magellan remain below 46 mean F1 in both languages, which is too low for practical matching on this benchmark. Their small mean language differences do not alter that conclusion. More generally, the mean language difference remains within ±2.31 F1 for every matcher except Ditto. GPT-5.2 is similarly stable: the mean differences are +0.27 F1 with the Simple prompt and +0.69 F1 with the Rule-Guided prompt, and no individual difference exceeds 3.34 F1. At this scale, the results do not suggest a substantial systematic language effect for GPT-5.2.

Ditto is the notable exception. It scores 5.73 F1 lower on average on the English version, which is surprising because its RoBERTa backbone was pre-trained primarily on English text. The English disadvantage is largest with Small training sets at 8.27 F1 on average and narrows to 4.63 with Medium and 4.28 with Large training sets. Although the disadvantage becomes smaller as the training set grows, the results do not explain why Ditto performs worse on English overall.

The broader training-size pattern provides partial support for the same interpretation. Across the six supervised matchers, the mean absolute language gap decreases from 5.25 F1 with Small training sets to 3.77 with Medium and 2.76 with Large sets. RoBERTa's extreme differences account for part of this trend. After excluding RoBERTa, the corresponding values are 4.36, 2.65, and 2.86 F1. Language sensitivity therefore tends to be greater when little training data is available, but the decrease is not consistent for every matcher. This pattern does not confirm a general English advantage: even with Small training sets, fewer than half of the non-RoBERTa comparisons favor English.

Comparing the results of the matchers on the English version of the billiger.de benchmark to the results of the same matchers on the WDC Products Benchmark, which also features English product data, shows that the F1 results on billiger.de are generally lower, meaning that billiger.de Products (English) is more difficult than WDC Products. A reason for this might that Billiger.de Products spans a broader range of product categories, whereas WDC Products is dominated by offers for electronic products.

The comparison above changes the language of both records at once. In applications such as European price comparison, matchers also have to compare records across languages. This section evaluates this setting directly: all supervised matchers are trained on the German 80 % corner-case large training set (26,571 pairs) and selected on the corresponding German validation set (4,436 pairs). Each matcher is then evaluated on five variants of the 50 % unseen test set (4,437 pairs each) in which only the language of the records changes. In DE-DE both records are German, in DE-EN and EN-DE one record is German and the other English, in EN-EN both records are English, and Random mixes the four combinations evenly (1,109 or 1,110 pairs each). Pair IDs, labels, and the seen/unseen composition are identical in all five variants, so the differences between the columns are attributable to the language combination alone.

WordCooc and Magellan both lose performance once records leave the training language, but to a different degree. WordCooc drops from 43.6 F1 on DE-DE to 28.2 on EN-EN, since translation removes the exact token overlap it relies on. Magellan stays within 3.1 F1 of its DE-DE score across all variants, as its character-level similarity features depend less on the language. Both matchers operate between 28.2 and 43.6 F1 on this test set, so these differences carry limited practical weight. R-SupCon is stable across the four variants that still contain German records (45.2 to 48.5 F1) and only loses performance on the fully English variant (42.3), though again at a low absolute level.

HierGAT reaches its best score on the German pairs it was trained on (60.8 F1), closely followed by the fully English variant (57.6), while both mixed variants drop to around 49 F1. RoBERTa and Ditto show the same ordering, and for all three models the Random variant, in which half of the pairs are same-language, lies in between. The models therefore do not fail on English data as such. As long as both records switch language together, the cues learned on German pairs, such as overlapping tokens between the two names and descriptions, stay intact, and the English pre-training of the underlying language models likely supports the transfer. In a mixed pair, the translation removes this surface overlap on one side only, and pairs that cross the language boundary never occur in the German training data. For a price comparison portal this distinction matters, because the cross-border case is exactly the mixed-language case. Translating one record into the language of the other before matching may recover much of the loss, although the experiment does not test this.

GPT-5.2 delivers the best scores in every variant: 80.9 to 83.5 F1 with the Rule-Guided prompt and 70.5 to 72.9 with the Simple prompt. Taking run-to-run variation into account, the differences between the language combinations are minimal, so the results do not indicate a language effect for the zero-shot models. For applications that have to match offers across languages, and where the budget allows it, a zero-shot LLM is therefore the most reliable choice among the evaluated systems.

The benchmark is distributed as gzipped JSON files with one record pair per line. Each line contains both product records, the binary match label, and a hard-negative indicator. The file name encodes the configuration: products80cc20rnd000un_train_large.json.gz denotes 80 % corner cases, 20 % random pairs, and the large training split. Training files always use 000un. The unseen-product share is defined by validation and test products relative to the training data. The files can be read directly with pandas:

import pandas as pd

pairs = pd.read_json(
    "products80cc20rnd000un_train_large.json.gz",
    compression="gzip",
    lines=True,
)

The complete reproduction code is available in the GitHub repository. From the repository root, install the fixed dependencies and generate all model-specific inputs with:

python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install -r environments/requirements.txt

python src/processing/prepare_pairs.py
python src/processing/prepare_pretraining.py
python src/processing/prepare_ditto_hiergat.py
python src/processing/prepare_wordcooc.py
python src/processing/prepare_magellan.py

The launchers in slurm_runs/ and the model directories run the complete German and English experiment matrix with three fixed seeds. All generated metrics are written below results/generated/ and can be collected into one CSV with:

python src/summarize_results.py

Exact commands for WordCooc, Magellan, RoBERTa, R-SupCon, Ditto, HierGAT, and GPT-5.2 are documented in REPRODUCTION.md. The GPT-5.2 runs require an OPENAI_API_KEY and use the Batch API.

For supervised matchers, pick a training set, select hyperparameters on the corresponding validation set, and report F1 on the nine test sets. The test sets are the same for all training set sizes, so results obtained with different amounts of training data remain comparable.

For zero-shot and in-context learning, demonstration examples should be drawn from the training or validation sets.

For language comparisons, run the same configuration on the German and the English version. Splits, labels, and identifiers are identical, so any difference in the scores is attributable to the language of the records and the translation process. Since the record identifiers line up across the two versions, German records can also be paired with English ones to build cross-lingual matching tasks.

Both language versions are available as zip archives containing all training, validation, and test sets described above. The five aligned test set variants of the cross-language experiment are available as a separate archive.

Billiger.de Products extends product matching evaluation beyond English with a large German benchmark and an aligned English translation spanning thirteen product categories. The 27 variants of the benchmark vary the training set size, the share of corner cases, and the fraction of unseen products in the test set.

The validation of the benchmark showed that both the benchmark variant and the product category influence the results. Supervised matchers generally benefit from the larger training sets and score lower as the share of corner-cases or the amount of unseen entities in the test set increases. Tools and electronics are among the easiest product categories, whereas for clothing and furniture matchers generally achieve lower scores.

The comparison of the performance of the matchers on billiger.de Products (English) to the performance of the same matchers on WDC Products showed that Billiger.de Products is more difficult than WDC Products. This feature might make the English version of Billiger.de Products an interesting alternative to WDC Products for matchers that have saturated WDC Products.

The cross-language experiment showed that supervised matchers trained on German pairs keep most of their performance when both records switch to English but lose about 10 F1 points on mixed German-English pairs. Zero-shot GPT-5.2 reaches the best scores on all language combinations and shows no language effect, which makes LLMs the most reliable choice for cross-language matching among the evaluated systems.

We hope that the benchmark will support the evaluation and development of entity matching methods for monolingual German and English settings, as well as for cross-lingual matching scenarios.

Questions and feedback are welcome. Please contact Aaron Steiner or open an issue in the GitHub repository.

A paper describing the benchmark will be published on arXiv in August 2026. Citation information will be added here as soon as it is available.