Billiger.de Products: A Bilingual Entity Matching Benchmark

Billiger.de Products is an entity matching benchmark requiring matchers to identify product offers that refer to the same real-world product. It contains 13,568 offers describing 2,168 distinct products, collected from the German price-comparison platform billiger.de. The benchmark provides both the original German offers and machine-translated English versions of all training, validation, and test sets, enabling controlled analyses of how language affects entity matching performance. Following the multidimensional design of the WDC Products benchmark, Billiger.de Products provides dataset variants that systematically vary the size of the training set, the proportion of difficult-to-match corner cases, and the proportion of products that are not observed during training. These variants support fine-grained evaluations of data efficiency, robustness to challenging examples, and generalization to previously unseen products. We validate the benchmark using six supervised entity matching methods and GPT-5.2 in a zero-shot setting.

Entity matching is the task of determining whether two records refer to the same real-world entity, for example whether two product offers describe the same product. Widely used entity matching benchmarks, such as Abt-Buy, Amazon-Google, or WDC Products, are monolingual: each contains records in only one language, typically English. However, many real-world applications must process and integrate data in multiple languages. For example, a European shopper searching for the lowest price of a product may not care whether the corresponding offer is published by a German, French, or Italian online retailer. A price-comparison platform serving such users must therefore be able to match product offers both within and across languages. Despite the practical importance of this setting, benchmarks for systematically evaluating entity matching systems on multilingual and cross-lingual data remain scarce.

Billiger.de Products is a step in this direction. The benchmark uses product data from the German price comparison platform billiger.de and is, to the best of our knowledge, the largest entity matching benchmark for German-language product data. In addition to the German original, we provide a machine-translated English version of every record. Splits, labels, and identifiers are identical in both versions, so the benchmark can be used as a German benchmark, as an English benchmark, or as a pair of parallel benchmarks for measuring how the language affects matching performance. Since every German record can also be paired with the English translation of its counterpart, the benchmark is also a starting point for cross-lingual matching experiments.

The benchmark follows the multi-dimensional design of WDC Products and controls task difficulty along three dimensions: the size of the training set, the fraction of hard-to-match corner cases, and the fraction of entities unseen during training. The sections below describe the benchmark in detail and report baseline results for six supervised matchers and for GPT-5.2 in a zero-shot setting on both language versions, as well as a cross-language experiment that evaluates matchers trained on German pairs on mixed German-English record pairs.

The benchmark uses product offers originating from the German price comparison portal billiger.de. The benchmark (version 1.1) consists of 13,568 product offers describing 2,168 distinct products. Every offer carries the attributes name, description, brand, and price. Name and price are available in every offer, while description has a value density of over 99 % and brand of over 94 % in all variants. Identifiers such as EAN or manufacturer part numbers are deleted from the offers so that matchers need to rely on the textual description of a product rather than identifier lookup.

Following the design of WDC Products, task difficulty is controlled along three dimensions, resulting in 9 training sets, 27 validation sets, and 9 test sets per language:

• Training set size: the small, medium, and large training sets contain 2,461 to 2,475, 5,897 to 5,913, and 26,571 to 27,112 pairs, respectively. The smaller sets are subsets of the larger ones, and all three are built from the same 500 products.
• Corner cases: groups of five highly similar but distinct products generate hard positives and hard negatives. Datasets contain 20 %, 50 %, or 80 % corner-case product groups.
• Unseen entities: in the Seen, Half-Seen, and Unseen test sets, 0 %, 50 %, or 100 % of the to-be-matched products do not occur in the training data. The Half-Seen and Unseen test sets evaluate the ability of a matcher to generalize to new and thus unseen products.
• Two languages: every record exists in the German original and in an English machine translation. Splits, labels, and identifiers are identical, so performance differences are attributable to the language of the records and the translation process.

The products belong to thirteen different product categories. The figure below shows the amount of product offers that belong to each category. Furniture, clothing, and electronics account for the largest shares.

The record pairs in the training, validation, and test sets are either labeled as matches or non-matches. The labels are derived from a grouping (clustering) of offers that refer to the same product which was provided to us by billiger.de. Positive pairs (matches) are directly taken from these groups, negative pairs (non-matches) are generated by combining offers from one group with similar offers from other groups, resulting in hard negatives, with one randomly chosen negative per group. The amount of hard negatives per product grows with the training set size.

The test sets contain 4,372 to 4,458 pairs each with a class imbalance of about 1:10 to 1:13 between matches and non-matches. All nine test sets went through a two-round manual verification process involving two human annotators, who checked the pairs using a strict definition of what constitutes a match: product variants that differ in size, color, configuration, capacity, or package quantity were labeled as non-matches. The audit corrected pairs in which similar but distinct product variants had been grouped together. To assess the resulting label quality, one author re-annotated a stratified random sample of 100 test pairs, 50 matches and 50 non-matches, blind to the released labels. The re-annotation agreed with all 50 match labels and with 49 of the 50 non-match labels, which corresponds to an estimated disagreement rate of 1.9 % when weighted by the class imbalance of the test pairs.

The following matching and non-matching pairs illustrate the variety of product categories and comparison difficulties contained in the benchmark.

A real corner case from the benchmark: two wardrobe offers that agree in brand, model name, color, and almost the entire description, yet refer to different product variants (168 cm vs. 85 cm width). Under the strict identity definition of the benchmark, this pair is a non-match. Matchers that rely on surface similarity are reliably fooled by such pairs.

Two offers for the same De'Longhi coffee machine use different spelling, word order, and levels of detail, but share the identifying model number ECAM 650.55 MS. This pair is a match.

These two METABO power tools share a brand and technical vocabulary, but the model numbers and product types conflict: one is a mitre saw, the other a rotary hammer. The hard-negative pair is therefore a non-match.

Six supervised matchers spanning four methodological generations (word co-occurrence, feature-based machine learning, fine-tuned transformers, contrastive and graph-based architectures) were evaluated across all 27 configurations of both language versions, plus GPT-5.2 in a strict zero-shot setting. The table shows F1 scores on the German benchmark for all three training sizes. GPT-5.2 uses no training data, and results are shown for both the Simple and the Rule-Guided prompt.

The baseline results underline the difficulty of the benchmark. The traditional feature-based approaches WordCooc and Magellan average only 41.67 and 42.04 F1 across the 27 German variants, which is too low for practical use on this benchmark. Neural matchers such as Ditto reach up to 86.66 F1 in favorable configurations, but, as reported for WDC Products, they remain strongly affected by unseen entities. Averaged across all corner-case ratios and training sizes, Ditto loses 21.17 F1 from Seen to Unseen test sets. Even zero-shot GPT-5.2 does not solve the benchmark: with the stronger Rule-Guided prompt, its scores remain between 77.99 and 84.88 F1. The benchmark therefore remains challenging for traditional matchers, supervised neural models, and the evaluated LLM.

The three difficulty dimensions separate the benchmark variants as intended. F1 generally increases with the training set size, decreases as the corner-case ratio grows from 20 % to 80 %, and decreases from the Seen to the Unseen test sets. Individual medium-to-large results can deviate from the overall training-size trend. Comparing the Large, 20 % corner-case, Seen row with the Small, 80 % corner-case, Unseen row, the losses range from 24.60 F1 for Magellan (57.03 to 32.43) to 55.47 F1 for R-SupCon (84.85 to 29.38). In the Large, 20 % corner-case rows, the Seen-to-Unseen drops for R-SupCon, Ditto, and RoBERTa are 39.16, 25.75, and 26.88 F1, compared with 26.59, 9.60, and 9.16 F1 reported for WDC Products. The broader range of product categories may contribute to the larger drops, but this comparison does not establish their cause.

Ditto and HierGAT are the strongest supervised systems and reach 86.66 and 83.58 F1 on the easiest German variant. Zero-shot GPT-5.2 behaves differently. Under the Rule-Guided prompt its F1 varies by 6.89 points across all difficulty settings (20.02 under the Simple prompt), against 24.60 to 55.47 points for the supervised models. With the Rule-Guided prompt, GPT-5.2 reaches the highest F1 in 24 of 27 German variants and leads on every Unseen test set, where supervised models lose the training signal they depend on. The Simple prompt leads on the Small and Medium Seen variants at 20 % corner cases. Supervised matchers overtake GPT-5.2 only on the easiest variant (Large, 20 % corner cases, Seen), where Ditto is ahead by 0.39 F1. Averaged over the nine German test sets, GPT-5.2 reaches 76.66 F1 with the Simple and 81.30 F1 with the Rule-Guided prompt. A caveat applies to interpreting the GPT-5.2 results on the Seen/Unseen dimension. GPT-5.2 was trained on a large, undisclosed corpus, and all products in the benchmark were publicly available online. Some or all individual product records may therefore have appeared in its pre-training data. Here, Seen and Unseen only indicate whether the products occur in the benchmark's supervised training sets, not whether GPT-5.2 encountered them during pre-training.

Performance by Product Category

The difficulty of matching decisions depends on the product category. The table reports the same-category F1 per category, averaged over all nine German test sets, together with the number of records per category in the training and test pairs of the recommended configuration. The category ranking is stable across the model families. Among the nine largest categories, Clothing & Accessories and Furniture & Living are the two hardest for every model, and Clothing is the single hardest category for five of the eight model columns. Electronics & Computers ranks among the three easiest categories for every model and Health & Care for seven of the eight model columns, while Tools & DIY and Toys & Baby are in the upper half for every model. Distinctive model numbers and standardized technical specifications give both lexical and neural matchers strong signals in the easier categories, while variant-heavy, near-duplicate descriptions weaken matching for clothing and furniture.

Matching difficulty does not track the amount of training data: Health & Care, with 1,057 training records against more than 9,000 in each of the three largest categories, scores above Clothing and Furniture for every model and above Electronics for six of the eight model columns. Since electronics rank among the easiest categories, evaluations on electronics-centric benchmarks are likely to overestimate matching performance on broader product assortments.

Mean same-category F1 on the German version, averaged over all nine test sets and, per test set, over seeds and the three development set sizes. Train and Test give the number of records in the pairs of the recommended configuration, i.e. the large 80 % corner-case training set and the Half-Seen test set. Each pair contributes its two records, so an offer is counted once per pair it occurs in and the columns sum to twice the number of pairs. Rows are ordered by the number of training records. WordCooc and Magellan use the selected classifier per configuration. Pet Supplies is omitted, as it has no records in the recommended configuration. Bold marks the best value per category. ^†Categories with fewer than 100 test records in the recommended configuration, reported for completeness.

The category ranking is largely language-independent. Repeating the analysis on the English benchmark version for WordCooc, Magellan, RoBERTa, R-SupCon, and GPT-5.2 yields the same picture: the ranking of the categories stays largely unchanged, and the per-category shifts are of the same order as each matcher's overall language difference reported in the next section. The table below reports the same-category F1 on the English version.

Mean same-category F1 on the English version, computed with the same aggregation as the German table above. The record counts per category are identical to the German version, as both language versions share all splits and identifiers. WordCooc and Magellan use the classifier selected on the English validation sets. The English category analysis covers WordCooc, Magellan, RoBERTa, R-SupCon, and GPT-5.2; HierGAT and Ditto are not included. Bold marks the best value per category. ^†Categories with fewer than 100 test records in the recommended configuration, reported for completeness.

In addition to the German version of the benchmark, we also provide an English version of the benchmark. The English version was created by translating the German product offers to English using the GPT-5-mini model. The English version offers exactly the same variants and the same splits as the German version. The table below compares the F1 results of the eight matchers on the English version to the results on the original German version. The table reports the difference of the F1 score on the English version minus the F1 score on the German version for all 27 benchmark variants.

The main result is that changing the language rarely changes the practical assessment of a matcher. WordCooc and Magellan remain below 46 mean F1 in both languages, which is too low for practical matching on this benchmark. Averaged across all variants, every transformer-based matcher scores higher on the English version, and GPT-5.2 is nearly indifferent to the language: its mean differences are +0.03 F1 with the Simple prompt and +0.42 F1 with the Rule-Guided prompt, and no individual difference exceeds 3.40 F1. This largest GPT-5.2 difference stems from single runs per variant, and the cross-language section below reports a run-to-run spread of 1.10 F1 for repeated DE-DE requests, so a small language effect under the Rule-Guided prompt remains plausible.

WordCooc shows a slight advantage on English that is most visible on the Unseen test sets, reaching 6.64 F1 (Large, 20 % corner cases), possibly because German compounds reduce exact token overlap between matching records. Magellan instead slightly favours German (mean −1.47 F1). RoBERTa favours English with a mean difference of +3.99 F1 and English ahead in 25 of 27 variants, consistent with its predominantly English pre-training. The advantage is largest with Small training sets (+5.49 F1) and shrinks to +2.27 F1 with Large sets, so fine-tuning an English model on German data costs only little matching performance once enough training data is available. HierGAT shows the largest mean difference (+7.21 F1, English ahead in 25 of 27 variants), while Ditto (+1.41 F1) and R-SupCon (+0.62 F1) are close to parity, although all three fine-tune the same RoBERTa backbone.

Across the six supervised matchers, the mean absolute language gap decreases from 5.24 F1 with Small training sets to 3.52 with Medium and 2.40 with Large sets. Language sensitivity therefore tends to be larger when little training data is available.

The multilingual encoder XLM-R, evaluated on the 80 % corner-case variants, scores above RoBERTa on German in all nine cells, by 1.9 to 9.6 F1, with the largest gains on the Small training sets, and stays within 3.8 F1 of RoBERTa on English, with RoBERTa ahead on the Large sets. The average language difference is small with the multilingual backbone: over the nine cells, XLM-R scores 0.69 F1 higher on English than on German, RoBERTa 5.79 F1. This indicates that the English advantage of RoBERTa stems mainly from its backbone rather than from differences between the German and the translated English data. For matching in a non-English market, a multilingual encoder is the better starting point: it gains 1.9 to 9.6 F1 on German at a cost of at most 3.2 F1 on the Large English sets.

F1 of RoBERTa and XLM-R (xlm-roberta-base, fine-tuned with the RoBERTa pipeline at learning rate 2e-5, warmup ratio 0.1, early-stopping patience 25, and batch size 32) on the 80 % corner-case variants of both language versions, mean over three seeds.

Comparing the results of the matchers on the English version of the billiger.de benchmark to the results of the same matchers on the WDC Products Benchmark, which also features English product data, shows that the F1 results on billiger.de are generally lower: averaged over the 27 shared configurations, RoBERTa, Ditto, HierGAT, and R-SupCon score 9.8 to 19.6 F1 lower on the English version of Billiger.de Products than on WDC Products, while the ranking of the six matchers is the same on both benchmarks. Billiger.de Products (English) is therefore more difficult than WDC Products. Even on the easiest English configuration (20 % corner cases, Large, Seen), RoBERTa, Ditto, and R-SupCon score 1.2 to 8.7 F1 below their published Abt-Buy results. A reason for this might be that Billiger.de Products spans a broader range of product categories, whereas WDC Products is dominated by offers for electronic products.

The comparison above changes the language of both records at once. In applications such as European price comparison, matchers also have to compare records across languages. This section evaluates this setting directly: all supervised matchers are trained on the German 80 % corner-case large training set (26,571 pairs) and selected on the corresponding German Seen validation set (4,452 pairs). The multilingual encoder XLM-R is trained and selected in the same way. Each matcher is then evaluated, without further training, on five variants of the 80 % corner-case Half-Seen test set (4,427 pairs each) in which only the language of the records changes. In DE-DE both records are German, in DE-EN and EN-DE one record is German and the other English, in EN-EN both records are English, and Mixed combines the four combinations in equal shares (1,106 or 1,107 pairs each). Pair IDs, labels, and the seen/unseen composition are identical in all five variants. GPT-5.2 receives the German prompt in all five variants, and its zero-shot requests were repeated three times per prompt and variant. The supervised matchers report the mean over three seeds from training runs that are separate from the main evaluation but use the same training and validation splits, so the DE-DE scores can differ from the corresponding cells of the main table. For WordCooc and XLM-R, whose training is deterministic, the scores coincide. Language effects are measured relative to each model's own DE-DE baseline.

WordCooc and Magellan both lose performance once records leave the training language, but to a different degree. WordCooc loses most on the fully English variant (10.5 F1), since its co-occurrence features fire only when a token appears in both records and newly shared English tokens may lie outside the feature space fitted on German pairs. Magellan stays within 3.0 F1 points of its DE-DE score across all variants, as its similarity features depend less on the language. Both matchers operate between 25.7 and 36.2 F1 on this test set, so these differences carry limited practical weight. R-SupCon is the exception among the transformer-based matchers: it stays within 1.8 F1 of DE-DE on the two cross-language variants and loses most on the fully English variant (5.8 points). Its cross-language F1 remains below XLM-R and Ditto, so the smaller loss does not imply stronger absolute performance.

The four transformer-based matchers RoBERTa, XLM-R, HierGAT, and Ditto lose 4.8 to 7.7 F1 on the two cross-language variants DE-EN and EN-DE, and each of them loses less when both records switch to English, where the differences range from a gain of 0.3 to a loss of 5.3 F1. The Mixed variant, in which half of the pairs are same-language, loses less than the two cross-language variants. The models therefore do not fail on English data as such. As long as both records switch language together, the cues learned on German pairs, such as overlapping tokens between the two names and descriptions, stay intact. In a cross-language pair, the translation removes this surface overlap on one side only, and pairs that cross the language boundary never occur in the German training data. XLM-R loses 4.8 and 5.6 F1 on DE-EN and EN-DE but gains 0.3 F1 on EN-EN, so multilingual pre-training alone does not prevent the cross-language loss in this setting. For a price comparison portal this distinction matters, because the cross-border case is exactly the cross-language case.

GPT-5.2 delivers the best scores in every variant and reaches 79.6 F1 with the Rule-Guided prompt and 68.4 F1 with the Simple prompt on DE-DE. Its scores differ from DE-DE by at most 1.4 F1 in any variant. With both prompts, the spread across the five variants (0.77 F1 Rule-Guided, 1.85 F1 Simple) is smaller than the spread across three repetitions of identical DE-DE requests (1.10 and 1.98 F1). These comparisons describe the observed spread rather than establish statistical significance. Neither prompt shows a practically large penalty for cross-language pairs. For applications that have to match offers across languages, and where the budget allows it, a zero-shot LLM is therefore the most reliable choice among the evaluated systems.

The benchmark is distributed as gzipped JSON files with one record pair per line. Each line contains both product records, the binary match label, and a hard-negative indicator. The file name encodes the configuration: products80cc20rnd000un_train_large.json.gz denotes 80 % corner cases, 20 % random pairs, and the large training split. Training files always use 000un. The unseen-product share is defined by validation and test products relative to the training data. The files can be read directly with pandas:

import pandas as pd

pairs = pd.read_json(
    "products80cc20rnd000un_train_large.json.gz",
    compression="gzip",
    lines=True,
)

The complete reproduction code is available in the GitHub repository. From the repository root, install the fixed dependencies and generate all model-specific inputs with:

python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install -r environments/requirements.txt

python src/processing/prepare_pairs.py
python src/processing/prepare_pretraining.py
python src/processing/prepare_ditto_hiergat.py
python src/processing/prepare_wordcooc.py
python src/processing/prepare_magellan.py

The launchers in slurm_runs/ and the model directories run the complete German and English experiment matrix with three fixed seeds. All generated metrics are written below results/generated/ and can be collected into one CSV with:

python src/summarize_results.py

Exact commands for WordCooc, Magellan, RoBERTa, R-SupCon, Ditto, HierGAT, and GPT-5.2 are documented in REPRODUCTION.md. The GPT-5.2 runs require an OPENAI_API_KEY and use the Batch API.

For supervised matchers, pick a training set, select hyperparameters on the Seen validation set of the same corner-case ratio and size (000un_valid), and report F1 on the nine test sets as mean and standard deviation over three seeds. The test sets are the same for all training set sizes, so results obtained with different amounts of training data remain comparable.

For zero-shot and in-context learning, demonstration examples should be drawn from the training or validation sets.

For language comparisons, run the same configuration on the German and the English version. Splits, labels, and identifiers are identical, so any difference in the scores is attributable to the language of the records and the translation process. Since the record identifiers line up across the two versions, German records can also be paired with English ones to build cross-lingual matching tasks.

Both language versions are available as zip archives containing all training, validation, and test sets described above. The five aligned test set variants of the cross-language experiment are available as a separate archive. The archives are stored with Git LFS in the data folder of the GitHub repository, which also contains the unpacked German and English files. Per-seed results, standard deviations, precision, and recall for all matchers are available under reports/final_results.

The downloads are version 1.1 of the benchmark (git tag v1.1.0). The changes relative to version 1.0 are documented in the CHANGELOG; version 1.0 remains available as git tag v1.0.0.

Billiger.de Products extends product matching evaluation beyond English with a large German benchmark and an aligned English translation spanning thirteen product categories. The 27 variants of the benchmark vary the training set size, the share of corner cases, and the fraction of unseen products in the test set.

The validation of the benchmark showed that both the benchmark variant and the product category influence the results. Supervised matchers generally benefit from the larger training sets and score lower as the share of corner-cases or the amount of unseen entities in the test set increases. No supervised matcher exceeds 55 F1 in the 80 % corner-case Unseen variants. Clothing and furniture are the hardest product categories for every matcher and electronics is among the easiest, which shows the value of evaluating matchers beyond electronics.

The comparison of the performance of the matchers on billiger.de Products (English) to the performance of the same matchers on WDC Products showed that Billiger.de Products is more difficult than WDC Products: the PLM-based matchers score 10 to 20 F1 below their published results on WDC Products. This feature might make the English version of Billiger.de Products an interesting alternative to WDC Products for matchers that have saturated WDC Products.

The cross-language experiment showed that supervised matchers trained on German pairs keep most of their performance when both records switch to English, but RoBERTa, XLM-R, HierGAT, and Ditto lose 5 to 8 F1 points on cross-language German-English pairs, whereas R-SupCon stays close to its German score. Zero-shot GPT-5.2 reaches the best scores on all language combinations and changes by at most 1.4 F1, which makes LLMs the most reliable choice for cross-language matching among the evaluated systems.

We hope that the benchmark will support the evaluation and development of entity matching methods for monolingual German and English settings, as well as for cross-lingual matching scenarios.

Questions and feedback are welcome. Please contact Aaron Steiner or open an issue in the GitHub repository.

If you use the benchmark, please cite the paper describing it:

@misc{steiner2026billiger,
  author = {Steiner, Aaron and Elagin, Ksenia and Peeters, Ralph and Knopp, Johannes and Bizer, Christian},
  title  = {Billiger.de Products: A Bilingual Entity Matching Benchmark},
  year   = {2026},
  eprint = {2609.37713},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url    = {https://arxiv.org/abs/2609.37713}
}