Ontology Alignment Evaluation Initiative - OAEI-2009 CampaignOAEI

Results of the OAEI 2009 Library Thesaurus Mapping Task

This page presents the results of the OAEI 2009 Library Track. The evaluation was done according to two scenarios. From a methodological perspective, it is very similar to what was done for the evaluation of the previous 2008 Library Track, which featured a "thesaurus merging" evaluation (based on a post-hoc gold standard) and a "re-indexing" one (assessing the use of mappings for translating subject annotations from one thesaurus to another). The main difference is that in the TELplus case, the first evaluation method has been replaced by comparing to an already existing partial reference alignment, avoiding to assess the results of participants manually.

For each scenario, this page presents the raw results that were obtained. For further details, we refer the reader to this report and to the slides presented at the Ontology Matching workshop.

The results sent by the unique TaxoMap participant are available in the data section of the track. The statistics for those results are:

Type of relation LCSH-RAMEAU RAMEAU-SWD LCSH-SWD
exactMatch 5074 1265 38
broadMatch 116789 17220 0
narrowMatch 48817 6690 0
relatedMatch 13205 1317 0

Comparing with partial reference alignment (MACS)

As no participant used the training set we provided, we use the complete MACS mappings as gold standard. This reference alignment comprises 87183 LCSH-RAMEAU mappings, 13723 RAMEAU-SWD mappings, and 12203 LCSH-SWD mappings.

The table below shows the results when taking into account all mappings that belong to a certain relation selection. "-" for a given relation selection means that no extra relation was provided at that level, hence the results are identical to the ones of the previous selection level.

TaxoMap links evaluated LCSH-RAMEAU RAMEAU-SWD LCSH-SWD
Precision Coverage Precision Coverage Precision Coverage
exactMatch 72.1 5.7 27.1 1.4 44.4 0.03
exactMatch + broadMatch 3.6 6.9 2.3 1.9 - -
exactMatch + broadMatch + narrowMatch 2.8 7.3 1.8 2.0 - -
all relations 2.7 7.5 1.9 2.2 - -

The table below shows the results obtained when selecting only the "best" available mapping for one concept (that is, the one with the highest confidence measure), and discarding the others.

TaxoMap links evaluated LCSH-RAMEAU RAMEAU-SWD LCSH-SWD
Precision Coverage Precision Coverage Precision Coverage
exactMatch 78.7 5.7 39.5 1.4 44.4 0.03
exactMatch + broadMatch 22.0 6.0 13.5 1.6 - -
exactMatch + broadMatch + narrowMatch 14.4 5.9 10.8 1.6 - -
all relations 13.4 5.8 10.9 1.7 - -

Results for the re-indexing scenario

Those are the results for automated evaluation, using a gold standard of books indexed against both thesauri of the pair considered:

TaxoMap links evaluated LCSH-RAMEAU RAMEAU-SWD LCSH-SWD
Precision Recall Jaccard Precision Recall Jaccard Precision Recall Jaccard
exactMatch 22.3 6.1 5.5 14.2 3.1 2.4 1.3 0.003 0.002
exactMatch + broadMatch 2.1 7.8 1.5 2.3 3.6 1.1 - - -
exactMatch + broadMatch + narrowMatch 1.2 9.2 1.0 0.8 3.9 0.5 - - -
all relations 1.1 9.3 0.9 0.7 4.0 0.5 - - -

The table below shows the results obtained when selecting only the "best" available mapping for one concept (that is, the one with the highest confidence measure), and discarding the others.

TaxoMap links evaluated LCSH-RAMEAU RAMEAU-SWD LCSH-SWD
Precision Recall Jaccard Precision Recall Jaccard Precision Recall Jaccard
exactMatch 22.8 5.8 5.3 14.2 1.9 1.7 1.2 0.002 0.002
exactMatch + broadMatch 10.2 6.0 4.9 6.9 2.0 1.7 - - -
exactMatch + broadMatch + narrowMatch 7.2 4.5 3.3 5.9 1.9 1.5 - - -
all relations 6.4 4.0 2.9 5.8 1.9 1.5 - - -