OAEI 2006 food
Thesaurus Mapping Task
Abstract
To fulfill the OAEI 2006 food thesaurus mapping task, participants are required to align two SKOS thesauri using relations from the SKOS Mapping vocabulary. Each participant is asked to evaluate part of the results of the other participants. These evaluations are collected and validated by domain experts.
Current Status
The results of the participants are in. The following teams have entered their final results: South East University (Falcon-AO), University of Pittsburgh (Prior), Tsinghua University (KEG), University of Leipzig (COMA++), and Università degli Studi di Milano (HMatch).
The results of the participants can be found at the following web address: http://www.few.vu.nl/~wrvhage/oaei2006/results
The Gold Standard used to evaluate Precision and Recall can be found at the following web address: http://www.few.vu.nl/~wrvhage/oaei2006/gold_standard
Task
Create an alignment between the SKOS version of the United Nations Food and Agriculture Organization (FAO) AGROVOC thesaurus (±16.000 terms, multilingual: en, fr, es, ar, zh, pt, cs, ja, th, sk) and the United States National Agricultural Library (NAL) Agricultural thesaurus (±41.000 terms, monolingual: en), preferably using relations from the SKOS Mapping Vocabulary.
A specification of the SKOS vocabularies can be found at the SKOS website. (http://www.w3.org/2004/02/skos/)
A description of these relations can be found in the SKOS Mapping Vocabulary. (http://www.w3.org/2004/02/skos/mapping/)
Participants are advised to use the same API as used in the previous OAEI to produce the common format for alignments, but using the following mapping relations:
http://www.w3.org/2004/02/skos/mapping#narrowMatch
http://www.w3.org/2004/02/skos/mapping#exactMatch
http://www.w3.org/2004/02/skos/mapping#broadMatch
http://www.w3.org/2004/02/skos/mapping#minorMatch
http://www.w3.org/2004/02/skos/mapping#majorMatch
http://www.w3.org/2004/02/skos/mapping#AND
http://www.w3.org/2004/02/skos/mapping#OR
http://www.w3.org/2004/02/skos/mapping#NOT
An example broaderMatch mapping between AGROVOC “hard cheese” and NALT “cheeses” in the common format for alignments, produced by the API looks like this:
<rdf:RDF xmlns="http://knowledgeweb.semanticweb.org/heterogeneity/alignment"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:xsd="http://www.w3.org/2001/XMLSchema#">
<Alignment>
<xml>yes</xml>
<level>0</level>
<type>**</type>
<onto1>http://www.fao.org/aos/agrovoc/skos-ext</onto1>
<onto2>http://agclass.nal.usda.gov/nalt/2006.xml</onto2>
<map>
<Cell>
<entity1 rdf:resource="http://www.fao.org/aos/agrovoc/skos-ext#16492" />
<entity2 rdf:resource="http://agclass.nal.usda.gov/nalt/2006.xml#cheeses" />
<measure rdf:datatype="&xsd;float">1.0</measure>
<relation>http://www.w3.org/2004/02/skos/mapping#broadMatch</relation>
</Cell>
</map>
</Alignment>
</rdf:RDF>
The file containing the alignments should be submitted by e-mail to .
Evaluation procedure
- Each participant submits his preliminary mappings, in the common format for alignments, before September 4th.
- Each participant submits his final mappings before September 15th.
- The mappings will be distributed at random and anonymously over a group of domain experts. (food safety researchers and food product development researchers)
- A (small) sample of the mappings are distributed at random and anonymously over the other participants.
- The domain experts and participants are required to assess the mappings appointed to him before October 5th.
- The results are published before October 9th.
- Evaluation measurements of the participants' systems calculated based on this list of reference alignments.
A web tool for the assessment of the results has been set up at the Free University Amsterdam. Participants will be contacted personally about their assessment loads.
An example assessment page can be found at the following web address:
http://prauw.cs.vu.nl/~wrvhage/oaei2006
Results
Precision
| RiMOM | Falcon-AO | Prior | COMA++ | HMatch | ||
|---|---|---|---|---|---|---|
| biological & chemical mappings | 0.85 | 0.8 | 0.81 | 0.76 | 0.83 | |
| taxonomical mappings | 0.82 | 0.83 | 0.68 | 0.43 | 0.48 | |
| miscellaneous mappings (geography, legislation, food stuffs, etc.) | 0.78 | 0.83 | 0.74 | 0.7 | 0.8 | |
| all-round | 0.81 | 0.83 | 0.71 | 0.54 | 0.61 |
The Falcon-AO system of South East University has a significantly better Precision than all other systems at mapping taxonomical and miscellaneous concepts (such as geographical concepts and food stuffs). The RiMOM system of Tsinghua University (KEG) has a significantly better Precision than all other systems at mapping biological and chemical concepts.
Recall
| RiMOM | Falcon-AO | Prior | COMA++ | HMatch | |
|---|---|---|---|---|---|
| considering only exactMatch mappings | 0.71 | 0.65 | 0.64 | 0.33 | 0.65 |
| considering all types of mappings | 0.50 | 0.46 | 0.45 | 0.23 | 0.46 |
OAEI 2006 food results presentation
Thesauri
The latest SKOS version of the thesauri can be downloaded from the directories listed below. (updated may 30th 2006)
AGROVOC
Download AGROVOC.
Read more about AGROVOC at http://www.fao.org/agrovoc.
NAL thesaurus
Download the NAL thesaurus.
Read more about the NAL thesaurus at http://agclass.nal.usda.gov/agt.
Organization
Send any questions, comments, or suggestions to:
Willem Robert van Hage (
)
