Ontology Alignment Evaluation Initiative - Submitting a data set

Submitting a data set to OAEI

The aim of this (short) document is to explain how to propose a dataset and evaluation modalities to OAEI. Since 2010, these rules have been made more precise.

Hereby, organiser denotes a group of person willing to propose a data set to evaluate or an experiment in a particular OAEI campain. OAEI denotes the people coordinating the OAEI campaigns including its steering committee. In practice, OAEI is the set of all organisers of a particular campaign (so, may be you).

Data set submission

Potential organisers must send an email providing the folowing information to OAEI:

• A short presentation of the dataset and evaluation in very few lines like there are in the call for participation to campaigns;
• Name and email of the organisers;
• Explanation of what the evaluation is supposed to show (e.g., test a particular feature of matchers);
• Description of the data sets (where do they come from, if they are in public access -preferred- or not, their size, languages and features);
• Description of the evaluation modalities and which conclusion could be derived from the eventual results;
• Potential list of participating system with indication of their asserted willingness to participate if they have been contacted (this can also be "all usual participants of OAEI").

In principle, this should be sent at the beginning of the year (between January and March, the earlier the best).

Your duties as an organiser

If your evaluation proposal is accepted, you will have to:

• Set up a web page/site exposing the data set and explaining the rules to participate. This web page should follow the template provided by the organisers.
• Answer to the participants and update the web page with these answers
• Evaluate the participant results in due time and providing them with feedback.
• Providing the final results in due times as (1) a web page/site, (2) a section in the OAEI summary paper.
• Both web pages/sites will be relocated within the OAEI web site at the end of the campaign. They should be maintained until then. Alternatively, it is possible to ask OAEI to maintain the data track web page/site but this would be at the expense of responsiveness and at the discretion of the web site maintainer.

What OAEI will do for you

In fact, not much. OAEI is rather targeted as tool developers and you will be also working for these tool developers. However, OAEI will:

• Maintain this web site;
• Link your web page to its campaign web site;
• Advertise the campaigns (but you can do it as well);
• Provide advise and tools from former organisers;
• Organise the final meeting (usually collocated with some other event);
• Invite you to write and edit the final reports usually published on the web;
• Bug you in order to get everybody satisfied.

Additional rules

• Any track which has less than two participants independent from the organisers will be declared fruitless.
• It is not forbidden that a system participates in a track that its promoters have organised. However, fairness would require that such systems participate at least in two other tracks.