The Fourth International Workshop on Semantic Evaluations (SemEval-2007) took place in June 23-24 2007, as a co-located event with the 45th Annual Meeting of the ACL. It was the fourth international evaluation exercise following the series of successful Senseval workshops.
The purpose of Senseval was to evaluate the strengths and weaknesses of word sense disambiguation systems with respect to different words, different varieties of language, and different languages. Senseval was run by a small committee under the auspices of ACL-SIGLEX (the Special Interest Group on the LEXicon of the Association for Computational Linguistics).
Senseval-1 took place in the summer of 1998 for English, French, and Italian, culminating in a workshop held at Herstmonceux Castle, Sussex, England on September 2-4. Senseval-2 took place in the summer of 2001, and was followed by a workshop held in July 2001 in Toulouse, in conjunction with ACL 2001. Senseval-2 included tasks for Basque, Chinese, Czech, Danish, Dutch, English, Estonian, Italian, Japanese, Korean, Spanish, Swedish. Senseval-3 took place in March-April 2004, followed by a workshop held in July 2004 in Barcelona, in conjunction with ACL 2004. Senseval-3 included 14 different tasks for core word sense disambiguation, as well as identification of semantic roles, multilingual annotations, logic forms, subcategorization acquisition.
For the fourth edition, Senseval changed its name to SemEval (for Semantic Evaluation). The change of name was motivated by a desire to broaden the spectrum of accepted works to all aspects of computational semantic analysis of language. SemEval-2007 was very successful. The call for tasks solicited 27 task proposals. After a careful review process and a call for interest in participation, 18 tasks were selected to be part of the evaluation. Over 100 teams participated with over 125 unique systems. As a comparison, Senseval-3 (2004) organized 14 tasks with 55 teams. Some tasks were updated versions of tasks found in Senseval-3, including lexical-sample word sense disambiguation tasks in Catalan, English, Spanish and Turkish, two all-words English word sense disambiguation tasks, and two multilingual lexical sample tasks (Chinese-English). The updates included using coarse-sense inventories, or combining word sense disambiguation and semantic role classification. The rest of the tasks were novel to this evaluation exercises, and some have been organized for the first time.
In recent years, the deployment of multiple semantic lexicons and accordingly tagged corpora (WordNet-SemCor, VerbNet-PropBank, FrameNet and Prague Dependency Treebank, to name a few) have radically changed the way semantic analysis is performed, especially at the disambiguation stage. Some of the tasks at SemEval-2007 tried to take this one step further by proposing multiple layers of semantic annotation of the same corpus (e.g. senses, roles, name-entity classes), allowing for novel research to be performed.
We invite submissions of papers describing in-depth evaluation exercises on computational semantics of text. A natural candidate is an extended "task description" paper from SemEval-2007, which includes a broad description of the problem, resources used, system comparative, analysis of results, and some novel conclusions. But we do not limit the contributions to SemEval-related works. Instead, other works presenting substantial experimental evaluation on natural language semantics are also welcome. Finally, papers describing concrete NLP systems/approaches on one of those semantic tasks are welcome too, provided they present relevant evaluation on the task at hand and highlight important properties of the problem, resources, etc. System description papers which don't add insights on the specific problem they tackled are out of the scope of this issue.
The list of topics of interest includes but is not limited to:
| Call for papers | 1 October 2007 | |
| Submissions of papers | 29 February 2008 | |
| Preliminary decisions to authors | June 2008 (preliminary) | |
| Submission of revised articles | July 2008 (preliminary) | |
| Final decisions to authors | September 2008 (preliminary) | |
| Final versions due from authors | October 2008 (preliminary) |