Organizers

Sameer Pradhan, Martha Palmer, and Edward Loper

 

Subtask 1: Coarse-grained English Lexical Sample WSD

This task consists of lexical sample style training and testing data for 35 nouns and 65 verbs in the WSJ Penn Treebank II as well as the Brown corpus. This data will include, for each target item: OntoNotes sense tags (these are groupings of WordNet senses that are more coarse-grained than traditional WN entries, and which have achieved on average 90% ITA), as well as the sense inventory for these lemmas.

• Track 1: Closed track -- participants can use only the training data supplied, plus Charniak parses provided with the data, along with any other features that can be extracted from WordNet 2.1 as well as any unsupervised techniques. 
• Track 2: Open track -- participants can use any additional data, including the entire training portions (sections 02-21) of Penn Treebank and PropBank (if they are LDC members), or tools trained on these data, other potential knowledge sources for WSD as well as any unsupervised techniques.

This data has been made available to Eneko Agirre and Aitor Soroa for use in the Word Sense Induction task, and also to German Rigau and Montse Cuadros for their evaluation of lexical resources. As described above, the OntoNotes senses have links to WN senses.

Subtask 2: Coarse-grained English Lexical Sample SRL

For the same lemmas (but not necessarily exactly the same training and testing instances), we will also supply:

• PropBank annotation
• VerbNet class membership and VerbNet thematic role labels for the same targets.
• Charniak parses.

This will support a second subtask for SRL, in both PropBank style and VerbNet style. We propose that the SRL subtask have two evaluation tracks.

• Track 1: Closed track -- participants can use only the training data supplied, plus downloadable VerbNet as well as any unsupervised techniques.
• Track 2: Open track -- participants can use any additional data, including the entire training portions (sections 02-21) of Treebank and PropBank (if they are LDC members) or tools trained on those data.

Subtask 3: English fine-grained All-Words

We have supplied a 5000 word chunk of WSJ where all of the verbs and the head words of the verb arguments have WordNet 2.1 sense tags. This is for testing purposes only, and has no training annotation associated with it, or PropBank or VerbNet labels. Participants can of course use Semcor and the previous Senseval data as training data if they choose to. We have coordinated with Roberto Navigli and Ken Litkowski so that their Coarse-grained All-Words task annotates the same data. Since there is no training data for this task there is no Closed track.

• Track: Open track -- participants can use any additional data, including the entire Treebank and PropBank (if they are LDC members).

The data tagged with OntoNotes sense tags was prepared under DARPA GALE funding at the University of Colorado (verbs) and ISI (nouns). The VerbNet labels were attached to the PropBank data at the University of Colorado using a semi-automatic process that involved a hand correction step. This was funded by AQUAINT, and is described in a NAACL-07 paper, (Yi, Loper, Palmer).

The thematic role labels will be evaluated using Precision and Recall against the Gold Standard test data the same way PropBank was evaluated at CoNLL. The sense tags will be evaluated using Precision and Recall the same way Senseval English sense tags have been evaluated.

LDC agreed to let Semeval distribute the WSJ raw text for the data.

 

Deadline:

We have extended the deadline for submitting the results to the midnight of April 4^th.