Towards Improved Evaluation Measures for Parsing Systems

Schedule (Final Version)

9:15 – 9:30
Opening: Aims of the workshop

9:30 – 10:00
Relational evaluation schemes
Ted Briscoe, John Carroll, Jonny Graham, Ann Copestake
Universities of Cambridge & Sussex

10:00 – 10:30
Towards a dependency-oriented evaluation for partial parsing
Sandra Kübler, Heike Telljohann
University of Tübingen

10:30 – 11:00
LinGO Redwoods -- a rich and dynamic treebank for HPSG
Stephan Oepen, Ezra Callahan, Dan Flickinger, Christoper D. Manning
Stanford University

Coffee Break (30 mins)

11:30 – 12:30
Panel: Parser evaluation in context
John Carroll, Patrick Paroubek, Owen Rambow, Hans Uszkoreit

Lunch Break

14:00 – 14:30
A test of the leaf-ancestor metric for parse accuracy
Geoffrey Sampson, Anna Babarczy
University of Sussex

14:30 – 15:00
Evaluating parser accuracy using edit distance
Brian Roark
AT&T-Labs Research

Short Break (10 mins)

15:10 – 15:40
Evaluating syllabification: one category shared by many grammars
Karin Müller
Saarland University

15:40 – 16:10
Towards comparing parsers from different linguistic frameworks:
an information theoretic approach

Gabriele Musillo, Khalil Sima'an
University of Amsterdam

16:10 – 16:40
Evaluation of the Gramotron parser for German
Franz Beil, Detlef Prescher, Helmut Schmid, Sabine Schulte im Walde
TEMIS Paris, DFKI Saarbrücken & IMS Stuttgart

Coffee Break (30 mins)

17:10 – 17:40
Evaluating a wide-coverage CCG parser
Stephen Clark, Julia Hockenmaier
University of Edinburgh

17:40 – 18:10
A comparison of evaluation metrics for a broad coverage parser
Richard Crouch, Ronald M. Kaplan, Tracy H. King, Stefan Riezler
Palo Alto Research Center

18:10 – 20:00
Wrap up and kick-off: Initiatives and action plans (open end)