This workshop will provide a forum for researchers interested in the development and evaluation of natural language grammars and parsing systems, and in the creation of syntactically annotated reference corpora.

Organisers: John Carroll, Roberto Basili, Nicoletta Calzolari,
Robert Gaizauskas, Gregory Grefenstette

ACCEPTED PAPERS

A survey of parser evaluation methods
  John Carroll, Ted Briscoe
  University of Sussex & University of Cambridge, UK

Evaluating a Robust Parser for Italian Language
  Roberto Basili, Maria Teresa Pazienza, Fabio Massimo Zanzotto
  Universita' di Roma Tor Vergata, Rome, Italy

Evaluation of the syntactic analysis component of an 
  information extraction system for German
  Thierry Declerck, Judith Klein, Guenter Neumann
  DFKI, Saarbruecken, Germany

Chunking Italian. Linguistic and task-oriented evaluation
  Stefano Federici, Simonetta Montemagni, Vito Pirrelli
  ILC-CNR Pisa, Italy

Modifying existing annotated corpora for general comparative
  evaluation of parsing
  Rob Gaizauskas, Mark Hepple, Chris Huyck 
  University of Sheffield, UK

Dependency-based evaluation of MINIPAR
  Dekang Lin
  University of Manitoba, Canada

Evaluating parses for spoken language dialogue systems
  Wolfgang Minker, Lin Chase
  LIMSI, France

Corpus-based parse pruning
  Sonja Mueller-Landmann
  IBM, Heidelberg, Germany

The TOSCA parsing system reviewed
  Nelleke Oostdijk
  Katholieke Universiteit Nijmegen, The Netherlands

Grammar & parser evaluation in the XTAG project
  Srinivas Bangalore, Anoop Sarkar, Christine Doran, 
  Beth Ann Hockey
  AT&T Labs-Research & IRCS, University of Pennsylvania, USA

The aim of this workshop is to provide a forum for discussion of evaluation methods for parsing systems, and proposals for the development of syntactically annotated language resources.

With increased attention to evaluation of component technology in language engineering, evaluation of parsing systems is rapidly becoming a key issue. Numerous methods have been proposed and while one, the Parseval/Penn Treebank scheme, has gained wide usage, this has to some extent been due to the absence of workable alternatives rather than to whole-hearted support. Parseval/PTB evaluation has several limitations and drawbacks, including a commitment to a particular style of grammatical analysis, and oversensitivity to certain innocuous types of misanalysis while failing to penalise other common types of more serious mistake. Also, the original published description of the scheme -- and the evaluation software widely distributed as a follow-up to it -- is specific to the English language. It may be that there are currently no alternative more workable schemes or proposals, but this needs to be more fully discussed: this workshop will provide an opportunity for such a debate.

This workshop is particularly timely given the large number of CEC Language Engineering projects that involve parsing in one form or another and which need to evaluate and share the results of their efforts. Parsing is an essential part of many larger applications, such as Information Extraction, which have gained in importance over the last few years. Often in such systems, the strength of the parser and grammar has a direct effect on the desired results, and thus achieving good results rests on being able to determine and improve weaknesses in the parser/grammar. Without a reliable parser evaluation method this cannot be done effectively.

A parsing evaluation workshop is also appropriate at this time given the imminent creation of large-scale syntactically annotated resources for European languages. Contributions from those involved in such activities are welcomed, so as to improve communication between the resource construction and the resource utilisation communities. This should ensure that the resources constructed are maximally useful to the general language engineering community.

The workshop is jointly organised by the CEC Language Engineering 1 projects SPARKLE and ECRAN

PROGRAMME COMMITTEE

Roberto Basili
Ted Briscoe
Nicoletta Calzolari
John Carroll
Roberta Catizone
Robert Gaizauskas
Gregory Grefenstette
Mark Hepple
Tony McEnery
Maria Teresa Pazienza
Paola Velardi
Yorick Wilks

IMPORTANT DATES

Paper submission deadline (hard copy/electronic): February 15th
Notification of acceptance: March 10th
Camera-ready papers due: April 10th
Workshop: May 26th

CONFERENCE INFORMATION

General information about the conference is at:
http://www.icp.inpg.fr/ELRA/conflre.html

Specific queries about the conference should be directed to:

LREC Secretariat
Facultad de Traduccion e Interpretacion
Dpto. de Traduccion e Interpretacion
C/ Puentezuelas, 55
18002 Granada, SPAIN
Tel: +34 58 24 41 00 - Fax: +34 58 24 41 04
reli98@goliat.ugr.es

------

The Evaluation of Parsing Systems, May 26th - final programme

14:40-15:00
A survey of parser evaluation methods
  John Carroll, Ted Briscoe
  University of Sussex & University of Cambridge, UK

15:00-15:20
Evaluating parses for spoken language dialogue systems
  Wolfgang Minker, Lin Chase
  LIMSI, France

15:20-15:40
Evaluation of the syntactic analysis component of an information extraction
system for German
  Judith Klein, Thierry Declerck, Guenter Neumann
  DFKI, Saarbruecken, Germany

15:40-16:00
Chunking Italian: linguistic and task-oriented evaluation
  Stefano Federici, Simonetta Montemagni, Vito Pirrelli
  ILC-CNR Pisa, Italy

16:00-16:20
Evaluating a robust parser for Italian
  Roberto Basili, Maria Teresa Pazienza, Fabio Massimo Zanzotto
  Universita' di Roma Tor Vergata, Italy

16:20-17:00
Coffee Break

17:00-17:20
The TOSCA parsing system reviewed
  Nelleke Oostdijk
  University of Nijmegen, The Netherlands

17:20-17:40
A corpus for parse pruning
  Sonja Mueller-Landmann
  IBM, Heidelberg, Germany

17:40-18:00
Dependency-based evaluation of MINIPAR
  Dekang Lin
  University of Manitoba, Canada

18:00-18:20
Break

18:20-18:40
Grammar & parser evaluation in the XTAG project
  Srinivas Bangalore, Anoop Sarkar, Christine Doran, Beth Ann Hockey
  AT&T Labs-Research & IRCS, University of Pennsylvania, USA

18:40-19:00
Modifying existing annotated corpora for general comparative evaluation of
parsing
  Rob Gaizauskas, Mark Hepple, Chris Huyck 
  University of Sheffield, UK

19:00-19:30
Plenary session