Motivation and Aims

Individual organizations and countries have been investing in the creation of resources and methods for the evaluation of resources, technologies, products and applications. This is evident in the US DARPA HLT programme, the EU HLT programme under FP5-IST, the German MTI Program, the Francophone AUF programme and others. The European 6th Framework program (FP6), planned for a start in 2003, includes multilingual and multisensorial communication as one of the major R&D issues.

Substantial mutual benefits can be expected from addressing these issues through international cooperation. A key element to international cooperation is sharing knowledge about problems and solutions. Another are joint activities that go beyond information sharing to co-creation of solutions to global challenges. Although the evaluation paradigm has been a key element in large national and international programs, there has been limited explicit global planning or coordination. A key first step toward this type of international cooperation is a shared plan for moving forward.

The primary purpose of this workshop will be to refine a roadmap (www-nlpir.nist.gov/projects/duc/papers/ qa.Roadmap-paper_v2.doc) for question answering applications and the methods for the creation and evaluation of resources for the next decade in support of these systems. The workshop will draw upon research in the TREC Q&A track, the AQUAINT program, and efforts planned for the ARDA Northeast Regional Research Center (NRRC). Participants will help formulate grand challenge problems, discuss possible data sets and/or evaluation metrics/methods, articulate the role of and necessary advances in resources and evaluation to solve these challenges, as well as strategize jointly about the most effective and efficient path forward.

Workshop Agenda

The workshop will be a mix of short presentations and facilitated sessions with the intent of joint articulation of grand challenge problems and identification of a strategy for facilitating the creation of question answering resources and evaluation strategies. The agenda will consist of two working sessions, the first focused on the creation of a question answering roadmap and the second addressing resources required to achieve milestones on the roadmap.

The workshop will run from 8:00 to 13:30 for the morning session and from 14:30 to 20:00 for the afternoon session.

Morning Session - Question Answering Resources and Evaluation

• Short statements by select participants addressing key question answering resources, processing techniques, evaluation methods, and research challenges
• Facilitated group session to identify key question answering resources, evaluation methods, and desired outcomes/ capabilities over the next ten years
• Facilitated breakout groups to expand and refine findings regarding resources, evaluation, and Q&A capabilities

Afternoon Session - Question Answering Roadmap

• - Regroup with report outs from breakout groups
• - Further group and (possible) breakout discussions regarding annotation/coding schemes, tools and methods for creation and annotation of question anwering resources and evaluation methods
• Creation and refinement of roadmap of expected outcomes over next five to ten years, including possible course of actions for individuals, organizations, and governments

Topics to be addressed in the workshop include, but are not limited to:

• Roadmaps for question answering language resources (LR) and scientific algorithm developments
• Existing question answering language resources
• Guidelines, standards, specifications, models and best practices for question answering LR
• Methods, tools, and procedures for the acquisition, creation, management, access, distribution, and use of question answering LR
• LR and evaluation and benchmarking of question answering systems and algorithms for tasks including:
  • Advanced question analysis
  • Answer discovery and integration
  • Answer explanation and presentation generation
  • Interactive question answering
• LR and evaluation methods for advanced question answering challenges, including but not limited to:
  • Question answering from heterogeneous (structure, unstructure, semi-structured) sources
  • Multimedia (e.g., text, graphics, audio, video) and Multimodal (i.e., audiotory, visual) question answering
  • Multilingual question answering
  • Answering questions from multiple perpsectives
• Question answering components, architectures or instrumentation that faciliates evaluation

Possible joint products to be created in preparation for and at the workshop could include:

• List of existing resources and ones under development (with planned release dates)
• Updates to ARDA Q&A Roadmap (www-nlpir.nist.gov/projects/duc/papers/qa.Roadmap-paper_v2.doc)
• List of Evaluation methods and benchmarks of question answering systems
• List of unresolved research problems and/or areas in question answering
• Shared knowledge of research groups and efforts

Submissions

This workshop will consist primarily of working sessions, however, presentation of short positions and participation in the workshop will be based on an assessment of a 4 page extended position statement which addressees one or more of the fundamental Q&A road map and/or Q&A resource/evaluation issues posed by the workshop. Submissions must be in English, no more than 4 pages long, and in single column format. The first page should include the title, names and affiliations of the authors; the full address of the first author (or a contact person), including phone, fax, email, URL; and 5 keywords.

Submissions should be sent electronically in Word (preferably) or PDF or ASCII text format to arrive no later than 15 January 2002 to Paula MacDonald.
Demonstrations of Q&A LR and related tools will be considered as well. Please send a demonstration outline of 2 pages. As soon as possible, authors are encouraged to send a brief email indicating their intention to participate to include their contact information and the topic they intend to address in their submission. Proceedings of the workshop will be printed.

Important Dates

Organizing Committee

Workshop Registration Fees

• If you are not attending LREC: 140 EURO
• If you are attending LREC: 90 Euro

These fees will include a coffee break and the Proceedings of the Workshop. The Proceedings of the Workshop will be published by the Local Organising Committee. eak and the Proceedings of the Workshop. The Proceedings of the Workshop will be published by the Local Organising Committee.