Individual organizations and countries have been investing in the creation of resources and methods for the evaluation of resources, technologies, products and applications. This is evident in the US DARPA HLT programme, the EU HLT programme under FP5-IST, the German MTI Program, the Francophone AUF programme and others. The European 6th Framework program (FP6), planned for a start in 2003, includes multilingual and multisensorial communication as one of the major R&D issues.
Substantial mutual benefits can be expected from addressing these issues through international cooperation. A key element to international cooperation is sharing knowledge about problems and solutions. Another are joint activities that go beyond information sharing to co-creation of solutions to global challenges. Although the evaluation paradigm has been a key element in large national and international programs, there has been limited explicit global planning or coordination. A key first step toward this type of international cooperation is a shared plan for moving forward.
The primary purpose of this workshop will be to refine a roadmap (www-nlpir.nist.gov/projects/duc/papers/ qa.Roadmap-paper_v2.doc) for question answering applications and the methods for the creation and evaluation of resources for the next decade in support of these systems. The workshop will draw upon research in the TREC Q&A track, the AQUAINT program, and efforts planned for the ARDA Northeast Regional Research Center (NRRC). Participants will help formulate grand challenge problems, discuss possible data sets and/or evaluation metrics/methods, articulate the role of and necessary advances in resources and evaluation to solve these challenges, as well as strategize jointly about the most effective and efficient path forward.
The workshop will be a mix of short presentations and facilitated sessions with the intent of joint articulation of grand challenge problems and identification of a strategy for facilitating the creation of question answering resources and evaluation strategies. The agenda will consist of two working sessions, the first focused on the creation of a question answering roadmap and the second addressing resources required to achieve milestones on the roadmap.
The workshop will run from 8:00 to 13:30 for the morning session and from 14:30 to 20:00 for the afternoon session.
Topics to be addressed in the workshop include, but are not limited to:
Possible joint products to be created in preparation for and at the workshop could include:
This workshop will consist primarily of working sessions, however, presentation of short positions and participation in the workshop will be based on an assessment of a 4 page extended position statement which addressees one or more of the fundamental Q&A road map and/or Q&A resource/evaluation issues posed by the workshop. Submissions must be in English, no more than 4 pages long, and in single column format. The first page should include the title, names and affiliations of the authors; the full address of the first author (or a contact person), including phone, fax, email, URL; and 5 keywords.
Submissions should be sent electronically in Word (preferably) or PDF or ASCII text format to arrive no later than 15 January 2002 to Paula MacDonald.
Demonstrations of Q&A LR and related tools will be considered as well. Please send a demonstration outline of 2 pages. As soon as possible, authors are encouraged to send a brief email indicating their intention to participate to include their contact information and the topic they intend to address in their submission. Proceedings of the workshop will be printed.
|Submission Deadline||15th February 2002|
|Notification, Stylesheets Available||1st March 2002|
|Camera Ready Papers Due||15th March 2002|
|Workshop Program Due||3rd April 2002|
|Proceedings Due||20th April 2002|
|Workshop||28th May 2002|
|Mark Maybury||MITRE (chair)||firstname.lastname@example.org|
|Karen Sparck Jones||University of Cambridgeemail@example.com|
|Sanda Harabagiu||University of Texas, Austinfirstname.lastname@example.org|
|Liz Liddy||University of Syracuseemail@example.com|
These fees will include a coffee break and the Proceedings of the Workshop. The Proceedings of the Workshop will be published by the Local Organising Committee.
eak and the Proceedings of the Workshop. The Proceedings of the Workshop will be published by the Local Organising Committee.