The effect of back-formulating questions in question answering evaluation

摘要

It is pleasing to report that the 27th Conference on Research and Development in Information Retrieval has again attracted a large number of papers and posters from the global IR community. There were 267 paper submissions with 58 (22%) being accepted. Likewise, there were over 130 poster submissions -- 67% more than in 2003 -- with 65 (nearly 50%) being accepted. In addition to the papers and posters, the conference program contains 8 tutorials, 11 workshops as well as 7 demonstrations. The contributions in these proceedings represent a diverse and comprehensive coverage of IR research including theoretical models, techniques related to specific tasks, Web IR, cross-language IR, and retrieval in multimedia environments as well as evaluative studies.The selection of quality contributions for the SIGIR conference is dependent on a robust and equitable two tier reviewing process. The PC consisted of three Chairs representing three geographic regions, 27 Area Coordinators, 2 for each of the 12 areas, with the exception being machine learning and text classification, which was served by four area coordinators. This reflects the growing number of submissions to this area over the last few years. Finally, there was a mobile fire-brigade of three area coordinators "at large". The area coordinators nominated over 150 first tier reviewers. Each reviewer was manually assigned between 3 and 8 papers by the PC chairs in accordance to reviewers' stated subject expertise and each paper was allocated the minimum of three reviewers. The role of the Area Coordinators was to oversee the process for each of their topic area, by resolving disagreements between reviewers and producing an area coordinator review for each paper synthesizing the first tier reviews. The area coordinator review then served as the basis of discussion at the Programme Committee Meeting. Area Coordinators were selected for their subject expertise in the different topic areas and attention was also paid to balancing geographicrepresentation. Posters and demos were assessed by three reviewers each, and tutorials and workshops by 7 and 8 reviewers, respectively. All the reviewing was double blind with the identification of authors / proposers being released only after the selection of papers, posters, demos and events was completed. Finally, we are unhappy to mention the growing occurrence of papers being submitted in parallel despite clear guidelines to the contrary. The SIGIR executive committee is currently considering this serious issue.We are indebted to all our colleagues who agreed to act as reviewers and area coordinators and for their cooperation in meeting the very tight deadlines. We are also grateful to all the authors who submitted papers and in spite of the keen competition continue to do so. Our thanks to Doug Oard and his team who volunteered to act as mentors for authors who needed help and advise on submissions. It is worth noting that out of the 58 papers accepted, more than half (30) are student papers, an encouraging sign for this growing field of research.In addition to papers the PC Committee also confirmed the selection of posters, demos, tutorials and workshops. We are grateful to Gareth Jones, Ian Ruthven, Fabio Crestani, and Mounia Lalmas who were the respective chairs for each of these and to John Tait who chaired the best paper awards committee. This year SIGIR also has a novelty in the program, the Doctoral Consortium, where the team chaired by David Harper has set a baseline for the future. We also wish to thank Mounia Lalmas for hosting the PC Meeting at the University of London and Mark Sanderson the conference chair who mindfully shepherded us. Finally special thanks to our keynote speakers Gordon Bell and Peter Willett for agreeing to participate in this 27^th anniversary conference of SIGIR.