Last Friday I went to an interesting seminar by William Webber (blog), the basic premise of which was that IR researchers should consider constructing their own test collections, and outlined how to go about that. Here's the abstract.

I thought one slide pointing out how reusing test collections can lead to an unhealthily narrow focus was especially pithy, and I reproduce it here with William's permission.

Methodology section before TREC:

We identify as experimental variables: user characteristics; problem statement; question statement; question characteristics; search strategy; search characteristics …

Methodology section after TREC:

We take the TREC 8 AdHoc track collection. Our evaluation metrics are P@10 and MAP.