Ebook: Computerised Test Generation for Cross-National Military Recruitment

The advent of the 1980s was a portentous moment in educational and psychological assessment. Rapid advances had been made in a number of foundational fields: The cognitive revolution remade our understanding of how people learn, act, and solve problems. Microcomputers made digital technology and interactive capability available anytime, anywhere. Statistics offered more powerful ways of modeling complicated problems, again in real time. The two-fold challenge now was to figure out how this potential could come together in a principled way to advance assessment, and to put it to work to do astonishing things that had never been done before.

The British Army Recruitment Battery (BARB) is just such a thing.

In this book Sidney Irvine tells the story of BARB, his BARB. It traverses a widespread network, interleaved with connections to thousands of people contributing on BARB itself or working on seemingly unrelated projects years before, from mathematical equations to warrior vernacular, from a postwar army camp in the Mideast to the Britannia Royal Naval College to ivy-covered research labs in Princeton. The result after many years and many adventures is BARB: A computer-administered recruitment test battery which creates, on the fly, brand new parallel tests for every examinee, grounded in psychological science, created and scored with high reliability, and demonstrated valid for the purpose for which they were designed.

At several points in the discursion, Sid refers to our work, some of the many elements that contributed to BARB intellectually and at certain points directly, and which was furthered by what we learned from Sid and BARB. We unashamedly admit the collegial association, and offer some observations from our perspectives.

Bob's connection is the assessment framework he developed with Linda Steinberg and Russell Almond, called Evidence-Centered Design, or ECD. ECD aims to draw out, then make explicit in language and representations, the underlying connections in assessment arguments. Just what is it that one wants to make inferences about, as informed by whatever capabilities that are at issue and whatever purposes are in mind? What kinds of things do we need to see people say or do or make, in what kinds of situations, to get evidence to back our claims? What forms of task design, evaluation, scoring, and analysis embody the argument in the “pieces of machinery” that constitute an operational assessment? The resulting framework builds on foundational principles of assessment to help understand and design familiar kinds of assessments, but more importantly, to create new ones made possible by the foundational advances mentioned above–in particular, BARB. Confluences between ECD and BARB occurred in a joint study of mixed strategies for mental rotation tasks, a conference presentation and chapter on ECD in the Irvine and Kyllonen volume on item generation, and on-going informal conversations during Sid's visits to Educational Testing Service in Princeton.

A particularly intriguing – and for BARB, crucial – line of research was a theory of task generation. Pioneering work such as Jack Carroll's on the cognitive processes that underlie test performance helped bring out some of the connections, and that of Susan Embretson that pointed the way to how to integrate cognition, task design, and psychometric analysis. Task generation was a natural extension of this line of work. Isaac was funded by the Office of Naval Research in the 1980's to pursue what many (including the funding agency!) considered an idea with limited potential, namely item generation coupled to the theories of cognition that were emerging as part of the “cognitive revolution.” Isaac initially focused his work on spatial ability for which there was a growing cognitive foundation. It would be many years before item generation would become a widely accepted approach. Sid, however, unbeknown to us and probably most American researchers, was deeply engaged in the development of BARB at the same time and quickly recognized the ‘grammatical’ implications of the research taking place in the States for automatic item construction. It was no accident that Sid found out about this work as it was happening (before there was a Google), even though it would remain inaccessible in obscure technical reports for years before it appeared in journals. Sid had a long-standing connection with ETS, especially to Sam Messick, visiting ETS frequently, making it his business to find out about the on-going research.

In a Zen koan, Master Hyakujo brings out a drinking water jar, puts it down and says “You cannot call it a water jar. Then, what will you call it?” Isan kicks the jar over and walks away. In a psychometric koan, Dr. John Anderson asks “What would tests be like if there were no fixed item files, no IRT (item-response theory) – and no money?” Dr. Sidney Irvine swallows: then he recruits ideas, psychologists, software engineers, armed services funders, a network of researchers and personnel experts; works for 15 years of fits and starts, through successes and “surprises”; creates, tests, and validates BARB; produces its successors in the United States and Europe, and finally writes a book about the experience. Not so pithy, but a damned good answer, in the same spirit!

Isaac I. Bejar & Robert J. Mislevy

Educational Testing Service August 27, 2013