Chapter Appendix B
Developing a Test Item Bank
Developing a bank of test questions is difficult and time consuming. This task demands a mastery of the subject, an ability to write clearly, and an ability to visualize realistic situations for use in developing problems. Because it is so difficult to develop good test items, many instructors rely on commercially-prepared test materials. However, there are a few instructors who write their own questions and maintain them in a database or a system of folders. [Figure B-1]
Written Test Items
Supply Type
Supply type test items ask the learner to furnish a response in the form of a word, sentence, or paragraph. The supply-type item allows the learner to organize knowledge. It shows an ability to express ideas and measures the learner’s generalized understanding of a subject. For example, the supply type item on a pre-solo knowledge test can be very helpful in determining whether the pilot-in-training has adequate knowledge of procedures.
There are several disadvantages of supply-type items. First, they cannot be graded with reliability. The same test graded by different instructors could be assigned different scores. Even the same test graded by the same instructor on consecutive days might be assigned a different score. Second, supply-type items require more time for the learner to complete and more time for the instructor to grade.
Selection Type
Selection type test items require the learner to select from two or more alternatives. There is a single correct response for each item. It assumes all learners should learn the same thing, and relies on rote memorization of facts. Written tests made up of selection type items are highly reliable, meaning that the results would be graded the same regardless of the learner taking the test or the person grading it. In fact, this type of test item lends itself very well to machine scoring.
Also, selection type items make it possible to directly compare learner accomplishment. For example, it is possible to compare the performance of learners within one class to learners in a different class, or learners under one instructor with those under another instructor. By using selection type items, the instructor can test on many more areas of knowledge in a given time than could be done by requiring the learner to supply written responses. This increase in comprehensiveness can be expected to increase validity and discrimination. Selection type tests are well adapted to statistical item analysis.
True-False
The true-false test item consists of a statement followed by an opportunity for the learner to choose whether the statement is true or false. This item type has a wide range of usage. It is well adapted for testing knowledge of facts and details, especially when there are only two possible answers.
The chief disadvantage is that true-false questions create the greatest probability of guessing. Also, true-false questions are more likely to utilize rote memory than knowledge of the subject. In general, therefore, true-false questions are not considered valid (i.e., they do not measure what they are intended to measure).
To use true-false questions, consider the following guidelines for effective test items:
- Include only one idea in each statement.
- Use original statements rather than verbatim text.
- Make the statement entirely true or entirely false.
- Avoid the unnecessary use of negatives, which tend to confuse the reader.
- Underline or otherwise emphasize the negative word(s) if they must be used.
- Keep wording and sentence structure as simple as possible.
- Make statements both definite and clear.
- Avoid the use of ambiguous words and terms (some, any, generally, most times, etc.)
- Use terms which mean the same thing to all learners whenever possible.
- Avoid absolutes (all, every, only, no, never, etc.). These words are known as determiners because they provide clues to the correct answer.
- Avoid patterns in the sequence of correct responses because learners can often identify the patterns.
- Make statements brief and approximately same length.
- State the source of a statement if it is controversial (sources have differing information).
Multiple Choice
A multiple choice test item consists of two parts: the stem, which includes the question, statement, or problem; and a list of possible responses. Incorrect answers are called distractors. When properly devised and constructed, multiple choice items offer several advantages that make this type more widely used and versatile than either the matching or the true-false items. [Figure B-2]
Multiple choice test questions can help determine learner achievement, ranging from acquisition of facts to understanding, reasoning, and ability to apply what has been learned. It is appropriate to use multiple choice when the question, statement, or problem has the following characteristics:
- Built-in and unique solution, such as a specific application of laws or principles.
- Wording of the item is clearly limiting, so that the learner must choose the best of several offered solutions rather than a universal solution.
- Several options that are plausible or even scientifically accurate.
- Several pertinent solutions, with the learner asked to identify the most appropriate solution.
Three major challenges are common in the construction of multiple choice test items. One is the development of a question or an item stem that must be expressed clearly and without ambiguity. A second is that the statement of an answer or correct response cannot be refuted. Finally, the distractors must be written in such a way that they are attractive to those learners who do not possess the knowledge or understanding necessary to recognize the keyed response.