Getting to Grips With the Data
Can there be too much automation? Well, yes — but even in some apparently tedious tasks that we really don’t want to do by hand, automation can hide important details.
Some colleagues of mine are engaged in education research in Physics, and as part of that (and also as part of being good educators in general) they get their students to tackle the Force Concept Inventory (FCI), which gives an idea of their grasp of basic mechanics.
Recently, when going through responses to tests on paper rather than on-line, and entering the results into Excel manually, they noticed that one question (say, ‘A‘) was frequently answered correctly, only for the correct answer to be scrubbed out and a particular wrong one chosen.
Most on-line quizzes would not capture this, and many tests, including the FCI itself, are often delivered on-line these days.
It turned out that the question after this one (say, ‘B‘) was on a similar topic and was causing students to rethink their previous answers and change them from correct to wrong. This caused a study of interaction between questions and a useful publication: ‘”On Second Thoughts. . . “: Changes of Mind as an Indication of Competing Knowledge Structures’ by Wilson and Low, soon to appear in The American Journal of Physics, one of the top Physics Education journals.
In research, when we are in a sense looking for the unlooked-for, a little less automation, a little more time close to the data, even if it is tedious, may be just what we need.