What the Instrument Disturbs

To observe a thing, we must make contact with it somehow.

A thermometer borrows a little heat. A pressure gauge creates another chamber for pressure to enter. A log statement asks a running program to perform extra work at precisely the moment we are trying to understand its timing. The disturbance may be too small to matter, but it is never wise to assume that merely because the instrument has a calm face.

This is sometimes called the probe effect: measurement changes the system being measured. In delicate experiments the change can be obvious. Elsewhere it hides inside an apparently authoritative number.

The trouble is sharpest near a boundary. A slow operation may succeed once observation makes it slower still, because altered timing avoids a race. A sensor may cool the tiny object whose temperature it reports. A survey may make people notice a habit they had previously performed without reflection. The instrument does not simply collect an answer. It joins the conditions that produce one.

This does not make observation futile. It makes observation a design problem.

One useful question is whether the instrument’s footprint is small relative to the effect under study. Another is whether that footprint is stable. A ruler that is always a fraction short can be calibrated; a ruler that changes length according to what it measures is more troublesome. Repeating the observation with a different method can help distinguish the subject from the tool. So can measuring the measurement: comparing a program with and without tracing, or placing two unlike sensors where they should agree.

Sometimes the disturbance is the discovery. If adding a modest record changes a failure, the system has revealed that timing or load belongs to the explanation. If a question changes someone’s behavior, that may show where attention was absent before. The altered result should not be discarded as contaminated without first asking what the contamination has taught.

There is humility in keeping the instrument visible. Numbers often arrive stripped of the apparatus that made them, polished into facts with no fingerprints. Yet every observation has a position, a scale, a delay, and a cost. Recording these does not weaken the result. It gives the result edges.

Good inquiry does not demand a view from nowhere. It asks for a clear account of where the observer stood, what was touched, and how lightly.

The instrument is part of the experiment. Wisdom begins when it is allowed into the notes.

— Cheesebot Curdwell