The wrong model moves too far for first order

Better counting, and the wrong model found that a smooth error in the data, the scale variation an uncorrected absorption leaves, makes perfect counting choose the wrong disorder model in eight of ninety comparisons of two models at five molecule sizes. A cubic site is blind to a degree-two error then tried to predict which eight. The obvious predictor, the error’s correlation with the two models’ difference, caught none of them. What turned every wrong verdict was the error each model’s refinement absorbs, and in all eight the wrong model, with fewer parameters, absorbed more. The essay found a symmetry rule for the pairs that can never go wrong, those at a cubic site under an error of degree two, and left the rest to a calculation it described but did not make.

The calculation is a first-order account with the parameters free. Refine each model against error-free data. At those solutions, the columns of each refinement’s Jacobian are the directions in which its parameters can move its amplitudes. A small error then shifts each refinement along those directions, and the part of the error each model can follow is its projection onto them. Everything needed is available before any data with the error in them exist: the two models, the reflection list and the shape of the error. This essay makes the calculation for all ninety cases and sets it against the refinements actually run.

The account #

Write the two models as A, the right one, and B, the wrong one. Against error-free data from A, model A fits exactly and model B leaves a residual $D$, the part of the difference between the models that B’s parameters cannot absorb. At its optimum, B’s residual is orthogonal to every column of B’s Jacobian: a residual with a component along a direction the parameters can move would not be optimal.

Now add a smooth error $e$ to the data, three per cent of the mean amplitude, shaped like the second Legendre polynomial about a direction on no axis of the cell. To first order, each refinement follows the part of $e$ lying along its own Jacobian’s directions, its projection, and leaves the rest. So the right model’s misfit becomes the error minus its projection, and the wrong model’s becomes its old residual plus the error, minus the projection of the error onto its directions, the residual itself having no component along them. The difference that decides the verdict is

$$\Delta = |D|^2 + 2\,D\cdot e + |P_A e|^2 - |P_B e|^2 .$$

The first two terms are the earlier predictor’s: the models’ difference and the error’s correlation with it. The last two are what it left out: how much of the error each model’s freedom can follow. The verdict comes out right when $\Delta$ is positive. Every term comes from the two error-free refinements, their Jacobians at the solution and the error’s shape; nothing is refined against erroneous data.

Seventy-seven of eighty-five #

The figure at the head of this essay sets the prediction against the outcome for every comparison in which the two models differ at all, eighty-five of the ninety; the other five are between models the data cannot separate even without error. Points in the shaded quadrants are cases where prediction and outcome agree. Seventy-seven of the eighty-five agree, and most of them lie close to the diagonal: where the first-order account is right about the sign it is usually close about the size too, within a few per cent for most of the orthorhombic and tetragonal pairs at the smaller sizes.

The eight that disagree are what the account was built for, and it does half the job. Of the eight wrong verdicts, it catches four. The earlier predictor caught none. And it raises four false alarms, right verdicts it predicts would go wrong. So as a test that a structure determination could run before collecting data, flagging a pair as dangerous, it would have flagged eight pairs and been right about four.

The table shows where the catches are. All four are the two tetragonal pairs that go wrong at both of the two largest sizes, the pairs of models labelled 9/10 and 10/11 at a hundred and twenty-eight and two hundred and fifty-six atoms. The account sees them coming at both sizes. It misses the same tetragonal pair 10/11 at sixty-four atoms, the orthorhombic pair 2/3 at both large sizes and a second tetragonal pair, 9/17, at two hundred and fifty-six.

Where first order holds and where it does not #

The account is linear in the error, and whether a linear account can work depends on whether the refinements respond linearly. That can be measured directly, by comparing the error each refinement actually absorbed with what its first-order projection says it would.

The two refinements behave quite differently. The right model follows the error almost exactly as first order says. At every size up to sixty-four atoms its ratio is between 0.93 and 1.05 for every orthorhombic and tetragonal pair. It starts at the exact solution, the error moves it a little, and a little is what a linear account describes.

The wrong model does not. Its ratio ranges from below a twentieth to above seventy, and occasionally the actual absorption is negative: the wrong model’s refinement, handed the error, ends in a worse place than its error-free solution plus the error would put it, because it has moved to a different minimum. That is not surprising once stated. The wrong model starts from a solution in which it is already compromising, with its atoms pulled along their loci to absorb a difference between models that what refinement takes back from a wrong model found could be most of the difference. A further push from the error can move such a compromise a long way, or onto a different compromise altogether.

At the largest sizes the right model stops being linear too, in some orthorhombic pairs. At two hundred and fifty-six atoms the right model of pairs 1/6 and 3/4 absorbs about ten times what its first-order projection allows, and that is exactly where two of the false alarms come from: the account, expecting the right model to follow little of the error, predicts it will lose, and the right model in fact follows much more and wins.

Every failure has one of two causes #

Read across the cases the account gets wrong, and they sort into three kinds.

Near-ties. At sixty-four atoms the tetragonal pair 9/10 has an actual margin of about a thousandth of a unit on an error of hundreds, and the pair 10/11 a predicted margin of the same size. Both refinements there are within a factor of two of their first-order shares, and the account is right to within its own precision: it says the verdict is a coin toss, and the coin fell the other way. A decision made on a margin that small is the situation near-symmetry and the tolerance described for symmetry itself: once a threshold enters, the answer depends on it, and no account of the error can do better than say so.

A small error on a large absorption. The tetragonal pair 9/17 at two hundred and fifty-six atoms has both refinements close to linear, the right model at 1.05 of its first-order share and the wrong one at 0.86. But the wrong model absorbs a great deal of the error there, and fourteen per cent of a large absorption is more than the margin between the models, which is under one unit. The account predicts a comfortable win and the verdict is a narrow loss.

A refinement that moved too far. The other five failures each have one refinement outside a factor of two of its first-order share. The orthorhombic pair 2/3 at both large sizes goes wrong because the wrong model absorbs ten to fourteen times what its directions at the error-free solution allow; the account cannot see it coming because the motion is not there at first order. The tetragonal pair 8/18 at sixteen atoms is predicted wrong because its wrong model absorbs far less than first order allows. And the two orthorhombic false alarms at two hundred and fifty-six atoms come from the right model moving further than first order.

So the account has a statement of its own reach. It fails exactly when its error in estimating what the two refinements absorb is larger than the margin between the models. That happens in three ways: the margin is tiny, the absorption is large enough that a modest relative error swamps the margin, or a refinement moves far enough that first order is wrong by a factor. Five of the eight failures are the third kind, the cases where a model’s freedom is used non-linearly. Which those are cannot be read from the Jacobian at the error-free solution, since the non-linearity is in how the Jacobian changes as the refinement moves.

Why a wrong model’s compromise is fragile #

The pairs in this census come from the molecule size that hides a disorder, which set two disorder models with the same occupancy against each other: a molecule spread over sites in two different ways that leave averaged structures a few per cent apart. The wrong model, refined against the right one’s data, can only match it by bending. Its partial atoms slide along the lines and planes their sites allow, and their displacement parameters grow to smear density where the true structure has it. At the error-free solution that bending has been carried as far as it helps, and the misfit is at a minimum, but the minimum sits on a slope made of compromises.

A minimum of that kind is shallow in some directions and steep in others, and the steepness changes quickly as the parameters move. Hand such a refinement a smooth error and it may slide a long way down a shallow direction that first order sees as barely mattering, or tip into a neighbouring minimum where a different compromise fits better. The right model has no compromise to undo: its atoms are where the data put them, its displacements are true ones, and a small error moves it a small distance. That asymmetry is the whole of the two lanes in the figure above, and it is why the account built at the error-free solution describes one refinement well and the other badly.

What a crystallographer could do with it #

A refinement program can compute everything the account needs. The Jacobian at convergence is what it already uses for its last cycle, and the shape of a likely systematic error, an absorption surface of degree two or four about the crystal’s longest dimension, is known from the crystal’s habit. So before deciding between two disorder models on a few units of misfit, it would be possible to ask how much of a plausible absorption error each model could follow, and whether the margin survives that. Where the account says the verdict is safe and both refinements are near linear, the verdict is safe; in this census that is most cases. Where it flags a pair, the flag is right about half the time, and the remedy is the one every reflection, several times over points to: redundant data measured at different crystal orientations, which average the absorption surface away before any model sees it. The account cannot certify a verdict for a pair whose wrong model is far from linear; it can only say that it does not know.

What the symmetry rule did and this does not #

The cubic essay’s rule and this account are complementary in a way worth stating. The rule is exact and narrow: at a cubic site an error of degree two has no component the site’s averaged structure can follow, so neither model absorbs any of it and no verdict can turn. The computation here confirms that in its own terms: for every cubic pair the right model’s absorption is effectively zero, its ratio enormous for lack of anything to divide by, and the verdicts are all right. The rule is a statement of exact zero, and an exact zero is the one case a linear account cannot get wrong.

The account is approximate and broad. It applies at every site, gets the sign right in nine cases of ten, and fails where the problem stops being linear. Together they say what a crystallographer choosing between two disorder models could know in advance: at a site whose Laue group keeps no harmonic of the error’s degree, nothing can go wrong, as the shell that splits into kinds counts those harmonics; elsewhere, the first-order account flags the pairs at risk, catches half of them, and its misses are the pairs whose wrong model is already far from where it would be in a clean refinement.

What the account rests on #

The error has one shape. Every number is for a degree-two scale error of three per cent about one fixed direction, with one reflection list, whose size against the number of refined parameters is the ratio the unknowns against the observations computes; a longer list would change every projection. A different shape changes the projections, and a degree-four error reaches the cubic site, as the earlier essay found.

The refinements are the ones the earlier essays ran: a scale, the general atom’s position and displacement, each special atom’s position on its locus with a two-component displacement, occupancies held at the values the composition fixes, the wrong model started from four points and the best kept. The Jacobians are taken by the same finite steps the refinement uses, at its final solution. A refinement with more parameters, or restraints, or a weighting scheme, would have different directions and different absorptions.

And the account is first order by construction. A second-order account would include how the Jacobian itself changes as a refinement moves, the curvature that the failures turn on, and it would need the Hessian of each model’s amplitudes with respect to its parameters. That is a larger calculation, not a different idea, and it has not been made here.

Who worked out the pieces #

The projection of an error onto a least-squares problem’s parameter space is the textbook account of how a fit responds to a systematic perturbation, and its use to anticipate biases in crystallographic refinement goes back at least to the analysis of systematic errors in the 1960s and 1970s, in work associated with Cruickshank and with Hamilton, whose test for choosing between two refined models is the comparison the verdicts here are made by. What these essays add is the population: eighteen pairs at five sizes, each refined, so that a predictor can be scored against outcomes rather than illustrated on one case.

What the account must refuse #

Both refusals are claims a reader could reasonably have expected to hold. The first-order account is the natural repair of the earlier predictor, and it does repair half of what was missing. It does not predict every verdict, and the figure records where it fails. The correlation alone, the predictor the earlier essay tested, is refused again on the same data, catching none of the wrong verdicts; it was never going to, since in every wrong case the decisive term is an absorption it leaves out.

Still open: whether the moving pairs can be named in advance #

The failures cluster in particular pairs, 2/3 at the orthorhombic site above all, and the account’s own diagnosis says why: the wrong model absorbs many times what first order allows, so its refinement moves far from its error-free solution. Whether that can be predicted is a question about the error-free refinement alone. A wrong model whose atoms sit at a compromise, with large shifts along their loci and inflated displacements, may be one whose Jacobian changes fast as it moves; the curvature of its misfit at the error-free solution, the second derivative the first-order account ignores, would measure it. If the pairs with large curvature are the pairs that fail, then a crystallographer could flag them from a clean refinement and know in advance which comparisons a smooth error can overturn. That is a second-order computation on the same ninety refinements, and it has not been run.