Three phases that do not move when the origin does

Half of every diffraction measurement is thrown away before it is written down. The detector records how much light arrives and not when, so each reflection gives an amplitude and no phase — and the phases are the half that carries the structure.

Nothing in the measurement recovers them. What recovers them is a fact about the object being measured, and the fact is almost embarrassingly weak: the electron density is never negative, and it is concentrated at a few hundred sharp peaks of roughly known shape. Both statements are local — about what is at a point — and between them they constrain a quantity as global as a phase.

A phase is not a property of a crystal #

Before anything can be constrained, it has to be established what there is to constrain, and a single phase is not it.

Move the origin of the coordinate system by t. Every atom’s position changes by −t, so every structure factor picks up a factor exp(2πi h·t) and every phase changes by 2π h·t. The crystal has not moved. A phase is a number about the description.

That is not a technicality to be waved past. It means a procedure that predicts phases has to predict them relative to something, and it means any quantity claimed to be determined by the structure must be unchanged by the shift.

The sum of three phases whose index vectors add to zero is unchanged, because the three shifts 2π h·t, 2π k·t and 2π l·t add to 2π (h+k+l)·t = 0. Such a sum is a structure invariant, and the smallest one there is. It is called a triplet.

What the triplet knows #

An invariant that took every value equally often would be a definition and not a tool. This one does not.

Cochran’s result of 1955: for N atoms placed at random in the cell, the triplet Φ is distributed as P(Φ) ∝ exp(κ cos Φ) with

$$\kappa = 2\,|E_{\mathbf{h}} E_{\mathbf{k}} E_{\mathbf{h}+\mathbf{k}}| \,/\, \sqrt{N}$$

where the E are normalised structure factors — amplitudes divided by their root-mean-square at that resolution, so that ⟨|E|²⟩ is one. The distribution is peaked at zero, and sharply so when all three amplitudes are large.

That is the whole of direct methods in one line. The strongest reflections have triplets near zero. Their phases are therefore not free: knowing two of a triplet nearly determines the third, and a structure with a few hundred strong reflections has tens of thousands of such near-determinations, wildly over-constraining the phase set.

Where the concentration comes from #

The formula’s shape deserves a sentence, because both of its ingredients are the two local facts from the opening.

The √N in the denominator is atomicity. A structure of N equal atoms has $\sigma_3/\sigma_2^{3/2} = N^{-1/2}$ for the moments of its scattering, and that ratio is what sets the concentration. A structure whose density were smooth rather than atomic would have a different ratio and a weaker constraint; a structure with one heavy atom among many light ones has a larger one, which is why heavy-atom structures were solved first.

The product of amplitudes is positivity. The relation is a statement about the sign of a triple product of density coefficients, and it holds because the density cannot go negative — a non-negative function’s Fourier coefficients satisfy inequalities, and the triplet relation is the weakest of them.

Everything above is stated in |E| rather than |F| for one reason. The concentration is a comparison against the typical amplitude at that resolution, and a raw amplitude falls off with angle for reasons that have nothing to do with the arrangement — so a product of three raw amplitudes would rank every low-angle triplet as strong and every high-angle one as weak, exactly backwards, since the high-angle reflections are the ones that resolve atoms. Normalising is what makes the comparison mean anything, and the plot that supplies it is a separate calculation done first.

The centrosymmetric case, where a phase is a sign #

With a centre of symmetry at the origin every structure factor is real, so a phase is 0 or π. The triplet relation becomes a statement about signs, and it is the form direct methods were invented in.

This is a probability, not a rule, and the distinction is the subject’s whole difficulty. A relation right nine times in ten is useless applied once and decisive applied ten thousand times, and the art is in the bookkeeping that turns many weak statements into one strong one.

How the origin test is actually run #

Claiming that a sum is invariant is easy; the claim is worth having only if something would have caught it being false. So it is run rather than asserted, and the way it is run is worth describing because it is the cheapest possible experiment.

The structure is built, its structure factors computed, and its triplets collected. Then the atoms are moved — every one of them, by the same arbitrary vector, chosen to be nothing special: 0.137 along one axis and 0.291 along the other. The structure factors are recomputed from scratch. Every phase has changed. Each triplet’s three new phases are added and compared with the old sum.

The largest disagreement over four hundred triplets is a part in ten thousand billion, which is double-precision arithmetic and not physics. That is the positive half.

The negative half is the one that makes it a test. The same three reflections are taken with the last one’s indices moved by one, so the three no longer sum to zero, and the same comparison is made. That sum moves by up to eight tenths of a radian. A check that has never distinguished anything is not a check, and this one distinguishes the two cases by fourteen orders of magnitude.

That distinction is what origin fixing is actually choosing between, and it arrives before any relation is used. A phase the permitted shifts cannot move is already determined; giving it a value asserts something about the crystal. A phase they can move is a frame, and giving it a value costs nothing.

Why anybody expected this to fail #

Direct methods were disbelieved for twenty years, and the reason is worth recording because it was a good reason.

The phase problem is under-determined in an obvious way: a diffraction pattern of n reflections gives n numbers and asks for 2n. Any proposal to get the missing half from the measured half looks like getting something for nothing, and the crystallographic establishment of the 1950s treated it that way. Hauptman and Karle’s monograph of 1953 was, by several accounts, largely ignored.

What the objection missed is that the unknowns are not free. A crystal is not an arbitrary non-negative function; it is a sum of a few hundred sharp peaks whose shapes are known. That is an enormous amount of prior information, and the whole content of direct methods is the arithmetic that converts it into constraints on numbers nothing measured.

Hauptman and Karle shared the 1985 Nobel Prize in Chemistry for it, thirty-two years after the monograph.

A last point about independence, since the argument leans on it. The relations are not truly independent: a triplet involving reflections h and k shares a reflection with every other triplet involving either, so the information they carry overlaps. What makes the overlap tolerable is that the number of triplets grows much faster than the number of unknowns — as the cube of the number of strong reflections rather than the square, which is the figure usually quoted and is one factor short. Choosing two strong reflections fixes the third, and the pair is a triplet only if that third reflection is strong as well, so the size of the strong set enters a third time. The unknowns grow linearly, so the redundancy grows faster than the dependence. That is the same accounting as everywhere else in this collection: a surplus of constraints over unknowns, with the surplus doing the work. It is also why halving the number of strong reflections is far worse than halving the number of atoms.

What the invariant cannot do #

Three things the triplet does not give, each worth stating because each has been claimed for it at some point.

It does not give a phase. It gives a relation among three of them. A set of relations with no starting point determines nothing, because the whole phase set can be shifted by any origin and every triplet stays where it was. Fixing the origin — choosing a few phases arbitrarily, subject to a parity condition — is a separate step and a necessary one, and it is the same freedom a normaliser measures from the other side.

It does not distinguish a structure from its mirror image. Friedel’s law makes the amplitudes of a structure and its enantiomorph identical, so every triplet is too. The sign of the whole phase set is a second arbitrary choice.

It weakens as the structure grows. The √N is in the denominator, so a structure of a thousand atoms has concentrations a fifth of a structure of forty. That is the reason direct methods solve small molecules routinely and proteins hardly ever, and it is arithmetic rather than a limitation of any program.

Higher invariants #

The triplet is the smallest invariant and not the only one. A sum of four whose indices add to zero gives a quartet, whose distribution depends on the amplitudes of three extra reflections — the cross terms h+k, h+l, k+l.

Quartets are useful for a reason triplets are not: they can be negative. When the cross terms are weak, the quartet’s distribution peaks at π rather than at zero, and a relation that says not zero is exactly what a procedure needs to reject a wrong solution. The figures of merit that make multisolution work are built on those negative quartets, and they arrived with Hauptman and Karle’s work in the 1950s and Schenk’s in the 1970s.

Nothing here computes quartets. They are named because the triplet on its own does not explain how a procedure chooses between solutions, and the next rung is about exactly that choice.

The invariant and the symmetry #

One more thing the triplet is quietly doing, which matters to a site about symmetry.

The relation $\Phi = \varphi_h + \varphi_k + \varphi_{-h-k}$ requires the three indices to sum to zero, and in a group with symmetry there are more ways for that to happen than the arithmetic suggests. If the point group carries h to h′, then $\varphi_{h'}$ is φ_h plus a known offset — the phase shift the operation’s translation part contributes — so a reflection’s phase is determined by its whole symmetry-equivalent set up to that offset. Triplets can therefore be formed between reflections that are not obviously related, using symmetry to supply one of the three.

That is a large multiplier. In a group of order eight, each reflection has up to eight equivalents, so the number of usable triplets grows by roughly the square of the group order. It is the reason a structure in a symmetric space group is easier to solve than the same structure in P1, and it is a purely book-keeping advantage: the same measurements, more relations among them.

The systematic absences contribute in the opposite direction, by removing reflections entirely — and a reflection required to be zero has no phase at all, which has to be handled rather than ignored.

What is owned, and what is not #

Owned: the arithmetic of phase relations among structure factors computed from point sets built here. The origin-invariance check, the distribution of triplets against Cochran’s closed form, the sign relation against its own probability, and the multisolution behaviour of the next rung.

Not owned: crystal structure refinement, any real data set, any R factor against one. Every structure solved in these two essays is one this collection built, and its answer is known before the procedure starts — which is what makes the procedure’s failures measurable at all.

What makes a constraint usable #

Positivity and atomicity are both weak statements, and it is worth asking why the second is so much more useful than the first.

Positivity alone gives inequalities. The Karle–Hauptman determinants — matrices of structure factors whose non-negativity follows from the density being non-negative — are exact and they are hard to use: they constrain phases jointly rather than one at a time, and extracting a usable estimate from them means solving a determinantal problem for every reflection.

Atomicity gives probabilities, and probabilities compose. The triplet’s distribution is a statement about one relation, and ten thousand such statements multiply into something sharp. What makes them multiply is that they are nearly independent — different triplets involve different reflections — so the information adds rather than merely accumulating.

That is the trade the subject made, and it is worth naming because it is not obvious that a probabilistic constraint beats an exact one. It does here for a reason about scale: the exact constraints are few and awkward, and the probabilistic ones are many and simple, and many-and-simple wins when the number of unknowns is in the hundreds.

More is invariant than the triplet condition says #

The invariance argument above uses an origin shift by an arbitrary vector. In a crystal with symmetry the origin cannot go anywhere it likes, and that restriction makes a larger class of quantities invariant than the sum-to-zero rule alone allows.

An origin shift has to carry the group’s symmetry elements onto themselves, so the permitted shifts are a finite set — the half-cell translations in a centrosymmetric group, and the corresponding sets elsewhere. A sum of phases is unchanged by every permitted shift whenever the corresponding sum of index vectors is zero modulo the arithmetic those shifts allow, which is a weaker condition than being zero outright.

Those quantities are the structure seminvariants, and they include things a triplet cannot reach: single phases whose indices have the right parities, and pairs of phases rather than triples. In a centrosymmetric group with origins at the half-cell translations, a reflection all of whose indices are even has a sign that no permitted origin shift changes — so its sign is a property of the crystal, determined before any relation is used.

That is what origin fixing is actually choosing between. The phases assigned arbitrarily at the start of a determination have to be ones that are not seminvariant, since a seminvariant phase is already determined and assigning it a value is asserting something rather than choosing a frame. Programs select the origin-fixing set by exactly that test, and a set chosen for its amplitudes alone will sometimes contain a seminvariant and fix the origin only partially — a failure that looks like poor convergence rather than like a mistake.

Why the amplitudes have to be normalised first #

The concentration formula is written in E rather than in F, and the substitution is not cosmetic. It is the step that makes the relation usable at all, and it depends on a quantity computed elsewhere.

A raw amplitude falls off with scattering angle, because atoms are not points and they vibrate — so a reflection at high angle is weak for a reason that has nothing to do with the structure. Cochran’s concentration is a product of three amplitudes, so feeding it raw amplitudes would rank every high-angle triplet as weak and every low-angle one as strong, when the information runs the other way: the high-angle reflections are the ones that resolve atoms, and a relation among them is worth more rather than less.

Normalising divides each amplitude by the root-mean-square at its own resolution, which is exactly the quantity Wilson’s plot supplies. What is left is a measure of how far a reflection stands above what the cell’s contents alone would produce — a statement about arrangement rather than about scattering power — and that is the quantity the probability theory is written in.

So the chain has three links and each is a separate calculation: the shell means come from the mean intensity, the normalised amplitudes come from the shell means, and the triplet concentrations come from the normalised amplitudes. A poor normalisation weakens every relation at once, and it does so silently, because the relations remain relations and merely stop being sharp.

Where the ladder goes next #

A set of relations is not a procedure. Bootstrapping a whole phase set out of them needs a starting set, an iteration, and a way of telling a good answer from a bad one without knowing the answer — and the last of those is where the difficulty turns out to live.

The other direction is towards what a measurement knows without any phases at all. The average intensity in a resolution shell is such a quantity, and it knows the content of the cell and nothing about the arrangement — which makes it the first thing anybody computes and the last thing anybody thinks about.

Both additions are about the same thing: a relation is only as good as the frame it is stated in. The seminvariants say which phases the frame has already fixed, and the normalisation says which amplitudes the frame has already accounted for. Getting either wrong leaves the arithmetic intact and the conclusions weaker, which is the failure mode this whole field has to be built against.