The average that knows the atoms and not where they are

A diffraction experiment produces a list of intensities on an arbitrary scale, measured from a crystal whose atoms are vibrating by an unknown amount. Two numbers are wanted before any structure can be discussed: the factor that puts the intensities on an absolute scale, and the displacement factor that describes the vibration.

Both come out of the data without any structure at all, and the argument is one line long.

The cancellation #

A structure factor is a sum over atoms:

$$F(\mathbf{h}) = \sum_j f_j \exp(2\pi i\, \mathbf{h}\cdot\mathbf{r}_j)$$

Square it and the result has two kinds of term. The diagonal terms f_j² carry no position. The cross terms f_j f_k exp(2πi h·(r_j − r_k)) carry all of them.

Now average over many reflections at the same resolution. The cross terms have phases 2π h·r_jk which, for reflections far enough out and vectors not too short, are spread over the circle — so they average to nearly nothing. The diagonal terms have no phase and average to themselves.

$$\langle |F|^2 \rangle \;\approx\; \sum_j f_j^2$$

That is Wilson’s observation of 1942, and its content is that the left-hand side is measurable and the right-hand side is a sum over the cell’s contents: how many atoms of each element, and nothing about where they are.

Reading the line #

With a displacement factor B applied to every atom, each f_j carries exp(−B s²) where s = sinθ/λ, so the whole average carries exp(−2B s²). With an unknown scale K on the measurements:

$$\ln\!\left( \langle I \rangle / \textstyle\sum f^2 \right) \;=\; \ln K \;-\; 2B s^2$$

A straight line against s², with slope −2B and intercept ln K. Plot it, fit it, read both.

In the computation above the answers are known, because the structure was built with a stated B of 3.4 and the intensities multiplied by a stated scale of 0.37. The fit returns 3.45 and 0.3665. That is the only way to check a procedure of this kind: give it a problem whose answer is on the other side of the paper.

The claim that the arrangement does not matter #

The whole argument rests on the cross terms cancelling, which is a claim about arrangements. It can be tested directly: take the same atoms and put them somewhere else.

That is as clean a separation of the local from the global as this subject offers. The Wilson plot is a global average that carries only local information: what the atoms are, and not the arrangement that every one of the individual intensities is about.

Where the line bends, and what the bend is #

Real Wilson plots are not straight. Textbooks say so and usually leave it there. The reason can be computed.

The cancellation assumed the cross terms have random phases, and in a molecule they do not. Two bonded atoms are always the same distance apart, so their cross term contributes coherently at every resolution — with a sign that oscillates as s grows. Sum over all pairs and the result is a wave superimposed on the straight line, whose period is set by the interatomic distances.

That is a prediction, and it can be checked against a completely separate calculation.

Debye’s formula is the angular average of the interference term. Averaging exp(2πi g·r) over all directions of g gives a Bessel function of 2π|g|r in the plane, so

$$\langle I \rangle = \sum_j f_j^2 + \sum_{j \neq k} f_j f_k\, J_0(2\pi |\mathbf{g}|\, r_{jk})$$

with the second sum running over the list of distances. The first term is Wilson’s; the second is what he threw away.

Why the correlation is 0.89 rather than 1 is worth saying. Debye’s average is over all directions; a crystal’s shell average is over the finitely many reciprocal-lattice directions the shell contains. The two agree as the shell fills up and differ where it does not, and that difference is the whole of the residual.

What “content” means, and why it has to be known #

The right-hand side of Wilson’s identity is Σ f_j², a sum over the atoms in the cell. Computing it requires knowing what is in the cell, and that is a real requirement rather than a formality.

In practice the content comes from chemistry — a formula and a guess at how many formula units fill the cell, which is itself estimated from the cell volume by a rule of thumb. So the Wilson plot’s inputs are the intensities and a chemical assumption, and a wrong assumption produces a wrong scale.

The wrongness is systematic and mostly harmless. Getting the content wrong by a constant factor shifts the intercept and leaves the slope alone, so the displacement factor survives and the scale does not. Getting the composition wrong — assuming carbon where there is sulphur — changes how Σf² falls with resolution and therefore tilts the line, which corrupts the displacement factor too.

That is the sense in which this is not quite a measurement of nothing. It is a measurement given a chemical formula, and the formula is the only thing about the structure it needs.

The order of operations, which is not obvious #

There is a circularity here that a first reading misses, and resolving it is part of what makes the method usable.

To normalise an amplitude, the expected intensity at its resolution is needed. To get the expected intensity, the displacement factor is needed. To get the displacement factor, a Wilson plot is needed — which is drawn from shell averages of the raw intensities and needs no normalisation at all.

So the order runs: raw intensities → shell averages → the line → scale and B → expected intensities → normalised amplitudes → phase relations. Nothing is circular, but each step depends on the last and an error early is an error everywhere.

The step most often done badly is the first: deciding which reflections belong to which shell, and whether the shells are of equal width in s² or of equal population. Equal width gives shells whose populations depend on the dimension and on the shape of the region summed over, and which end is the thin one is not obvious — in three dimensions the innermost shells hold tens where the outer ones hold thousands, and in the plane computation below the order is reversed. Either way a fit that weights every shell alike is trusting the thin ones as much as the fat ones. Equal population removes the question, and it is what modern programs do.

What the plot is used for #

Three uses, in decreasing order of how much anybody thinks about them.

Normalisation. Every E value in the phase relations of the neighbouring essays is an amplitude divided by its expected value at that resolution, and the expected value is exactly what this plot supplies. Direct methods run on E and E runs on Wilson.

Scale and displacement. The two numbers the fit returns are starting values for refinement, and a refinement started at a badly wrong scale often does not recover — which matters most where observations barely outnumber parameters.

A check on the data. A plot that is wildly non-linear, or whose slope is negative, says something is wrong before any model exists — twinning, an incorrect content, ice rings, a detector problem. It is the cheapest diagnostic in the subject and it is run on every data set. It plays the part a systematic absence plays for the space group: a fact visible in the data before any model exists.

How many reflections a shell needs before its mean means anything #

The cancellation is an argument about many terms, and many is a number. It can be estimated, and the estimate explains the low-resolution cutoff better than the appeal to non-randomness does.

Within a resolution shell the intensities of a structure with no centre are distributed exponentially about their own mean. An exponential distribution has a standard deviation equal to its mean, so a single reflection is a hopeless estimate of the shell average and a shell of m reflections has a standard error of one part in √m. Ten reflections give a shell mean uncertain by 32%; a hundred give 10%; a thousand give 3%.

That is the real reason the first few shells are cut. A sphere of radius s holds a number of reflections growing as s³, so a shell of fixed width holds a number growing as s²: the innermost shells are thin in exactly the sense that matters, holding tens of reflections where the outer ones hold thousands. The molecular structure’s departure from the line is largest there, and so is the sampling error, and the two are easy to confuse — one is a real feature of the arrangement and the other is noise. Choosing shells to hold equal numbers of reflections rather than equal widths separates them, at the cost of shells whose resolution ranges are no longer comparable.

And that is the reverse of the three-dimensional case, which is the one everybody quotes. In space a sphere of radius s holds a number of reflections growing as s³, so a shell of fixed width holds a number growing as s², and the innermost shells are the thin ones. In the plane a disc holds a number growing as s², so equal widths in s² hold equal numbers — until the annulus runs past the edges of the square of indices being summed, which is what makes the outermost shells of the computations above the noisy ones. Any wiggle at the right-hand edge of those plots has fourteen reflections behind it and a standard error of twenty-seven per cent, and should be read as nothing at all.

A centre makes it worse by a factor of √2. Where the structure is centrosymmetric the intensity distribution is not exponential but the square of a normal variable, whose standard deviation is √2 times its mean — the same fact that makes the distribution’s shape a test for a centre, read as a nuisance instead of as a signal. A centric shell therefore needs twice as many reflections as an acentric one for the same precision on its mean, and a fit weighted as though all shells were alike will trust the wrong ones.

The B that comes out is not any atom’s B #

The fit returns a slope, the slope is called −2B, and the letter invites a reading it does not deserve.

An atom’s displacement parameter describes how far that atom moves about its mean position. The quantity Wilson’s line returns is whatever makes the observed intensities fall off with resolution faster than Σ f² does — and thermal motion is only one of the things that does that. Static disorder over several sites falls off the same way. Absorption and extinction errors that were not corrected fall off the same way. So does a crystal whose outer reflections were measured for too short a time.

The overall B therefore absorbs all of them, which is why it is useful as a diagnostic and misleading as a measurement. A structure refined afterwards will have per-atom displacement parameters whose mean is generally smaller than the Wilson B, and the difference is not an error in either number: the refinement has a model for the atoms and puts everything else into the scale, and the Wilson fit has no model at all and puts everything into one exponential.

In the computation on this page the two agree exactly, because the structure was built with a stated B and nothing else was done to it. That agreement is a check on the arithmetic and it is not a claim about data — the useful statement here is the identity, and the extent to which a measured crystal satisfies it is a separate question with an experimental answer.

There is a symmetry between the two paragraphs above that is worth naming. The shell average is uncertain because a finite number of reflections is a finite sample, and the slope is ambiguous because a single exponential is a coarse model. Neither is a defect of Wilson’s identity, which is exact; both are properties of what has to be done to a measurement before the identity can be applied to it. The identity is a theorem and the plot is an estimator, and almost every disappointment with a Wilson plot comes from expecting the second to behave like the first.

Where the exactness stops #

Every number in this essay is computed from a stated point set with stated scattering factors, so nothing is measured. What is approximated is worth listing.

The cancellation is approximate, and its quality depends on how many reflections are in a shell. With few, the average is noisy; the two arrangements above differ by six hundredths in B for that reason and not because either is wrong.

The low-resolution shells are excluded from the fit by a stated cutoff. In this computation the atoms are independent, so the line is straight all the way down and the cutoff barely matters. In a real crystal it matters a great deal, and the reason is the bend above.

The form factors are Gaussians rather than tabulated values. That is a simplification of the scattering physics and not of the argument: the argument needs only that f_j be a known function of resolution. The same simplification runs through every diffraction figure here.

Two more things the average knows #

The identity above is the first moment of the intensity distribution in a shell. Two further quantities come from the same source and are worth listing, because together they are most of what a data set says before a model exists.

The shape of the distribution says whether there is a centre. Wilson’s other 1942 result: with a centre, the structure factors are real and their squares follow one distribution; without, they are complex and follow another, and the two differ most among the weak reflections. That is decided by a statistic and not by any single reflection, and it is computed from the same shell averages.

Departures within a shell say whether the crystal is twinned. A twinned crystal’s measured intensity is a sum of two, which narrows the distribution: extreme values become rarer because two independent draws are being averaged. The moments shift in a characteristic direction and a twin is detected without any structure — which is the same style of argument as everything else here, applied to a defect rather than to the content.

Behind both of them is the resolution limit that bounds everything: an experiment reaches a sphere of reflections and no further, so a map computed from what it measured is the structure convolved with the transform of that sphere. No amount of statistical care recovers what was never measured, and the statistics above are about extracting everything a bounded measurement contains rather than about extending it.

Who found it, and when it mattered #

A. J. C. Wilson published the argument in 1942, in a short Nature note, and the same paper contains the distribution that decides whether a structure has a centre. Both results come from the same source: what a structure factor’s statistics look like when the atoms are treated as randomly placed.

The timing is the interesting part. In 1942 there was no way to put intensities on an absolute scale except by comparing with a standard crystal, which was awkward and inaccurate. Wilson’s method needed nothing but the data and a chemical formula, and it made absolute scaling routine overnight. The statistical view of a diffraction pattern that direct methods would later depend on begins here.

What is owned here #

The arithmetic: structure factors from stated point sets, shell averages, the fit, the recovered scale and displacement factor against the values put in, the two-arrangement control, and the Debye calculation compared with the measured departure.

Not owned: any real data set, any tabulated scattering factor, and any claim about what a Wilson plot of a particular crystal looks like. The bend is computed for a cell of rings built here, and what it demonstrates is a mechanism rather than a measurement.

The assumption that is not quite true, and what it costs #

Wilson’s derivation treats the atoms as placed at random, and no crystal’s atoms are placed at random. It is worth being precise about which part of the conclusion survives that.

The identity ⟨|F|²⟩ = Σf² does not depend on randomness at all when the average runs over all reflections. It is Parseval’s relation: the mean square of a transform is the integral of the squared function, and the atoms’ positions cancel exactly rather than approximately.

What randomness buys is the shell version, where the average runs over reflections at one resolution only. There the cross terms cancel because their phases are spread across a shell, and how well they cancel depends on how many reflections the shell contains and on whether the interatomic vectors are short enough to make neighbouring reflections agree.

So the error is largest where the shells are thinnest, which is at low resolution, and it is largest where the interatomic distances are most repetitive, which is in a structure of identical molecules. Both corrections point the same way and both are visible in the figures above.

Where the ladder goes next #

Sideways, to the other statistic in Wilson’s paper: whether a structure has a centre of symmetry, which no single reflection carries and the distribution of all of them decides. The two results are the same idea applied to the first moment and to the shape.

Forward, to what the normalised amplitudes are for: the phase relations that turn a surplus of measurements into a structure, and the search that is needed because a surplus of relations is not the same as an answer.