A cell from a bag of spots

Indexing a powder pattern is hard because the experiment has thrown away the directions: what is left is a list of d spacings, and finding a cell whose reflections have those spacings is a search over cells rather than a construction.

A single-crystal experiment keeps the directions. Every reflection is a vector in reciprocal space, measured as a position on a detector and a setting of the goniometer, and the list is a bag of vectors with no labels on them. Each is a point of the reciprocal lattice, and which point is exactly what is unknown.

That version is easy, and it is worth seeing why.

Differences generate the lattice #

Every observed vector is a reciprocal-lattice vector. So are all their differences, since a lattice is closed under subtraction. And the differences of a large enough set of lattice vectors generate the whole lattice — not a sublattice — provided the set was not itself confined to one.

So the procedure is: take differences, find a basis of what they generate, reduce it. No indices are needed at any stage, and none are assigned until the end.

Finding a basis of the lattice generated by a list of integer vectors is Hermite normal form, and it is the same elimination as Gaussian elimination with the division replaced by a greatest common divisor. Reducing that basis to a short one is Lagrange or LLL, which is short work in three dimensions.

The whole computation is exact. The vectors are integers in the true basis; the elimination is integer arithmetic; the determinant of the reduced basis is the index of the recovered lattice in the true one, and it comes out at one.

How the elimination works #

The step that converts a pile of vectors into a basis is worth describing, because it is the only piece of machinery here and it is short.

Take the vectors as rows of a matrix. Work down the columns. In each column, find the rows with a non-zero entry and reduce them against each other by repeated subtraction — exactly Euclid’s algorithm, applied to whole rows — until one row is left with a non-zero entry in that column and every other row has a zero. Move to the next column.

What comes out is a triangular basis of the lattice the rows generate, and its determinant is the index of that lattice in ℤ³. The procedure is exact, terminates, and is the integer analogue of row echelon form.

The place it can go wrong is the division. Ordinary elimination divides, which leaves fractions and generates a finer lattice than the rows do. Integer elimination subtracts multiples instead, which is the whole difference between a basis of the lattice and a basis of the vector space it spans. A procedure that quietly used floating-point division here would return a cell too small by a factor nobody would notice.

That failure and the sublattice failure are the same failure in opposite directions, and both produce a self-consistent answer. One comes from the data being incomplete and one from the arithmetic being wrong, and the arithmetic is the half that can be made not to fail.

Why the differences and not the vectors #

A reader may ask why the procedure takes differences at all, since the observed vectors are themselves lattice vectors and generate the lattice directly.

They do, and the differences are a safeguard. If the origin of reciprocal space has been located wrongly — by a beam-centre error, which is the commonest instrumental fault — then the observed vectors are all offset by a constant, and none of them is a lattice vector. Their differences are unaffected, because the offset cancels.

So a procedure built on differences recovers the correct lattice from data with a misplaced origin, and then locates the origin afterwards by asking where it must be for the observations to have integer indices. That is a real robustness and it costs nothing but a quadratic number of subtractions.

It is also the same trick as the triplet. A phase sum whose indices cancel is unchanged by an origin shift; a difference of positions whose origin cancels is unchanged by an origin error. Both are the same manoeuvre — build a quantity from which the arbitrary part cancels — applied once to a position in reciprocal space and once to a structure factor’s phase.

And the cost of not taking differences is not a small loss of accuracy. Displace the origin by a vector that is not in the lattice and the observations stop being lattice vectors altogether; a generating set containing them then generates the lattice and the offset, which is a strictly finer lattice and here is the whole of the integer grid. The recovered volume is one against a true fifty-two — a cell fifty-two times too small, reported as three integer vectors with an integer determinant, exactly as a correct answer would be.

The ambiguity nothing removes #

Now the interesting part, and it is the sharpest form of a question that runs through the whole of indexing.

Suppose the experiment sees only some of the reflections. Not at random — systematically, because the structure is centred and half of the reciprocal lattice is systematically absent. The observed vectors then satisfy a condition, their differences satisfy it too, and the differences generate a sublattice of the true reciprocal lattice.

The data are perfectly consistent with the larger cell. Every observed reflection has integer indices on it; the reduced basis is a genuine basis of a genuine lattice; the volume is a plausible number for a plausible compound. A procedure that reports a cell will report this one, and the report will contain no warning.

What settles it is a different kind of observation altogether: are the missing reflections absent for a reason? A body-centred structure has h + k + l odd reflections absent because the two halves of the cell cancel — an interference effect, exact and predicted. A primitive structure with a cell twice as large has those reflections present and merely weak. The two are distinguished by looking hard at where the missing ones should be, which is a measurement rather than an inference from the positions.

Why this is the right way round #

It is worth stating the resulting convention, because it looks arbitrary and is not.

A crystallographer indexes on the smallest cell consistent with the observed reflections, and then looks for absences. The alternative — index on a large cell and treat the missing reflections as accidentally zero — is always available and is always wrong, because it explains nothing. A systematic absence is a prediction: given the centring, the absent reflections are named in advance and all of them are absent. An accidentally-zero reflection is an observation with no prediction behind it.

So the ambiguity is not resolved by the indexing at all. It is resolved by the ordinary business of reading a space group from its absences, and the indexing’s job is only to produce a cell small enough that the absences have something to say.

What the procedure is not #

Three limitations, each of which the real problem has and this computation does not.

Errors. Real reflection positions are measured, so the differences are not exactly lattice vectors and the elimination cannot be integer arithmetic. Real indexing procedures work with a tolerance, and the tolerance is what makes them hard: too tight and the true lattice is missed, too loose and a spurious finer one is found. The computation here uses exact positions, so it demonstrates the structure of the problem and not its difficulty.

Multiple lattices. A real crystal may be twinned, or two crystals may be in the beam, in which case the bag of vectors contains two interleaved lattices and their differences generate something finer than either. That is the same failure as the sublattice case in reverse, and it is diagnosed the same way — by whether the reflections predicted by the finer lattice are there.

Outliers. A single misplaced spot is a vector that is not in the lattice, and one such vector’s differences with the rest generate a much finer lattice. Real procedures are robust against a few, by searching for lattices that account for most of the observations rather than all of them, and the arithmetic here has no such robustness.

The same shape as the powder problem #

Setting the two indexing problems side by side makes clear what the directions buy.

A powder pattern gives |g| for each reflection and no direction, so the task is: find a cell such that the lengths of some of its reciprocal-lattice vectors are the observed list. That is a search over cells — six parameters, with the observed list as the objective — and every classical algorithm for it (Ito’s, de Wolff’s, and their descendants) is a search with heuristics.

A single-crystal pattern gives the vectors, so the task is: find the lattice they generate. That is a construction, and it finishes.

The difference is exactly the information the powder averaging destroyed, and what a powder pattern loses is a good account of how much. Everything hard about powder indexing is hard because a search replaced a construction.

The other conditions that produce a sublattice #

Centring is the commonest reason a data set is confined to a sublattice, and it is not the only one.

Glide planes and screw axes produce absences too, but only along rows and planes of reciprocal space rather than throughout it — so their differences still generate the whole lattice, and they do not corrupt an indexing. That difference between centring absences and translational-symmetry absences is the same one that makes centring visible in a reduced cell and glides invisible.

A superstructure that was missed does the opposite: the true cell is larger than the one indexed, and the extra reflections are weak rather than absent. A data set collected with too short an exposure sees only the strong ones, indexes on the substructure’s cell, and reports a smaller cell than the crystal has. That is the same ambiguity from the other side and it is what a superlattice’s extra reflections are.

Both errors are errors of a factor, and both are settled by looking where the doubtful reflections should be rather than by any amount of arithmetic on the ones already recorded.

The distinction between the two kinds of absence is worth stating as a rule, because it decides which conditions an indexing has to worry about and which it can ignore. A centring condition is a condition on every reflection, so the observed set is closed under addition and is a genuine sublattice; its differences generate that sublattice and nothing more. A glide or screw condition applies only to a plane or a row — h0l for a glide, 0k0 for a screw — and a plane and a row are a vanishingly small part of reciprocal space. Almost every reflection is still recorded, the recorded ones still contain a generating set of the whole lattice, and the recovered cell is exactly right.

So the rule is: an absence corrupts an indexing when it is a condition on all three indices at once, and not otherwise. That is the same statement as the one made in reading a space group from its absences from the other side, where the centring conditions are the ones read off the whole data set and the glide conditions are the ones read off single zones — and it is why a reduced cell reveals centring and says nothing at all about glides.

What the cell does and does not decide #

A cell is not much on its own, and it is worth listing what has and has not been settled once one is in hand.

Settled: the translation lattice of the structure, hence the volume, hence — with a density measurement or a chemical formula — how many formula units are in it. That last number is the input the Wilson plot needs to put intensities on an absolute scale.

Not settled: the centring, which needs the absences; the point group, which needs the intensities and their symmetry; the space group, which needs both; and the structure, which needs the phases. The cell is the first of five things and the only one that comes from positions alone.

Also not settled: which cell. The reduced basis is short but the conventional cell — the one a report would print — is chosen by a convention that depends on the lattice’s symmetry, and getting from one to the other is the reduction the Tables define rather than the one an algorithm produces.

What is owned here #

The construction: reflection positions generated from a stated basis, their differences, the integer elimination that gives a basis of the lattice they generate, the reduction, and the determinant compared with the true one. The sublattice failure, produced by imposing a stated condition on which reflections are recorded, with the factor confirmed rather than asserted — and the five other conditions beside it, three that corrupt the cell and two that do not. The origin error, with the recovered volume computed both ways. The outlier failure, with one foreign reflection and then a second crystal’s worth.

Every one of those numbers is a determinant of an integer matrix compared with another determinant of an integer matrix, so every one of them is exactly right or exactly wrong, and none of them is a tolerance. That is the point of doing the demonstration on generated positions rather than on measured ones: the arithmetic can be made to answer without hedging, and what is then visible is the structure of each failure rather than its size.

Not owned: any real diffraction data, any tolerance-based indexing algorithm, any treatment of errors, twinning or outliers. What is demonstrated is the arithmetic a real procedure approximates, and the ambiguity that survives however well it is approximated.

What the recovered cell is a cell of #

One clarification, because the word cell is doing two jobs and the essay has used both.

What the differences generate is the reciprocal lattice, so the basis that comes out is a basis of reciprocal space. The direct cell is its dual, and computing one from the other is an inversion — a step so mechanical that it is easy to forget it happened, and easy to report a reciprocal cell as though it were a direct one.

The volume relation is the check that catches it. The direct and reciprocal cells have reciprocal volumes, so a recovered cell whose volume is the reciprocal of the expected one has been reported in the wrong space. That is an arithmetic check on a units error, and it is the kind that is worth building in precisely because it is embarrassing rather than subtle.

And the conventional cell is a third thing again: the reduced basis is short, the conventional basis is chosen by the lattice’s symmetry, and neither is what the differences produce. Three cells, two changes of basis, and every published cell is the third of them.

What a real indexing program does instead #

The elimination above is exact and assumes exact data. The procedures a beamline actually runs are built round the same idea and around the fact that every number is measured, and the two standard ones are worth naming because both are recognisably this argument with the arithmetic loosened.

The difference-vector method takes the differences, as here, and puts them in a histogram of three-dimensional space rather than into an elimination. Real lattice vectors recur many times over — every pair of spots separated by the same lattice vector contributes the same difference — so the true short lattice vectors appear as peaks in a cloud of noise, and the basis is read off the three shortest independent peaks. Recurrence replaces exactness: a measured difference is never exactly a lattice vector, and it does not have to be, because a hundred nearly-equal ones make a peak.

The one-dimensional method does the same job by projection. Choose a candidate direction, project every observed vector onto it, and ask whether the projections fall into evenly spaced layers — which is a periodicity in one dimension and is found by a Fourier transform of the projected positions. Sweep the direction over a hemisphere, keep the directions where the periodicity is strong, and the three best independent ones are the reciprocal basis.

Both then feed their candidate basis into a reduction, exactly as the elimination here does, because a basis found this way is short but not canonical. So the shape of the procedure survives the loosening: differences, a lattice, a reduction, a cell — with a peak search in place of an exact identity.

Working from a subset, and checking on the rest #

The outlier problem the essay names has a standard answer, and it is the same answer used wherever a model must be fitted to data containing points that do not belong to it.

Index on a few, then test on the many. Take the strongest few dozen reflections, find a lattice from them, and then ask what fraction of all the observed spots have indices within a tolerance of whole numbers on that lattice. A correct lattice indexes nearly all of them; a lattice generated by an outlier’s differences is much finer, indexes everything trivially, and is caught by comparing cell volumes rather than by the fit.

The fraction indexed is reported, and it is the number a crystallographer looks at first. A fraction near one means one lattice and clean data. A fraction near a half means two lattices — a twin or a second crystal — and the unindexed spots are themselves a data set to be indexed separately. A fraction that is high only with a suspiciously small cell means the finer-lattice failure, and the cure is the volume check.

That is a robustness procedure rather than an exact one, and it is worth being clear that it changes the claim. The elimination here proves that a set of integer vectors generates a particular lattice. The practical version establishes that a lattice accounts for most of what was seen, which is a statement about a data set and carries a fraction with it.

Where the ladder goes next #

Forward into the absences, which are what resolves the ambiguity: reading a space group from what is missing turns a cell into a short list of candidate groups, and the list is short because the absences are predictions rather than observations.

Sideways to the outside of the crystal, where the same problem is solved without any diffraction at all. Assigning indices to faces from measured angles is the same step — a small set of integers chosen so that everything else comes out small — with a refusal in it for the same reason.