The lead taken out solves what no threshold could

The solver that knows no symmetry is a short procedure with a strange power. Start from the measured amplitudes and random phases, make a map, reverse the sign of every point of the map below a small threshold, transform back, keep the new phases and put the measured amplitudes on again, and repeat. For a structure of separated atoms the map settles on the structure. The heavy atom the flipping solver cannot see past then gave it cells of unequal atoms and found something the theory of unequal atoms had not predicted. Sulphur among carbon helped it. A single lead atom among nineteen carbons stopped it completely: no success in forty starts at any threshold, though the map made from the true phases passed every test.

The reason given there was the threshold’s unit. The solver flips everything below $\delta$ standard deviations of the map, and in the lead cell the lead owns the standard deviation, so a carbon peak stands only about one and a half of them above the mean and no setting of $\delta$ separates carbons from noise. Two repairs followed from that diagnosis, and the essay named both without running either. One sets the threshold from the chemistry: the composition says how high a carbon’s peak should be, so flip below a fraction of that. The other takes the lead out of the map before each flip, once its position is known, so that what is left belongs to the carbons. This essay runs both, on the same four cells, arrangements and starts.

Four rules #

The cells are the earlier ones: twenty carbons; fifteen carbons and five sulphurs; nineteen carbons and one sulphur; nineteen carbons and one lead, each atom given a Gaussian fall-off so that a true map passes the success test. Every rate is pooled over five random arrangements of the atoms and eight random starts on each, forty runs, at every setting of the rule’s one parameter, and the best setting’s rate is reported beside the whole curve.

The original rule flips below $\delta$ standard deviations of the current map. The composition, absolute flips below $\kappa$ times the height a carbon’s peak should have, which a weight and the reflection list give before any phasing: a point atom of weight $z$ with the data’s fall-off puts $z \sum_h e^{-B|h|^2}$ at its centre, divided by the grid. The composition, from the mean flips below the map’s mean plus that same height. The heavy atom taken out subtracts the known heavy atoms’ calculated density from the map, flips what is left by the original rule, and adds the heavy atoms back before the amplitudes go on. On the cell of carbon alone the last rule subtracts nothing, and the computation confirms that it then reproduces the original solver map for map, start for start, to twelve decimals.

The heavy atom’s position is taken as known. That is what the heavy-atom method supplies, from the strongest peak of the map that needs no phases, and for one lead among nineteen carbons the Patterson’s lead–lead vector is unmissable. Taking it as given is the honest form of the repair: it combines the two methods rather than asking flipping to find what an older method finds trivially.

What each rule does #

The figure at the head of this essay is the census. On the lead cell, only one rule solves anything. The original rule, nought of forty. The composition’s threshold measured from zero, nought at every $\kappa$. Measured from the map’s mean, one of forty at its best setting. With the lead taken out, twenty-two of forty, fifty-five per cent.

The sulphur cells say the same thing more loudly. With a quarter sulphur the original rule succeeded twenty-four times in forty; with the sulphurs taken out it succeeds forty times in forty, at every one of five thresholds. With one sulphur among twenty, twenty-two becomes thirty-eight. Sulphur was already a help to the original solver, the earlier essay’s surprise, and removing it helps more. On the cell of carbon alone, where there is nothing to take out, the rate is the original rule’s thirty-three per cent, as it must be.

The composition’s threshold, measured from the mean, is about as good as the original rule on every light cell: thirty-five per cent on carbon against thirty-three, sixty-three on a quarter sulphur against sixty, forty-five on one sulphur against fifty-five. It is a different way of placing a threshold that ends up in the same place. Measured from zero it is worse everywhere, and the reason is instructive.

An absolute threshold drifts #

The solver never fixes the reflection at the origin, the one that carries the map’s average. The data do not contain it, and the procedure leaves it as each transform gives it, so the map’s mean wanders from cycle to cycle. A threshold measured in standard deviations, or from the mean, moves with it. A threshold set at an absolute height does not, and the share of the map it flips drifts accordingly: on the lead cell from about sixty per cent at the first cycle to a third by the end, as the figure shows. A rule that flips a third of the map is flipping too little of the empty space to push the solution towards sparseness, and it fails for that reason before the lead enters into it.

Anchored to the mean, the composition’s threshold holds a steady share, and then the lead does enter into it. The share it holds is no different from the original rule’s, around eighty per cent of the map, and the map being thresholded still contains the lead’s density, with the ripples a heavy atom truncated at the data’s resolution throws across the cell. A carbon’s expected height is the right number for what a carbon looks like; it is not a number that lets a threshold see a carbon through a lead atom’s ripples.

The window that opens #

The curves show what the subtraction changes. The original solver works in a narrow window of thresholds, around one standard deviation, and the earlier essay found the window’s position moving from cell to cell. With the sulphurs taken out, the window for the sulphur cells is the whole range tried, from 0.6 to 1.4 standard deviations, forty successes in forty at every setting for the quarter-sulphur cell. With the lead taken out, the lead cell succeeds best at the lowest threshold, 0.6, and falls away above it. The composition’s window, on the right, is narrow on every light cell and absent on lead.

A wide window is worth as much as a high rate, because the threshold is the one thing a user has to choose without knowing the answer. A solver that succeeds at every threshold in a range needs no tuning, and a solver that succeeds only in a narrow band needs the band found, which is a search over the very quantity the method was supposed to make unnecessary.

The maps make the difference visible. Under the original rule and under the composition’s threshold, the final map from one start has a dominant lead peak and a scatter of lesser peaks that do not fall on the carbons at any shift or hand. With the lead taken out, the same start ends with its peaks on the carbon positions. The lead is in the right place in every case, since every rule keeps the strongest feature of the data; the difference is whether the light atoms are found around it.

How sure forty starts are #

A rate over forty runs is a count, and a count has an uncertainty the rates above do not print. Twenty-two successes in forty is fifty-five per cent with a binomial standard error of about eight points, so the lead cell’s rate under subtraction lies between roughly forty and seventy per cent with ordinary confidence. Nought in forty under the original rule is not proof that the rule never works on that cell; it says the rate is below about seven per cent at the same confidence. The comparison that matters survives both: the subtraction rule’s rate on the lead cell is far outside anything the original rule’s count allows.

The comparisons on the light cells are closer. Thirty-five per cent for the composition’s threshold on carbon against thirty-three for the original rule is one success in forty, well inside the noise, and the honest reading is that the two rules are indistinguishable there. Sixty-three against sixty on the quarter-sulphur cell is the same. Forty of forty against twenty-four of forty is not: a difference of forty points with standard errors near eight is the second clearest result in the census after the lead cell’s.

The runs are also not independent in one respect. The five arrangements are shared across rules, so a rule that does well on an easy arrangement does well for every rule on it, and a comparison between rules is fairer than a comparison of either rule against an absolute standard. That is the reason the census was built on the earlier essay’s arrangements and starts rather than new ones.

Pinning the origin #

There is a second thing the subtraction does that a threshold cannot. A map made from amplitudes and random phases has no fixed origin: shifting every atom by the same vector changes every phase and no amplitude, which is why three phases that do not move built the subject’s invariants from sums of phases that a shift cannot change. The original solver converges on the structure at whatever origin the random start happens to favour, and the success test has to search over shifts to recognise it.

Subtracting the lead at a stated position fixes the origin to that statement. The map with the lead taken out is a map relative to the lead, and in the run drawn above the solved map’s carbons sit at their true coordinates with no shift at all. That is a small convenience for the test and a larger one for the method. A solver that is told where one atom is has been told the origin, and a large part of what makes the phase problem hard in the general position is that nothing is.

Why subtraction works and the threshold does not #

The earlier diagnosis said the threshold was in the wrong units. The census says the diagnosis was half right. The threshold was in the wrong units, and putting it in the right ones, a carbon’s height measured from the map’s mean, does not help, because the units were not what was wrong with the map. What was wrong is that the map contains the lead.

A heavy atom is not only a tall peak. Truncated at the data’s resolution it is a tall peak with ripples, and the ripples are larger than a carbon. Any rule that thresholds a map containing the lead thresholds those ripples too, flipping some of them and keeping others, and the solver spends its cycles shaping the lead’s ripples rather than finding the carbons. Taking the lead out removes the peak and its ripples together, since its calculated density is computed from the same reflections the map is. What is left is a map of carbons with noise, and on that the original rule, with its threshold in standard deviations of what is left, works as it does on a light structure.

That is also why the sulphur cells improve. Five sulphurs among fifteen carbons are not heavy enough to stop the solver, but their ripples are a fraction of a carbon’s height, and taking them out leaves a cleaner map than the solver ever had. The repair that works is not a better threshold. It is the heavy-atom method and charge flipping combined, each doing the part it does well: the Patterson finds the heavy atoms, as solving from the vector set found atoms from vectors alone, which are easy, and flipping finds the light atoms, given a map in which they are the only thing to find.

What the census must refuse #

The estimate of a carbon’s height, made from its weight and the reflection list, agrees with the true map’s median carbon to within a quarter, the difference being that the map is sampled on a grid and a carbon rarely sits on a grid point; the threshold’s failure is not an error in that number. The equivalence of the subtraction rule with the original one on carbon alone is checked map for map, so that any difference in rates on the other cells is the subtraction’s doing. And the claim that motivated the first repair is refused by the census: no setting of the composition’s threshold, from zero or from the mean, solves the lead cell more than once in forty.

Where the census stops #

The heavy atom’s position is given, exactly. In practice it comes from a Patterson map or a heavy-atom search with a small error, and a lead atom subtracted a little off its true position leaves a residual peak and anti-peak whose size grows with the error. How much positional error the subtraction tolerates before it fails, and whether the lead’s position can be refined during the flipping cycles rather than fixed, are the questions a practical method would need answered.

The cells are small, two-dimensional and in the group with no symmetry. Twenty atoms on a grid of thirty-two by thirty-two, one arrangement at a time, with Gaussian atoms. Real structures have more atoms, three dimensions, symmetry and noise, and charge flipping’s behaviour on those is known from practice to be broadly similar but not identical. The rates here are comparisons between rules on identical problems, which is what they are good for.

The success test is a test on peaks. A run counts as solved when the twenty strongest peaks of its final map lie, on average, within about a grid step of the twenty atoms after the best shift and choice of hand. A run that finds most of the carbons and misplaces three is a failure by that test and would be a useful start for a refinement in practice, so every rate here is a rate of complete solutions; partial ones would raise all of them, the lead cell’s under the original rule perhaps least, since its maps did not resemble the structure at all.

And one heavy atom is the easy case. A cell with several heavy atoms that do not dominate individually, a metal cluster or a selenium substructure, needs the substructure found first, which is a phasing problem of its own, and a map of the atoms that break the law is one route to it.

Who combined what #

Charge flipping was introduced by Oszlányi and Sütő in 2004, and the observation that it struggles when a few atoms dominate the scattering was made early in its use; variants of the flipping rule have been proposed since. The heavy-atom method itself is older than direct methods, going back to the structure determinations of the 1930s, Robertson’s phthalocyanines among them. What this census adds is the comparison on identical problems: the same cells, arrangements and starts, under a threshold that knows the composition and under a map that has lost its heavy atom, so that which repair works is a measurement and not an argument.

Still open: how far off the lead may be #

The subtraction assumed the lead’s position exact. Shift it by a fraction of a grid step and the subtracted density no longer cancels the lead’s own: a positive residual on one side and a negative one on the other, of a height proportional to the shift times the lead’s weight. For a weight of eighty-two that residual exceeds a carbon’s height at a shift of a few hundredths of the cell. The census here could be rerun with the subtracted lead displaced by known amounts in random directions, and the success rate measured against the displacement would say how good a Patterson’s estimate of the heavy atom has to be before taking it out helps rather than hurts. A second run, refining the lead’s position from the map at each cycle, would say whether flipping can correct a rough estimate on its own. Neither has been made.