Somewhere in your inspection software there is a dropdown. It might say Gauss and Chebyshev; it might say least squares and min-max; it might be buried in a default nobody has opened in years. That dropdown decides what the word "diameter" means on your report. This note explains what the two rules actually do, using arithmetic you can check by hand, and shows why the same scan can honestly produce two different numbers.
Contents
A scanner or a CMM never measures "a circle." It measures points — hundreds or millions of individual locations on the surface of your part. Those points are never perfectly arranged: the part has real form error left by machining, and the instrument adds a little noise of its own. So the points scatter.
To get from that scatter to a single number — a diameter, a flatness, a position — the software lays an ideal shape over the measured points: a perfect circle, a perfect plane. The ideal shape has adjustable settings: where it sits, how it is tilted, how big it is. The software's job is to choose those settings so that the ideal shape represents the points as well as possible. That choice is called fitting (the standards call it association), and "as well as possible" is not one idea. It is a rule you pick. Different rules, different settings, different number — from the same points.
To talk about any fitting rule, we need exactly one concept: the gap between a measured point and the candidate ideal shape. Metrologists call it the residual.
| i | a label saying which measured point we mean: point 1, point 2, point 3, and so on. |
| di | the gap for point i: how far that one point sits from the candidate shape. It carries a sign — positive on one side of the surface (say, outside), negative on the other (inside). A point exactly on the shape has a gap of zero. |
| θ | (the Greek letter theta) shorthand for the candidate shape's settings, all bundled together — where it sits, how it is tilted, how big it is. Move or resize the candidate, and every gap changes; that is why di "depends on" θ. |
In words: slide an imaginary perfect shape over your cloud of measured points. For every point, the residual is that point's own private gap to the shape — signed, so the two sides of the surface can be told apart.
Every fitting rule in every inspection package is an answer to one question: given all these gaps, which candidate shape is "best"? The two answers that matter were given by two mathematicians in the nineteenth century, and they disagree in an instructive way.
The first rule says: a good fit is one where the gaps are small overall. Not the worst gap — the whole population of gaps, taken together. To turn that into something a computer can do, the rule squares every gap and adds them up. The candidate shape with the smallest total wins.
| S | the total score of a candidate shape: all the squared gaps, added up. Smaller is better under this rule. |
| ∑ | (the Greek letter sigma) an instruction, not a quantity: "add up the thing that follows, once for each point." The small i = 1 underneath and n on top say: start at point 1, stop at point n. |
| n | how many points were measured in total. |
| di(θ)² | each point's gap (Equation 1), multiplied by itself. Squaring does two useful things: it makes every contribution positive, so gaps on opposite sides of the surface cannot cancel each other out; and it punishes big gaps much more than small ones — a gap of 2 counts four times as much as a gap of 1. |
| θ | the candidate shape's settings, as in Equation 1 — position, tilt, size. |
In words: try candidate shapes. For each one, measure every gap, square each gap, and add them all up. The fit is the candidate with the smallest total. Every single point gets a vote.
This is the famous method of least squares. Adrien-Marie Legendre published it in 1805; Carl Friedrich Gauss — who said he had been using it for years in the orbit work that famously recovered the asteroid Ceres — published his own account in 1809, together with the reason it works so well: when measurement errors scatter in the familiar bell-curve way around the truth, the least-squares answer is the most probable one. That bell-shaped curve now carries Gauss's name, and so does the fit: in ZEISS INSPECT and its GOM predecessors, the fitting method is labelled Gaussian best fit.
Formulas hide how simple this is, so let's shrink the problem until the arithmetic fits on a napkin. Suppose you measure the thickness of one ground shim at five spots and get, in millimetres:
10.02 10.05 9.98 10.06 10.04
Someone asks: "so what is the thickness?" You need one number to represent five. That, believe it or not, is a fit — the simplest one there is. The "candidate shape" is just a single value T, and each gap is a measurement minus T. Apply the Gaussian rule — choose T to make the sum of squared gaps smallest — and something familiar happens: the answer is the ordinary average.
| x1, x2, … | the individual measurements — here, the five thickness readings. |
| x̄ | ("x-bar") the average of the measurements. It is exactly the value that makes the sum of squared gaps as small as it can be — the Gaussian fit of a single number. |
| n | how many measurements — here, 5. |
In words: the everyday average you have computed all your life is not just a convention. It is the least-squares fit in its smallest possible form. Every larger Gaussian fit — a line, a plane, a circle through a million scanned points — is this same idea with more adjustable settings.
This connection also explains the Gaussian rule's superpower: averaging cancels noise. Random instrument error pushes some readings up and some down; in the average, those pushes offset. The more points you take, the harder they cancel:
| σ | (the Greek letter sigma, lower-case this time) the typical size of the random wobble in a single reading — the instrument's noise level. |
| √n | the square root of the number of points: the number which, multiplied by itself, gives n. For 100 points it is 10; for 10,000 points it is 100. |
In words: average 100 noisy readings and the result wobbles ten times less than any single reading; average 10,000 and it wobbles a hundred times less. One honest condition attaches: this works for random noise, where each reading's wobble is unrelated to the next. A shared, systematic error — a warm part, a miscalibrated sensor — does not shrink no matter how many points you take. Still, this cancellation is why Gaussian fits through dense scan data are so beautifully repeatable — and it is a property the second rule does not share.
Now the rival philosophy. It says: a good fit is one whose worst gap is small. Not the total, not the average — the single biggest miss. Choose the candidate shape that makes its worst offender as mild as possible.
| W | the score of a candidate shape under this rule: the size of its single worst gap. |
| maxi | an instruction: "look through all the points and take the largest value you find." The little i says the search runs over the points. |
| |di| | the size of point i's gap with the sign stripped off (the vertical bars mean "absolute value"). A gap of −0.04 and a gap of +0.04 are equally bad here: 0.04 either way. |
| θ | the candidate shape's settings, as in Equation 1 — position, tilt, size. |
In words: try candidate shapes. For each one, find its single worst gap. The fit is the candidate whose worst gap is smallest. Most points get no say at all — only the extremes matter. The name comes from what it does: it minimizes a maximum, hence minimax.
The name on the dropdown honours Pafnuty Chebyshev, the nineteenth-century Russian mathematician who built the theory of approximating one thing by another so that the worst error is as small as possible. His theory contains the detail that makes this rule feel so different in practice: the best minimax answer is always pinned in place by a small handful of extreme points that all achieve that same worst gap at once. Everything else is a spectator.
If that sounds abstract, you already know it physically. A four-legged table wobbles because of its highest and lowest feet — the average length of its legs is beside the point. A doorway is tall enough for a crowd only if it clears the tallest person. A shaft goes into a bore only if the widest part of the shaft clears the narrowest part of the bore. Assembly is a worst-case business, and Chebyshev is the worst-case fit.
Back to the five thickness readings. The Gaussian answer was the average, 10.030. What is the Chebyshev answer — the single value whose worst disagreement with the five readings is smallest? A moment's thought gives it: park exactly halfway between the highest and lowest reading. Any other position moves you further from one of them.
| TC | the Chebyshev fit of the thickness: the value halfway between the largest and smallest reading (the "midrange"). |
In words: the Chebyshev fit of a set of numbers ignores all but two of them. The three middle readings — 10.02, 10.04, 10.05 — could move anywhere between the extremes and the answer would not change by a micron. That indifference is not a flaw; it is the rule's entire personality.
So the same five readings give 10.030 mm under one rule and 10.020 mm under the other — a 10 µm disagreement, from arithmetic you can check on paper. Neither is a mistake. Here is the full bookkeeping; note that each answer wins its own column and loses the other's:
| spot | measured (mm) | gap from 10.030 (Gaussian fit) | gap from 10.020 (Chebyshev fit) |
|---|---|---|---|
| 1 | 10.02 | −0.010 | 0.000 |
| 2 | 10.05 | +0.020 | +0.030 |
| 3 | 9.98 | −0.050 | −0.040 |
| 4 | 10.06 | +0.030 | +0.040 |
| 5 | 10.04 | +0.010 | +0.020 |
| sum of squared gaps (Gaussian score S) | 0.0040 ✓ smaller | 0.0045 | |
| worst gap (Chebyshev score W) | 0.050 | 0.040 ✓ smaller | |
And notice why they disagree: the readings are lopsided — a cluster near the top (10.04, 10.05, 10.06) and a single low value at 9.98. The Gaussian rule lets the high cluster pull the answer up. The Chebyshev rule gives the cluster no vote: the one low reading balances the highest one all by itself, because only the two extremes speak. Whether that is a feature or a hazard depends entirely on the question you are asking — which is the whole point of this note.
Everything so far fitted a single number. Real inspection fits lines, planes, circles and cylinders — the only change is that the candidate shape has more settings θ to adjust, and the "gap" is measured square-on to a surface instead of along a ruler. The two rules keep their personalities exactly.
Below, the same 26 measured points from a nominally straight profile are fitted both ways — computed, not sketched. Top: the Gaussian line. Every point contributes a residual (the thin strokes), the crowd votes, noise cancels. Bottom: the Chebyshev result, drawn the way the standards think about it — as the narrowest pair of parallel lines that contain every point, the minimum zone. Three circled points touch the band; they alone decided it, exactly as Chebyshev's theory promises.
For a circle the residual has a pleasingly concrete form — it is the only equation in this note with actual coordinates in it:
| di | point i's gap to the candidate circle — the same residual as Equation 1, now written out in coordinates. |
| xi, yi | the coordinates of measured point i — where it sits left-right and up-down. |
| a, b | the coordinates of the candidate circle's centre. Together with R, these are the "settings" θ for a circle. |
| R | the candidate circle's radius. |
| √ ⋯ | the square root of everything under the bar. The expression under it is Pythagoras' theorem: the straight-line distance from the point to the candidate centre. |
In words: measure how far the point is from the circle's centre, then subtract the radius. Zero means the point lies exactly on the circle; positive means outside it; negative means inside. Feed these gaps into Equation 2 and you get the Gauss circle; feed them into Equation 5 and you get the Chebyshev circle.
Circles also reveal two important cousins of the Chebyshev fit, and your software offers both. Instead of treating inside and outside gaps even-handedly, you can push the worst-case logic to one side only:
These are the fits that answer assembly questions, because assembly happens at the extremes: the pin meets the hole's tightest cross-section, not its average one. The ISO size standard, ISO 14405-1, treats all of these as first-class citizens and even gives them letter codes that can be written on a drawing next to a dimension: GG for the Gaussian size, GX for maximum inscribed, GN for minimum circumscribed, and GC for the minimax (Chebyshev) criterion. A drawing that says nothing usually inherits a national default — and your software, left alone, almost always reports the Gaussian one.
Form tolerances — flatness, roundness, straightness — are a place where the standards have quietly already chosen a side. A flatness callout of 0.05 on a drawing means: the entire real surface must fit between two parallel planes 0.05 apart. It is a containment requirement. Nothing about averages appears anywhere in it.
So the honest way to report flatness is to find the narrowest pair of parallel planes that still contains every measured point, and report their separation. That narrowest containing band is called the minimum zone, and finding it is exactly a Chebyshev problem:
| t | the form error: the width of the narrowest band (pair of parallel planes, pair of concentric circles, …) that contains every measured point. |
| maxi di | the farthest any point sticks out on the positive side of the candidate reference shape. |
| mini di | the farthest any point sticks out on the negative side (this value is negative, so subtracting it adds the two protrusions together). |
| minθ [ ⋯ ] | an instruction: among all possible settings θ of the reference shape (its position and tilt, as in Equation 1), take the one that makes the bracketed width smallest. |
In words: squeeze the surface between two parallel ideal shapes, tilting and shifting them until the sandwich is as thin as it can possibly be while still containing every point. The thickness of that thinnest sandwich is the form error. It is a guarantee about the whole surface — not a statement about its average.
The ISO form standards (ISO 12181 for roundness, ISO 12780 for straightness, ISO 12781 for flatness) allow the deviation to be reported from several reference fits, least-squares included — but the minimum-zone reference is the one that matches the tolerance definition, and it always yields the smallest, most faithful value. A least-squares flatness is computed about the Gaussian plane, which was chosen to please the average, not to make the band thin; the spread about it is therefore always at least as wide as the minimum zone. In our 26-point example above, the minimum zone came out 16% narrower. A part can fail "flatness" computed the lazy way and genuinely conform to its drawing — and vice versa is impossible, when both values are computed on the same measured points with the same filter. That asymmetry is worth knowing before you scrap anything.
By now Chebyshev may sound like the righteous choice — it matches the drawing, it matches assembly, it makes guarantees. Here is the price. Remember Equation 4: averaging cancels noise, and the Gaussian fit is an average at heart. The Chebyshev fit is the opposite of an average: it hands the entire verdict to the most extreme points it can find. And what tends to be extreme, in real scan data?
To the Gaussian rule, one bad point in a hundred thousand is one vote among a hundred thousand — it moves the answer by roughly nothing. To the Chebyshev rule, one bad point is the answer, if it happens to be the extreme. A single speck of dust becomes your reported flatness. This is not a software bug; it is the rule doing precisely what it promises — reporting the worst case — applied to data whose worst case is not the part.
Practitioners manage this with two disciplines. First, filtering: form evaluation is supposed to happen at a declared spatial scale (the ISO 16610 family of standards exists for exactly this), so that single-point spikes are smoothed away before any fitting happens. Second, outlier handling: identifying and excluding points that are demonstrably not the surface. Both matter far more for Chebyshev-family fits than for Gaussian ones, and both are choices that should be recorded with the result — because changing either changes the number.
| You want to know… | The question really is… | Rule that answers it |
|---|---|---|
| Is my process drifting? What does this feature typically measure? | a question about the average surface | Gaussian |
| Will this pin enter this hole? Will these parts assemble? | a worst-case, one-sided question | maximum inscribed / minimum circumscribed (Chebyshev's one-sided cousins) |
| Does the surface conform to its flatness / roundness / straightness callout? | a containment question — the tolerance zone is a band | Chebyshev (minimum zone) |
| A stable reference to align repeated measurements against | repeatability matters more than worst case | Gaussian |
| Will it seal? Will it seat without rocking? | the extremes touch first | Chebyshev family — with disciplined filtering |
Datums deserve a special mention. A physical datum — a granite table, a locating pin — touches the part at its high points, not at its average. The standards' association rules for datum features reflect that contacting behaviour, which is another place worst-case fitting quietly enters everyday inspection, whether or not anyone opened the dropdown.
Manually, a metrologist who sees a suspicious number pokes at it: re-fits, excludes a point, notices the glint. Automation removes the poking. Whatever criterion, filter, and outlier policy are frozen into the routine will be applied identically, silently, to every part, forever. That is the entire value of automation — and its entire risk, because the choice happens once, usually at the moment someone writes the check or records the script, and often by default rather than decision.
Three habits keep it honest:
When I automate an inspection method, this table of choices — criterion, filter, outlier rule, per feature — is part of the deliverable, reviewed and signed by the customer's metrology owner before anything runs unattended. Not because ceremony is fun, but because the alternative is ten thousand reports carrying a number nobody can define.
Neither — the question assumes they estimate the same thing, and they don't. Ask instead: which question does my drawing ask? A containment callout asks a Chebyshev question. A process-control chart asks a Gaussian one. Accuracy is a property of the match between question and rule, not of the rule alone.
It is a defined, reproducible quantity, and for tracking a stable process it is arguably the better-behaved one. But it is not the quantity the roundness callout defines, and it is never smaller than the minimum-zone value. Know which one your acceptance decision uses, and be consistent about it.
No. On unfiltered scan data, Chebyshev-family fits chase noise and artifacts, and your reported numbers will get worse and jumpier without the parts changing at all. Adopt it where the callout demands it, with the filtering discipline it requires.
On well-machined surfaces with balanced form error and clean data, the average and the extremes tell a similar story, and the rules converge to within instrument noise. They diverge exactly when the surface is lopsided — one deep scratch, thermal bow, a lobed bore — which is to say, exactly on the parts where the distinction pays for itself.
It is one of the two usual reasons; the other is filtering. Before suspecting anyone's math, check that both systems used the same association rule and the same filter. Most cross-software "discrepancies" I have investigated dissolve at that point.
If your automated inspection reports numbers whose fitting rule nobody can name, that is a solvable problem: hello@metrologymaven.io