An online calculator gives a clean answer for one decimal calculation and a string of unexpected digits for the next. The difference is not random and it is not a software defect — it is a predictable consequence of how floating-point arithmetic represents numbers in binary. Once you understand the system, the surprises become readable: you can tell at a glance which results to expect exactly and which will carry a tiny hidden error.
The standard behind every browser-based calculator: IEEE 754
Every major programming language, and every browser that runs JavaScript, stores decimal numbers according to the IEEE 754 standard for binary floating-point arithmetic, first published in 1985 by the Institute of Electrical and Electronics Engineers. The current version, IEEE 754-2019, was published in July 2019. Nearly all hardware floating-point units, from laptop processors to smartphones, implement this standard.
Online calculators — including the one above — use the specific variant called double precision (binary64): 64 bits per number, with 53 bits dedicated to significant digits and the rest encoding scale (the exponent). That 53-bit significand gives you approximately 15 to 17 significant decimal digits of precision. For most everyday purposes that is far more than enough — but the format imposes a structural limitation that occasionally surfaces as unexpected output.
The root cause: binary fractions cannot represent all decimals exactly
A computer works in base 2. Every number it stores is built from powers of 2: 1/2, 1/4, 1/8, 1/16…. Any decimal fraction that can be expressed as a finite sum of those powers — such as 0.5 (= 1/2), 0.25 (= 1/4), or 0.125 (= 1/8) — is stored exactly. Every other decimal fraction requires an infinite binary sequence, which the 64-bit format must round to the nearest representable value.
The situation is directly analogous to decimal notation and fractions of 3. You cannot write 1/3 exactly in decimal — it is 0.333333… forever. You round it to some number of places and accept a tiny error. Floating-point arithmetic does the same thing, but in base 2, and the fractions that cause trouble are different: 1/5 (= 0.2), 1/10 (= 0.1), and 3/10 (= 0.3) are all infinite repeating fractions in binary.
How rounding errors accumulate across calculations
A single rounded value introduces an error so small — typically at the 16th or 17th decimal place — that it is invisible in everyday use. The problem grows when multiple rounded values interact across a chain of operations. Each step can either amplify the error or happen to cancel it out, and the outcome is not always predictable from inspection alone.
A few patterns that tend to surface unexpected digits:
- Adding many decimals — summing ten values like 0.1 each gives
0.9999999999999999rather than exactly 1, because the rounding error in each 0.1 accumulates. - Multiplying by numbers close to 1 —
1.1 × 3returns3.3000000000000003in JavaScript; the tiny overestimate of 1.1 in binary is magnified by the multiplication. - Subtraction of nearly equal numbers (catastrophic cancellation) — subtracting two close values can lose many significant digits:
1.0000000000000002 - 1surfaces the gap that would normally be invisible. - Repeated operations — division followed by multiplication by the same value does not always return the original number exactly, because each step introduces its own rounding.
Examples you can test in the calculator above
The calculator above uses JavaScript's native 64-bit floating-point engine — no extra rounding is applied beyond Math.round(result * 1e10) / 1e10, which smooths errors at the tenth decimal place but leaves anything beyond that visible. That makes it a reliable window into real floating-point behavior.
0.1 + 0.2→0.30000000000000004(the most famous case)1.1 × 3→3.30000000000000030.1 × 0.1→0.0100000000000000021 ÷ 3 × 3→1(errors cancel in this case)0.5 + 0.25→0.75(exact — both are powers of two)
Notice that the "clean" results appear precisely where the values involved are exact binary fractions. The unexpected digits appear where they are not.
What the precision limit means in practice
IEEE 754 double precision gives you about 15–17 significant decimal digits before any rounding is forced. For the vast majority of uses that precision is enormous: a measurement accurate to 10 decimal places, a financial figure with 14 significant digits, or a physical constant used in an engineering calculation all fall comfortably within this range.
The situations where the limit matters:
- Financial software — currency calculations accumulate tiny rounding errors across millions of transactions. Production financial systems use fixed-point arithmetic (integers representing whole cents) rather than floating-point to avoid this entirely.
- Scientific simulations — long-running numerical methods (climate models, fluid dynamics) accumulate rounding over millions of steps; researchers use extended precision or dedicated numerical libraries.
- Equality comparisons in code — asking "is this result exactly 0.3?" is almost always wrong; the standard practice is to check whether the result is within a small tolerance of 0.3.
How to work around floating-point surprises
- For everyday calculations — the error is at the 16th decimal place and has no practical impact. Trust the result.
- For financial work on a general calculator — round the final result to the required number of decimal places (e.g., 2 for currency) and treat only the rounded value as meaningful.
- To check if an error is floating-point — enter the same expression in a fresh session and compare; floating-point errors are deterministic, so the answer will be identical every time for the same expression.
- To identify safe inputs — values that are exact powers of 2 fractions (0.5, 0.25, 0.125, 0.0625…) or integers are always stored exactly. If you can reframe your problem in those terms, you eliminate the source of error.
See it for yourself: enter1.1 × 3in the calculator above, then0.5 × 3. The first gives a tiny tail of unexpected digits; the second gives a clean 1.5. That contrast captures the whole story of floating-point: binary-exact inputs, binary-exact answers; non-binary-exact inputs, rounding at the precision boundary.