I need to start by unwrapping this a bit so I can see what’s going on: SELECT 199.96 – ( 0.0 * FLOOR( CAST(1.0 AS DECIMAL(19, 4)) * CAST(199.96 AS DECIMAL(19, 4)) ) ) Now let’s see exactly what types SQLÂ Server is using for each side of the subtraction operation: SELECT SQL_VARIANT_PROPERTY (199.96 ,’BaseType’), SQL_VARIANT_PROPERTY … Read more
For a given IEEE-754 floating point number X, if 2^E <= abs(X) < 2^(E+1) then the distance from X to the next largest representable floating point number (epsilon) is: epsilon = 2^(E-52) % For a 64-bit float (double precision) epsilon = 2^(E-23) % For a 32-bit float (single precision) epsilon = 2^(E-10) % For a … Read more
Instead of calculating sine as a function of time, maintain a sine/cosine pair and advance it through complex number multiplication. This doesn’t require any trigonometric functions or lookup tables; only four multiplies and an occasional re-normalization: static const double a = 2 * M_PI * 280 * 30e-6; static const double dx = cos(a); static … Read more
Cast the operands to floats: float ans = (float)a / (float)b;
Because floating point arithmetic != real number arithmetic. An illustration of the difference due to imprecision is, for some floats a and b, (a+b)-b != a. This applies to any language using floats. Since floating point are binary numbers with finite precision, there’s a finite amount of representable numbers, which leads accuracy problems and surprises … Read more
Java doubles are in IEEE-754 format, therefore they have a 52-bit fraction; between any two adjacent powers of two (inclusive of one and exclusive of the next one), there will therefore be 2 to the 52th power different doubles (i.e., 4503599627370496 of them). For example, that’s the number of distinct doubles between 0.5 included and … Read more
A BigDecimal is defined by two values: an arbitrary precision integer and a 32-bit integer scale. The value of the BigDecimal is defined to be . Precision: The precision is the number of digits in the unscaled value. For instance, for the number 123.45, the precision returned is 5. So, precision indicates the length of … Read more
Why would DateTime.Now be made less precise than what most CPU clocks could handle? A good clock should be both precise and accurate; those are different. As the old joke goes, a stopped clock is exactly accurate twice a day, a clock a minute slow is never accurate at any time. But the clock a … Read more
It is safe to expect that the comparison will return true if and only if the double variable has a value of exactly 0.0 (which in your original code snippet is, of course, the case). This is consistent with the semantics of the == operator. a == b means “a is equal to b“. It … Read more
Your instinct is basically right, sorting in ascending order (of magnitude) usually improves things somewhat. Consider the case where we’re adding single-precision (32 bit) floats, and there are 1 billion values equal to 1 / (1 billion), and one value equal to 1. If the 1 comes first, then the sum will come to 1, … Read more