Handle non-canonical extFloat80 encodings in all extF80 operations

[johnwbyrd]

Summary

The 80-bit extended double format stores the integer bit (J bit) explicitly in the significand, making it possible to construct encodings where the J bit is inconsistent with the exponent. These non-canonical encodings -- unnormals, pseudo-denormals, pseudo-infinities, and pseudo-NaNs -- are accepted by hardware but were silently mishandled by SoftFloat, producing wrong results.

This PR adds four canonicalization functions that rewrite non-canonical encodings to their canonical equivalents, preserving the represented value:

• softfloat_canonicalizeExtF80 (FAST_INT64, unpacked exp/sig)
• softfloat_canonicalizeExtF80UI (FAST_INT64, packed signExp/sig)
• softfloat_canonicalizeExtF80M (non-FAST_INT64, unpacked exp/sig)
• softfloat_canonicalizeExtF80MUI (non-FAST_INT64, packed signExp/sig)

The appropriate function is called at the top of every extF80 operation so that all subsequent logic sees only canonical inputs. This fixes wrong results across all 37 affected functions on both code paths:

• 12 FAST_INT64 conversions (to f16/f32/f64/f128/i32/i64/ui32/ui64)
• 4 non-FAST_INT64 conversions (to i32/i64/ui32/ui64)
• 6 FAST_INT64 comparisons (eq, lt, le, and signaling/quiet variants)
• 6 non-FAST_INT64 comparisons
• FAST_INT64 add/sub (s_addMagsExtF80, s_subMagsExtF80)
• non-FAST_INT64 add/sub (s_addExtF80M)
• FAST_INT64 mul, sqrt, rem
• non-FAST_INT64 rem
• non-FAST_INT64 three-way compare (s_compareNonnormExtF80M)

Additional fixes beyond canonicalization

• extF80_mul: the unified infArg path tested expB | sigB to detect infinity times zero, but an unnormal has nonzero sigB even when its value is zero. Split into separate invalid and infinity labels with direct significand checks.

• extF80_sqrt: sqrt of a pseudo-infinity returned the raw input instead of a canonical infinity.

• extF80_rem: a pseudo-denormal divisor confused the normalization logic because sigB already had J=1 but expB was 0.

• s_addMagsExtF80: adding two pseudo-denormals could overflow the significand without being caught; infinity results propagated the raw (possibly non-canonical) significand.

• s_subMagsExtF80: equal-magnitude subtraction after alignment fell through to the wrong label instead of producing zero; infinity results propagated non-canonical significands.

• s_addExtF80M: replaced pointer-swapping with local variable swapping so that canonicalized values are used throughout.

Other changes

• Moved struct exp32_sig64 and softfloat_normSubnormalExtF80Sig above the SOFTFLOAT_FAST_INT64 guard in internals.h so both code paths can use them.
• Updated all 10 platform Makefiles with the new object files.
• This PR supersedes fix(extF80): correct add/sub for valid inputs with J-bit clear #38, fix(extF80): extF80_mul returns infinity for Inf times zero #39, and fix(extF80): rem and sqrt mishandle pseudo-infinity inputs #40, which addressed subsets of these issues individually.

Test plan

• TestFloat level-1 on FAST_INT64 (Linux-x86_64-GCC): 0 errors across all extF80 operations
• TestFloat level-1 on non-FAST_INT64 (Linux-386-GCC): 0 errors across all extF80 operations

[johnwbyrd]

See also ucb-bar/berkeley-testfloat-3#21, which extends TestFloat's extF80 test generator to cover all valid J-bit encodings, exercising the non-canonical input paths fixed here.

The above patchset may be best understood as consequences of this patch to TestFloat. While the patchset may seem far-reaching, all the changes were necessitated as a result of the improved TestFloat coverage.

[aswaterman]

Just making sure you've seen Sec. 4.4 and Sec. 10 of the SoftFloat docs: https://www.jhauser.us/arithmetic/SoftFloat-3/doc/SoftFloat.html

It seems that unspecified behavior when processing non-canonical F80 values is expected for this version of SoftFloat, but that the author, @jhauser-ucberkeley, agrees that making this behavior well-defined is a reasonable goal.

If there is semantic concordance across the range of modern x86 implementations, I agree it makes sense to change SoftFloat to reflect it. But I think we should wait for John Hauser to have the chance to opine on the general strategy, even if he isn't able to provide a code review.

[johnwbyrd]

I will save you some time then. These operations are well defined on the Intel 8087 and perhaps the '287; however, later processors in the line attempt to trap on unnormal math. However, the current SoftFloat does neither of these things; it reports fully incorrect answers on unambiguous inputs, which I have attempted to address in this patch.