Failing math::complex tests and NAN sign

Message ID
DKIM signature
Download raw message
I did some investigation of this issue and these are my findings:

One of the math::complex tests checks that
calike(math::complex::cosc128((math::NAN, 0f64)), (math::NAN, -0f64))
is true, which is failing on glibc.

The definition of cosc128 returns

	(math::NAN, -0f64 * math::copysignf64(0f64, math::NAN))

for this input, so we are checking that the sign of -0f64 *
math::copysignf64(0f64, math::NAN) is the same as the sign as -0f64.
Simplifying further, we are essentially checking that the sign bit
of math::NAN is unset. This ends up being true on musl, but not on
glibc, resulting in the test failure.

Why is this the case? math::NAN is defined as 0.0 / 0.0. When used
as a constant expression, this is compiled by gcc and clang as
nan(0), but when computed at runtime, we get -nan(0). Since this
is evaluated at runtime in harec's eval_expr, we get the latter,
which both musl and glibc format as "-nan". QBE then parses this
back into a float using strtod. However, strtod("-nan") is nan(0)
on musl, but -nan(0) on glibc. This means that the sign bit of
math::NAN ends up being set under glibc, causing the test to fail.

So, where's the bug? The documentation of math::NAN says

> The floating point value representing Not a Number, i.e. an
> undefined or unrepresentable value.

This is a bit vague and implies that there is only one NaN value,
which is not true with IEEE 754. I can see two potential clarifications:

1. The particular NaN value is implementation defined, and consequently
   so is the value of its sign bit (s/The/A/).

2. The NaN value is the unique value with payload 0 and sign bit

If 1 is chosen, then the bug is in the test (or the implementation
of cosc128). The sign bit of math::NAN may or may not be set, so
the imaginary part of cosc((nan, 0)) might be +0 or -0.

If 2 is chosen, then the bug is in the implementation of harec.
IEEE 754 says that the result of division(0, 0) "shall be a quiet
NaN that should provide some diagnostic information", but doesn't
say anything about the particular NaN. Both glibc and musl roundtrip
this NaN value with printf and strtod, so I don't think anything
would need to change about how harec emits floats or how qbe parses

Anyway, I'm not sure what the best path forward is, just figured
I'd dump some info here so we can find the right solution.
Reply to thread Export thread (mbox)