Fix two bugs with RGBASM fixed-point math (#1388)

- Fixed-point formulas are implemented using IEEE-754 floating-point
  internally, which could give infinity or NaN values whose conversion
  to fixed-point integer was platform-dependent.
- Formatting fixed-point $8000_0000 (INT32_MIN, -2147483648) was
  not putting the negative sign in front.

test/asm/format-extremes.asm

@@ -0,0 +1,8 @@
+MACRO test
+	def v = \1
+	println "{#09x:v} = {#012o:v} = {#033b:v} = {u:v}U = {+d:v} = {+.16f:v}"
+ENDM
+	test $7fff_ffff ; INT32_MAX
+	test $8000_0000 ; INT32_MIN
+	test $0000_0000 ; UINT32_MIN
+	test $ffff_ffff ; UINT32_MAX