Uploaded image for project: 'Daffodil'
  1. Daffodil
  2. DFDL-804

Float Decimal Precision behaving unexpectedly

XMLWordPrintableJSON

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • s12
    • None
    • Back End
    • None

      I've been creating tests for XPath numeric functions (abs, floor, ceiling, etc) and came across a strange issue when testing round-half-to-even:

      I set up a schema:

          <xs:element name="round-hte">
            <xs:complexType>
              <xs:sequence dfdl:separator=",">
                <xs:element name="num" type="xs:float" dfdl:lengthKind="delimited"/>
                <xs:element name="precision" type="xs:int" dfdl:lengthKind="delimited"/>
                <xs:element name="roundval" type="xs:float" dfdl:inputValueCalc="{ fn:round-half-to-even(../ex:num,../ex:precision) }"/>
              </xs:sequence>
            </xs:complexType>
          </xs:element>

      I have a test with the following as input data:

      3.0000075,6

      so the first element in my infoset should just be the input data, but I get this:

      <num>3.0000074</num>

      I have no idea why the 5 is getting changed to a 4, but it happens in another test I created as well.

      See test_xPathFunc_round_hte_07 and test_xPathFunc_round_hte_03. It also appears that floats have a decimal precision limit of 7 digits (see test_xPathFunc_round_hte_08) in:

      M daffodil-test/src/test/resources/edu/illinois/ncsa/daffodil/section23/dfdl_functions/Functions.tdml
      M daffodil-test/src/test/scala-debug/edu/illinois/ncsa/daffodil/section23/dfdl_expressions/TestDFDLExpressionsDebug.scala
      M daffodil-test/src/test/scala/edu/illinois/ncsa/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala

              jchab Jessie Chab
              jchab Jessie Chab
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: