Mitigate floating point errors?
For a project of mine, it's necessary to save a large quantity of part positions and orientations. I'm concerned about only using as many characters as I need, but I often find that a position in a chunk of data might look like so:
[[
{
"position": [4.50000000000000001, 9, -4.50000000000000001]
},
{
"position": [-13.500000000000001, 4.500000000000001, -4.5000000000000001]
}
]]
This isn't what the actual data looks like, but it's representative of the problem I'm facing. This uses MUCH more characters than necessary. I want to cut off at the first decimal place. How can I do this? If possible, I would like to solve this without using string methods.
1
0
0
1 answer
The issue you've come upon is called roundoff error. Unlike other errors, roundoff does not appear as an error in itself, but is evident when a number is printed and contains unnecessary digits.
What causes roundoff?
There are mostly two things in Lua that can result in roundoff:
- The
tostring()function. If you loop a set of numbers and print their string representations, some may round off. This is because of howtostring()converts numbers to strings; the numbers are not treated as literal values. - Incorrect float calculation, which is really the primary cause of roundoff. On occasion, if Lua is trying to represent a non-literal float (like if the number is encased in a string), it has to try to represent it exactly. Otherwise, it represents the number with an unnecessary number of digits, which is very close to the original number, but is not exactly the original number.
If it is indeed a string, you can convert it to a number and truncate it using math.floor():
local n = "3.4000000000001" -- A number that has rounded off. The original number is 3.4. print(math.floor(tonumber(n) * 10) / 10) -- Prints 3.4
Roundoff is really something you need to be aware of and be capable of fixing.