Scripting Helpers is winding down operations and is now read-only. More info→
Ad
Log in to vote
1

Mitigate floating point errors?

Asked by
Optikk 499 Donator Moderation Voter
4 years ago

For a project of mine, it's necessary to save a large quantity of part positions and orientations. I'm concerned about only using as many characters as I need, but I often find that a position in a chunk of data might look like so:

view source
1[[
2{
3    "position": [4.50000000000000001, 9, -4.50000000000000001]
4},
5{
6    "position": [-13.500000000000001, 4.500000000000001, -4.5000000000000001]
7}
8]]

This isn't what the actual data looks like, but it's representative of the problem I'm facing. This uses MUCH more characters than necessary. I want to cut off at the first decimal place. How can I do this? If possible, I would like to solve this without using string methods.

1
math.floor(n * 10) / 10 cuts it off to the first decimal place. n is the number that has rounded off. DeceptiveCaster 3761 — 4y
0
I also understand your issue, since roundoff is a real pain. DeceptiveCaster 3761 — 4y
0
That did the trick for me, if you wanna post that as an answer I'll accept it. Optikk 499 — 4y

1 answer

Log in to vote
1
Answered by
DeceptiveCaster 3761 Moderation Voter Community Moderator
4 years ago

The issue you've come upon is called roundoff error. Unlike other errors, roundoff does not appear as an error in itself, but is evident when a number is printed and contains unnecessary digits.

What causes roundoff?

There are mostly two things in Lua that can result in roundoff:

  • The tostring() function. If you loop a set of numbers and print their string representations, some may round off. This is because of how tostring() converts numbers to strings; the numbers are not treated as literal values.
  • Incorrect float calculation, which is really the primary cause of roundoff. On occasion, if Lua is trying to represent a non-literal float (like if the number is encased in a string), it has to try to represent it exactly. Otherwise, it represents the number with an unnecessary number of digits, which is very close to the original number, but is not exactly the original number.

If it is indeed a string, you can convert it to a number and truncate it using math.floor():

view source
1local n = "3.4000000000001" -- A number that has rounded off. The original number is 3.4.
2print(math.floor(tonumber(n) * 10) / 10) -- Prints 3.4

Roundoff is really something you need to be aware of and be capable of fixing.

Ad

Answer this question