Scripting HelpersARCHIVE
A little piece of Roblox history. Preserved for everyone, one question at a time.Read about the archive →
All questions/Question #115934
✓ ANSWER ACCEPTED

Mitigate floating point errors?

Asked 376 views1 answers

For a project of mine, it's necessary to save a large quantity of part positions and orientations. I'm concerned about only using as many characters as I need, but I often find that a position in a chunk of data might look like so:

[[
{
	"position": [4.50000000000000001, 9, -4.50000000000000001]
},
{
	"position": [-13.500000000000001, 4.500000000000001, -4.5000000000000001]
}
]]

This isn't what the actual data looks like, but it's representative of the problem I'm facing. This uses MUCH more characters than necessary. I want to cut off at the first decimal place. How can I do this? If possible, I would like to solve this without using string methods.

Anonymous

3 comments

1

math.floor(n * 10) / 10 cuts it off to the first decimal place. n is the number that has rounded off.

Anonymous#
0

I also understand your issue, since roundoff is a real pain.

Anonymous#
0

That did the trick for me, if you wanna post that as an answer I'll accept it.

Anonymous#

1 answer

✓ Accepted answer

The issue you've come upon is called roundoff error. Unlike other errors, roundoff does not appear as an error in itself, but is evident when a number is printed and contains unnecessary digits.

What causes roundoff?

There are mostly two things in Lua that can result in roundoff:

  • The tostring() function. If you loop a set of numbers and print their string representations, some may round off. This is because of how tostring() converts numbers to strings; the numbers are not treated as literal values.
  • Incorrect float calculation, which is really the primary cause of roundoff. On occasion, if Lua is trying to represent a non-literal float (like if the number is encased in a string), it has to try to represent it exactly. Otherwise, it represents the number with an unnecessary number of digits, which is very close to the original number, but is not exactly the original number.

If it is indeed a string, you can convert it to a number and truncate it using math.floor():

local n = "3.4000000000001" -- A number that has rounded off. The original number is 3.4.
print(math.floor(tonumber(n) * 10) / 10) -- Prints 3.4

Roundoff is really something you need to be aware of and be capable of fixing.

AnonymousPermalink

This discussion is preserved as it was. New answers, comments, and votes are closed.