[LUA] I Need a Very Fast Way to Accurately Truncate Floats.

[Bobert13]

I need a fast method for truncating floats to exactly 7 decimal places. All of the numbers I'll be dealing with will be between 0.0 and 1.0 if that makes this any easier.

The problem I'm having is whenever multiplying two double precision floats during my script, the result is inconsistent. I can literally perform the exact same multiplication 20 times and get 3 different results! I have no idea why nor how this is happening, though it occurs whether I'm using the default lua51_win32.dll or the luaJIT.dll. To get around this, I've decided to track down and truncate every number before multiplying it by another number. I've tried two different methods:

math.floor(number * 10^decimals)/ 10^decimals

This method is fast, but it leaves numbers in the lower insignificant decimal places. The insignificant digits are consistent; however, the product of multiplying two numbers that have been truncated in this way is not consistent!! It's putting numbers in the insignificant digits because there is no way to represent many decimal numbers in a floating point (0.1 for example [or 1/10] ends up being 0.1000000000000##### in a double precision float).

tonumber(string.format("%.7f",number))

This method prints out that it's working perfectly. I haven't fully confirmed whether or not the product of two numbers truncated in this way are consistent. It puts no numbers in the insignificant digits. However, it's ridiculously slow. Implementing this in two places in my script doubles the amount of time it takes to run. I'm probably going to have to use some form of truncation in hundreds of places throughout my script so this is unacceptable.

If anybody has any insight on the matter, a better truncation method, or a way I can change the problem so that I'm not running into this issue, I would highly appreciate your help on the matter.

 

[stackpointer]

Since the goal is high precision float multiplication, would multiplying the two numbers while they were integers work? Since you mentioned that the insignificant digits are consistent after the division.


Something like:

function floatM(numA, numB, dec)
   return (math.floor(numA * 10^dec) * math.floor(numB * 10^dec)) / 10^(2 * dec)
end

 

[Bobert13]

Since the goal is high precision float multiplication, would multiplying the two numbers while they were integers work? Since you mentioned that the insignificant digits are consistent after the division.


Something like:

function floatM(numA, numB, dec)
   return (math.floor(numA * 10^dec) * math.floor(numB * 10^dec)) / 10^(2 * dec)
end

I have a feeling that this would cause overflow in the opposite direction. I have no idea whether the products would be consistent or not due to this. However, implementing this will be very complicated in many scenarios. I'll do some testing with this but I'm thinking it's going to be just as problematic.

The real question is why the hell is lua 5.1 not consistent in how it handles rounding floats. Beyond that, how is it possible to get THREE different products from the same multiplication?!

 

[stackpointer]

Yeah, I just realized that the function would end up handling numbers going up to 10^14.

How much does the products vary by? Would it be possible to truncate the results of the multiplication if the product doesn't vary past seven decimal places?

 

[Bobert13]

Yeah, I just realized that the function would end up handling numbers going up to 10^14.

How much does the products vary by? Would it be possible to truncate the results of the multiplication if the product doesn't vary past seven decimal places?

I uploaded an image that shows this over in this post. The variance in that particular case is deceivingly small however. This is regarding the elevationMap in PerfectWorld3 so these numbers are multiplied over and over. I've already implemented a more consistent method for normalization so I don't feel it's an issue. However, functions such as the following are still producing inconsistent results even though I'm truncating every step of the way (even the addition and subtraction!):

Spoiler :

function CubicInterpolate(v0,v1,v2,v3,mu)
	mu = math.floor(mu*10000000)/10000000
	local mu2 = math.floor((mu * mu)*10000000)/10000000
	local a0 = math.floor((v3 - v2 - v0 + v1)*10000000)/10000000
	local a1 = math.floor((v0 - v1 - a0)*10000000)/10000000
	local a2 = math.floor((v2 - v0)*10000000)/10000000
	local a3 = v1

	return math.floor((a0 * mu * mu2 + a1 * mu2 + a2 * mu + a3)*10000000)/10000000
end
-------------------------------------------------------------------------------------------
function BicubicInterpolate(v,muX,muY,debugOn)
	local a0 = CubicInterpolate(v[1],v[2],v[3],v[4],muX);
	local a1 = CubicInterpolate(v[5],v[6],v[7],v[8],muX);
	local a2 = CubicInterpolate(v[9],v[10],v[11],v[12],muX);
	local a3 = CubicInterpolate(v[13],v[14],v[15],v[16],muX);

	if debugOn then
		print("a0:"..a0.." a1:"..a1.." a2:"..a2.." a3:"..a3.." return:"..CubicInterpolate(a0,a1,a2,a3,muY))
	end
	
	return CubicInterpolate(a0,a1,a2,a3,muY)
end

Then there's stuff like this, where I don't even know where to begin :

Spoiler :

tVal = (math.sin(tVal*math.pi-math.pi*0.5)*0.5+0.5)^0.25

Edit: I've also found that this truncation method itself is inconsistent (that's why BiCubic interpolation is inconsistent)! Sometimes when BiCubic gets called, it truncates every number leaving insignificant digits and sometimes it appears to be casting the variables themselves as single precision floats (with both the default .dll and the JIT .dll!) which cuts off the insignificant digits.

 

[Bobert13]

Is there support for type casting at all in Lua (with or without an extension library)? I'm thinking that may be the best solution for consistency and speed. If this requires an external library, would including that complicate distributing the script (the library would have to be distributed with the script correct?)? Could I possibly get around this by manually moving the methods used for type casting directly into my script? Would this behave the same on non-Windows platforms? Can I ask any more questions in one paragraph?

 

[Bobert13]

Well, I cut my idea down to 6 digits of precision instead of 7 and implemented the whole multiply while shifted deal stackpointer suggested and it's definitely improved consistency.

I'm still getting some variance in the digits past the 6th in BiCubic interpolation but the resulting elevation seems to be coming to the same exact number so far, regardless of the inconsistency in the middle of the operation . I'm still going to see if I can track down the inconsistency and try to get rid of it as, in theory, I may just be testing a particular case where it's not causing a problem.

I'm still interested in other methods or ways around this issue, so if anybody has any ideas, speak up.

 

[Bobert13]

Ok, so after a number of intensive Google sessions I finally ran across this blog on FP inconsistency.

To summarize: FP consistency on Windows using an x86 architecture derived processor is a pipe dream. I can take great measure to significantly reduce the error rate but it will never truly be consistent no matter how many times I truncate or force the FPU to operate in mode z or re-compile my own lau51_win32.dll that uses SP floats instead of doubles or...

There still may or may not be arguments for using a wrapper such as lua decNumber but I can't even follow the instructions to compile and use it, so expecting my users to do so would be ludicrous.

I'll be investing my efforts into other avenues, though I suppose I'll keep an eye on this thread in-case anyone would like to discuss inconsistent results in FP.