Questions tagged [numeric-precision]
The numeric-precision tag has no summary.
34 questions
-4
votes
2
answers
127
views
Successor in a floating point system [closed]
I’m starting at numerical methods and I am introducing to the machine epsilon and succesor. I have this question, which is the succesor of .13579x10^2 at a floating point system of (10,5,-2,6). My ...
- 3
7
votes
6
answers
2k
views
When are equality comparisons between IEEE 754 floating point values acceptable?
My understanding is that it's rarely a good idea to compare two floating point numbers due to inherent inaccuracies. However, my understanding of those inaccuracies are that they are deterministic and ...
user361757
1
vote
1
answer
2k
views
What is the proper way to deal with rounding error?
I am currently working on a Physics Engine in C++. The Physics Engine is for a sandbox video game. I want to build a powerful physics engine, so therefore, it needs to be very precise.
I have ...
- 129
1
vote
7
answers
740
views
Is the sum or subtraction of two positive machine numbers exact?
If you have two positive numbers that can be represented exactly by the finite arithmetic of your computer, that is, two machine numbers, if you perform their sum or subtraction with perfect ...
3
votes
5
answers
3k
views
Which layer should have responsibility for rounding numbers?
I am developing a financial system and want to have a defined policy for rounding monetary values.
Given the following layers:
View
API
Entity Model
Persistence
If I am passing a monetary value ...
- 139
1
vote
4
answers
2k
views
Unit test a generic floating point equality function
I've written a function that tests two floating point numbers for approximate equality (see this Code Review question). I'd like to unit test my function, but I'm not positive of the best way to do ...
- 1,291
1
vote
1
answer
1k
views
Is new object creation necessary with BigDecimal?
Problem description:
I am running performance test of calculation of PI number with primitives and with BigDecimal class.
Calculation of PI with primitives is around 5-6 times faster than the same ...
- 257
0
votes
1
answer
65
views
Historic (and prehistoric) dates in Ruby [closed]
In Ruby, is there a standard class or gem to work with dates that can be - but are not required to be - historical in nature? The dates have precision (to the nearest million years, decade, century, ...
- 121
0
votes
2
answers
814
views
Floating point accuracy and conversions
I am trying to build a class where user can compute floating operations with a particular accuracy. For example, if the user wants to have up to 4 digit accuracy, everything less than 0.0001 is zero ...
- 103
-1
votes
1
answer
516
views
Floating point absorption phenomena and ULP
Numerical analysis text books talks about the absorption phenomena (Introduction to Numerical Analysis and Scientific Computing; Nabil Nassif; Dolly Khoueiri Fayyad; CRC Press; 2014) when adding ...
- 1
-1
votes
1
answer
84
views
multiplication with insufficiently long registers [duplicate]
I want to multiply two numbers but since I don’t have long enough registers in the current architecture need to “break them down” into shorter ones and somehow perform the calculation.
For example:
...
- 107
9
votes
3
answers
3k
views
Is order of arguments in an arithmetic expression important to achieve as most exact result as possible (speed is not necessary)?
Actually, in this question I do not ask about particular language or architecture, however I understand there might be some differences.
In physics / engineering it is usually better to handle ...
- 201
0
votes
0
answers
544
views
BCMath vs MySQL: performance
Scenario
I am working on a web application that handles financial transactions, and my code will need to be reusable for everything from consumer carts to manufacturer/distributor shipments. The ...
- 224
2
votes
3
answers
3k
views
How can a computer round the last digit in a floating point representation?
I'm confused by how a computer rounds off the last digit in the floating point representation. For example, I'm told that x=1.24327789 is stored in a computer with a 6-digit capacity then it;s ...
- 301
1
vote
3
answers
517
views
Avoid numeric errors in Naive Bayes method
I am implementing the Naive Bayes method with Gaussian distribution.
The problem that I have is that the variance used on the Gaussian curve (calculated from a training set) is REALLY small. They ...
- 113