Firefox and Javascript Rounding Rules -


i don't know if i'm missing obvious here but...

in ie, opera , chrome, expect rounding numbers ending in 5:

125 toprecision(2) => 130 11.5 toprecision(2) => 12

this i'd expect.

firefox, though, little more 'sophisticated' yielding following:

125 toprecision(2) => 120 //wtf!!! 11.5 toprecision(2) => 12

after bit of head scratching, i've come conclusion firefox using 'rounding even' rule where, if digit before 5 number rounds down , if digit before 5 odd number rounds up:

0.5 => 0 1.5 => 2 2.5 => 2 3.5 => 4, etc.

i using rounded results test student solutions engineering questions pseudo-randomly generated question inputs. question input in chrome h=1020 mm h=1030 mm in ff, chrome or opera.

i need function make rounding consistent, i.e. want 0.0001235 round 0.000124 , want 1234 round 1240 can't use simple num = math.floor(num + 0.5); complicate matters bit, want input variables , student answers correct 3 sig digs unless first digit 1, in case want 4 sig digs:

234.5 => 235 134.5 => 134.5

i've hacked solution 3 or 4 sig digs depending upon first digit converting number string , testing first non-zero, non-decimal point , non-negative character '1' - not pretty, works. similar rounding problem, checking whether digit rounded 5 i'm wondering if there elegant bit-wise solution.

please take @ tests here

http://yuiblog.com/blog/2009/03/10/when-you-cant-count-on-your-numbers/

javascript has single number type: ieee 754 double precision floating point. having single number type 1 of javascript’s best features. multiple number types can source of complexity, confusion, , error. single type simplifying , stabilizing.

unfortunately, binary floating point type has significant disadvantages. worst cannot accurately represent decimal fractions, big problem because humanity has been doing commerce in decimals long, long time. there advantages switching binary-based number system, not going happen. consequence, 0.1 + 0.2 === 0.3 false, source of lot of confusion.

also take @ questions:

https://stackoverflow.com/questions/287744/good-open-source-javascript-math-library-for-floating-point-operations

and

https://stackoverflow.com/questions/744099/javascript-bigdecimal-library/1575569#1575569


Comments

Post a Comment

Popular posts from this blog

c++ - Is it possible to compile a VST on linux? -

for class project i'm attempting write vst plugin backed cuda. current cuda workflow on linux box, i'd prefer compile , link there. according wikipedia , should possible (i couldn't find steinberg documentation relevant linux) can't find makefile or instructions on how build if aren't using xcode or visual studio. i'm vst 3 sdk doesn't support linux. when try compile plugin under linux, error: ./base/source/fatomic.cpp:39:30: fatal error: libkern/osatomic.h: no such file or directory this issue caused following code in "vst3 sdk/base/source/fatomic.cpp" #if mac #include <libkern/osatomic.h> #if mac_os_x_version_min_required > mac_os_x_version_10_4 #define native_atomic_type (volatile int32_t*) #else #define native_atomic_type (int32_t*) #endif #elif windows #include <windows.h> #endif but hope compiling under linux work vst sdk 2.4. reading. jvstwrapper seems run on lin...
Read more

java - Output of Eclipse is rubbish -

i want write program in eclipse language different english. when display output using system.out program, displays output correctly in console editor. when accept input console , output it, output rubbish. how can solve it. many thnaks i'm not sure if problem, have character encoding problem. make sure you're using unicode, utf-8. you can read more @ official docs converting non-unicode text .
Read more

jquery - Confused with JSON data and normal data in Django ajax request -

i read json internet still have not got grasp of it. reading article http://webcloud.se/log/ajax-in-django-with-jquery/ i not understood first part function using json def xhr_test(request, format): if request.is_ajax(): if format == 'xml': mimetype = 'application/xml' if format == 'json': mimetype = 'application/javascript' data = serializers.serialize(format, examplemodel.objects.all()) return httpresponse(data,mimetype) # if want prevent non xhr calls else: return httpresponse(status=400) my main problems are from function getting format variable does format json mean data given function json or data recived json can give me simple example ouput of function data = serializers.serialize(format, examplemodel.objects.all()) how use data when response in jquery function if don't use json in above function how input , response chnage thanks ...
Read more