this post was submitted on 19 Apr 2025
117 points (98.3% liked)

Ask Lemmy

31148 readers
2418 users here now

A Fediverse community for open-ended, thought provoking questions


Rules: (interactive)


1) Be nice and; have funDoxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them


2) All posts must end with a '?'This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?


3) No spamPlease do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.


4) NSFW is okay, within reasonJust remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com. NSFW comments should be restricted to posts tagged [NSFW].


5) This is not a support community.
It is not a place for 'how do I?', type questions. If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.


6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !askusa@discuss.online


Reminder: The terms of service apply here too.

Partnered Communities:

Tech Support

No Stupid Questions

You Should Know

Reddit

Jokes

Ask Ouija


Logo design credit goes to: tubbadu


founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] chaosCruiser 19 points 4 days ago* (last edited 4 days ago)

Speaking of Excel, here’s a fun little experiment into the nature of binary numbers and rounding errors.

Start with some number and add a fraction like =A1+(1/3) to it. In the cell below, add that same fraction to the previous one. Copy this formula downwards and watch the numbers grow. After about 50 rows, you’ll have a number that looks like something specific, such as 71, but it isn’t exactly. There’s a sneaky rounding error hidden in there. The actual number is very close to the one displayed, but not exactly what you think it is.

If you’re using IF statements or XLOOKUP with numbers like this, you’ll run into some perplexing errors. If I recall correctly, you can even test the number with =A50=71, which will return TRUE but the xlookup still fails. It’s been a while since I tested this one, but I remember it being really weird in all sorts of unexpected ways. It’s weekend, so I’m not touching my work computer today.

You just need to know that a long series of fractions causes weird binary rounding errors to happen behind the scenes. Adding a series of whole numbers and neat decimal numbers was perfectly ok though.

Also, trying to explain this to some coworkers won’t be worth the effort.