You're making it sound trickier than it is. Nutrition data on all foods will already discount fiber from the calorie counts.

But in a sense you're also not wrong, that while calories are king when it comes to weight loss/gain, there are complications for that. For example if you give two different people the exact same food in the exact same amount of calories, they will gain or lose weight at different rates - highlighting the role of genetics. Another genetic factor related to calories only indirectly is how some people have much higher impulses to eat than others, making calories only a part of the story for their challenges with weight loss. I've also seen a headline for a study claiming that an amount of dairy caused more weight gain than the same amount of calories of peanut butter, though you may want to take that one with a grain of salt unless you actually see the study.

Personally I'm not a fan of measuring calories. Instead I use base knowledge to have ways to intuit calories more naturally. For example, I know that carbs and protein are 4 calories per gram, and fat is 9 calories per gram, making fat almost always the quickest way to make foods significantly more calorie dense. Other things can be very calorie dense too though, like sugary or other caloric beverages. Replacing those with water, coffee, or teas can be enough on its own for some people to start losing weight.

Some foods are more dense than others. Being that leafy greens and many other vegetables are naturally some of the least caloric foods you can eat, loading all of your meals full of them is an elegant way to reduce calorie consumption without needing to starve yourself. It also has the double benefit that high fiber foods are more satiating - they calm food cravings.

Point is, calorie management doesn't have to be a headache, and it doesn't mean a person has to starve themself.