The others have put good descriptions of why calories are an accurate measurement for food energy.
However, you are absolutely correct that calories are not a perfect measurement, and different types of foods are not one to one replaceable. 1500 calories of sugar is NOT the same as 1500 calories of protein!
Burning the food produces a reasonable and useful approximation of the available energy.
Does the human body burn food? Of course not. We transform food into useful components and then pump them around the body to be used by cells.
If you eat 1500 calories of protein, your body will use some of those calories simply as proteins, rather than breaking them down into energy (via sugar). Which means you will have less food-energy in your system and are more likely to run a deficit.
Again with protein, the transformation of protein into sugars which can be used as energy takes energy, so you end up with a smaller amount of calories actually being available.
TL;DR Calories are not perfectly interchangeable. However, they are our best, and most useful, quick way of approximating energy intake from food.