Doctor here in general practice.
This works well in principle. One of many problems here is healthcare need is not spread around uniformly. In your example you just consider number of people and number of providers. This is ok of you are just thinking of primary care (it works like this in many places). It breaks down when there are surges. What happens during flu season? What happens if there is a fire and 30 people need treatment for smoke inhalation. What happens when the doctor needs to take a vacation or gets COVID during flu season? There is redundancy built into a larger healthcare system which makes access more robust over a wider range of conditions.
Also, doctor's don't always want to work in all places.it can be harder to recruit doctors to some areas.
There are a whole host of issues here. I agree though that having a middlman take a large cut of money to "grease the system" does contribute to it's inefficiency. The healthcare system is broken on so many levels that any one change like this would be set up to fail. We need a major overhaul.