Slaves nowadays usually refers to either American-style chattel slavery or Roman-style slavery, both of which were systems much different than Serfdom.

But yes serfs will likely have built it, or were involved with the build under direction of hired stone masons, on order of a noble and with resources a noble paid for, under the general societal rule that serfs were to spend a certain amount of days a year working on infrastructure stuff as part of their taxes. For more details you'd have to dig into the law at that time at that particular place but those kinds of arrangements were incredibly common in Europe in the middle ages. You tended to be able to buy yourself out of having to work, and also pay your way in silver or gold out of paying in grain, livestock, etc.