In 1187 the fortress fell under the dominion of the St. Lawrence Cathedral (Perugia). Expansion works started shortly afterwards, in 1202, and transformed it into a true fortified castle able to protect the Perugia territories on the Tuscan side. The surrounding houses were embedded into the walls.
Throughout the Middle Ages, the Passignano castle changed owners several times and suffered  plundering and raids. But especially after the war between the dukedom of Tuscany and the Papal state (1479) its aspect underwent numerous changes. Many towers were destroyed and several houses were damaged.
In Renaissance times, numerous fights for the ownership of the fortress followed one another and involved some of the most important families of the area: Della Corgna, Baglioni and Degli Oddi. The stronghold was also involved in the “Barberina War” between the papal armies and the Tuscan troops of the Grand Duke Ferdinand II.
In 1778 a part of it was remoed, in order to allow for a better transit of carriages. Between 1816 and 1817, the Pontifical Government decided to demolish of the entire complex and ordered that the materials coming from towers and walls be used to lift the road level. Indeed, the lake overflowed many times, flooding and damaging the houses on the shore.