In order to be able to live in this solitary place, separated from the inhabited world, the hermits themselves made some improvements by creating various spaces for daily use inside. In addition to the refectory, infirmary, guest quarters, a tailoring workshop and cellar, a library was built, where residents studied and gathered to make important decisions. Small terraced gardens were created outside, protected by stone walls and cultivated by the hermits. A space was dedicated to guests who came on religious or pastoral visits.

The tower in the centre of the building was half torn-down and the space for the internal church of the Hermitage was created at its base. The church was open only to men on Sundays and feast days. The women of the community of Sassoferrato were only allowed access to the Hermitage of San Girolamo twice a year, on the Tuesday after Easter, and on 30th September, the feast of St Jerome.

The hermits who lived in this place came from various countries, including Italy, Austria, Spain, France, and many were Polish. This happened because in 1605, the Justinian reform entered Poland and the noble Polish man Nicolò Wolski founded a hermitage in Krakow.

Curiosity
During the Second World War, the hermitage was again sought by the people of Pascelupo and Perticano as a refuge from artillery and air fire. About fifty people, who had also come from Fabriano, stayed up here for over half a month, seeking refuge in the sacristy of the church, which had remained intact, and in the caves of the mountain. Once the fury of the war had passed, the path of destruction accelerated and the hermitage became a heap of rubble, until 1981 when restoration work began, allowing the Camaldolese hermits to reopen the hermitage in 1992.