The Annibaldi family briefly succeeded to the Pironti. And in 1301 it was Riccardo Annibaldi who sold "the whole castle of St. Felice, the place known as Paola, and the other place known as Santa Maria della Sorresca" to Pietro Caetani, thus marking the beginning of this family's rule, which lasted, almost uninterruptedly, for over 400 years.
In 1323, Cardinal Francesco Caetani inherited Circeo, together with Sermoneta, S.Donato and Bassiano; in 1324, he assigned the Circeo Promontory as a fief to Rodoisio, counsellor of Robert of Anjou. In 1332, the latter assigned back the Circeo fief to the Caetani family, namely to Nicolò I.
In 1378, the Circeo territory was affected by the Western Schism, albeit marginally. Antipope Clement VII, who resided at the castle of Onorato I Caetani, Count of Fondi, declared that Jacopo Caetani - who was instead a supporter of Pope Urban VI - lord of Sermoneta, Bassiano, S.Felice and Ninfa, together with Francesco and Piero Caetani, had lost his rights on these lands. Following these events, Circeo remained under Onorato I until his death in 1400.
From 1401 and for ten years, the church directly ruled on the area. Pope Boniface IX appointed a Vicariello Verrocchi as lord of the castle. This system of direct government lasted until 1411, when the fief was given back to the Caetanis.
In 1426, Ruggero I Caetani relinquished his rights on Bassiano, S.Felice, Ninfa and S.Donato in favour of his brother Giacomo IV Caetani; in 1432, Cristoforo Caetani did the same for his Latium fiefs, thus making of Giacomo IV the founder of the Caetanis of Sermoneta.
When Giacomo IV died in 1433, he was succeeded by his young son Onorato III Caetani, first under the guidance of Francesco Caetani di Maenza, and then alone. The first-born of the founder of the Sermoneta Caetanis had to face a long ordeal before his rights on the family fief were acknowledged.
In 1482 Alfonso, Duke of Calabria, occupied Circeo in his war against Pope Sixtus IV; then it was the turn of the French troops led by Charles VIII in 1495.
But it was at the end of the century that calamity struck the Caetanis, when Alexander VI Borgia, in his attempt to favour his family, systematically attacked the other dynasties: in 1499, Pope Alexander VI lured protonotary Giacomo Caetani (then head of his family) in Rome, had him imprisoned in Castel S. Angelo and, after having accused him of lese majesty and confiscated his family riches, had him poisoned. After this, Lucretia Borgia acquired Sermoneta and all the other Caetani fiefs.
When Alexander VI died in 1503, Guglielmo Caetani, son and heir of Onorato III, left Mantua, where he had found shelter by the Gonzaga family, and went to Sermoneta where the people welcomed him enthusiastically. In 1506, Pope Julius II conceded the right to rebuild the fortress, which represented the original nucleus of today's town of San Felice.