From 1264, when the Pisani took possession of the island, the fortress of Giglio Castello that had its present town planning with them, became strategically important. Located 405 metres above sea level, it is now the seat of the municipality that governs Giglio.

The castle is surrounded by stately walls with three towers with circular plan and seven towers with rectangular base, which carefully watch them, and three gates. The main gate still has its terrible defensive features intact, as the numerous positions where the archers could catch their enemies between a lethal crossed shooting. On the inside facade of the last gate there is still a marble plate portraying the coat of arms of the Medici who did much to defend the island.
This stately structure, whereas on the one hand it never failed to give its protection against the pirates' raids, on the other hand has made the island appear like a rich and promising place for the plunderers' bands. Improved and restored by the Grand dukes of Tuscany, Giglio Castello has kept its thirteenth-century architecture until now, characterised by narrow lanes, often surmounted by arches and with the typical external stairs to reach the higher levels of the houses, called "balzuoli". Inside the walls it is still possible to see the big cistern wanted by the Medici to keep the necessary water to guarantee the survival for the inhabitants when the Saracens besieged them.
In XVIII Novembre Square the Rocca Aldobrandesca still dominates, the massive defensive construction built by the Aldobrandi, one of the families that had the difficult task of governing and protecting the island.
In the village you can visit the Parish, guardian of many valuable sacred objects. Among them there is saint Mamiliano's ulna, the protector of the island who lived around the middle of the 5th century A.D., and a holy water stoup obtained from an old Corinthian capital of the age of Traiano, probably stolen from one of the ancient Roman settlements on the island. Just outside the village an old water mill is still visible and well preserved, which gives the name to the valley below the castle. Then many cellars that produce Ansonaco, the typical local wine, can be found while walking along the lanes.