The Palazzo Pastorio is already recognizable in the plan of Castiglione delle Stiviere printed by Pierre Mortier in 1704. It was a building of simple and severe proportions, located next to the Collegiate Church (now the Duomo), and the Oratory of Disciplines.

The Palace had been purchased, in the second half of the fifteenth century, by Bartolomeo Pastorio, a patrician of Paduan origin, who had settled in Castiglione around 1440 - 1450. The building had undergone an important transformation in the very first years of the seventeenth century, when Fausto (who was the first Abbot of Castiglione), and his brother Prospero (who provided fundamental support to Francesco Gonzaga in the military defense of the city), expanded it, creating, among other things, the large theater and ballroom.

Due to a division between cousins, in the decade between 1750 and 1760 the property of the palace was divided into two parts, which were both enlarged, one to the right and the other to the left, with wings passing "bridge-like" over alleys, then closed with gates. The right half maintained its appearance until the middle of the 20th century, when it underwent heavy modifications. A recent restoration (end of the 20th century) has largely recovered its original shape.

Between 1760 and 1765, the left-hand side - now Palazzo Bondoni Pastorio - underwent renovation by Giovan Giacomo Pastorio, who gave the building its current shape. While the left side was partially extended above the arch above a small street, clearly visible in the plan of the Mortier, the front was raised a few meters, so as to create a second floor. It was thus possible to create a façade of balanced proportions, horizontally divided into three levels: the ground floor marked by two windows and two main doors, decorated with ashlars; the upper floors each with four openings. The main floor has a higher height with classical tympanums, triangular in the centre and arched at the sides, while on the second floor the openings are completed by lower, lighter frames decorated with volutes. The choice of Giovan Giacomo Pastorio, in keeping with the tastes then in vogue among the Lombard aristocracy, responded to the dictates of that late classical baroque, lightened forms, which took the name of "Lombard baroque".

Giovan Giacomo Pastorio died suddenly and prematurely, around 1765. In the following years, Francesco, Giovan Giacomo's eldest son, and then his sons continued the work of celebrating the family and embellishing the palace.

The decoration of the Sala dei Paesaggi on the ground floor dates back to the years between the end of the 18th century and 1824. The ceiling of this room is decorated with monochrome frescoes with the symbols of poetry, music and theatre, while in many small sails the paintings, also monochrome, include, together with landscapes of the upper Mantua and classical ruins, representations of the main possessions of the Shepherds. In a lunette above the entrance door of the hall there is a faithful image of the palace itself.

On the main floor, the three rooms on the front are characterized by decorated ceilings, the central coffered one with floral friezes, the one on the right (on entering) by an architectural breakthrough ceiling, the one on the left by lunettes, in which the coats of arms of the noble families related to the Pastorio are frescoed. On the same floor, towards the inside, there is a large theatre and dance hall.