The residence originally belonged to the Takada clan of present-day Niigata prefecture, and changed hands several times until the ownership was transferred to Iwasaki Yataro, the founder of the Mitsubishi group, in 1878. More than 20 buildings have been built on the 49,500 m²-plus property over the decades.

Parts of the architecture have been designated as Important Cultural Properties of the country; the Western house and the billiards parlor in 1961, the Great Hall of the Japanese house and the Japanese style decorative screens housed in the East Wing of the Western house in 1969, and finally the entire area, including the land and the surrounding tiled walls, in 1999.

The Western house and billiard parlor were designed by eminent British architect Josiah Condor (1852-1920), and has been counted among his finest works. No less impressive are the gardens that were deeply inspired by gardens of preceding feudal era, and greatly influenced many that followed.