In 1794 Andreas Momsen, dairyman of the Dutch East India Company received in his private capacity a piece of land next to the dairy, in extent 1 407,98 m2. He probably lived in a house on roughly the site of the present day South African Museum and used the private land he was granted for agricultural purposes. He sold his garden in 1799 and died in 1812. Today this land forms the complex known as the University of Cape Town Orange Street Campus.

Hermanus ter Hoeven owned the property from 1799 until 1810 when he went insolvent. Records show that by 1810 there was an incomplete building on the site, which was sold to a relative, with the same name; Hermanus ter Hoeven. During his four years of ownership it is observed that a house was completed as it was described as "a piece of land with the buildings thereon" in the transfer deed when it passed to the Widow Smuts in 1815.
John Barker bought the property in 1839 for £1 381.17.6. No diagram was filed with any transfer deed before 1854 but on the available evidence, through the research of Margaret Cairns, it is presumed that Barker was responsible for the construction of Bertram House. It is generally agreed that because his wife Ann Bertram Findlay died in November 1838, he named the house after her.

An attorney and notary public from Yorkshire, Barker arrived at the Cape in 1823. He appears to have been an enthusiastic builder as revealed in a letter dated 1836, "I am much engaged with bricks and mortar, being my own architect/builder... with my new slate of, the front of English brick...". Barker remarried in 1845. His second wife was Maria Johanna Silberbauer, but there no children. He died in 1854 aged 57 leaving an "estate known as Bertram Place" to be sold after division into five lots described as follows:

(1) the dwelling house, coach-house and garden
(2) and (3) the vineyard,
(4) and (5) the cottage and a piece of ground.

The resulting sale divided the property into two main lots comprising lots 1,2,3 and lots 4 & 5. This account will concentrate on the subsequent ownership of what remained as Bertram Place, namely lots 1 - 3.

Augustus Frederick Carrew, a master mariner and ship owner owned the property from the end of 1854 until his death in 1857. His widow married Abraham Jozua de Villiers. Although the assessment rolls for 1860 to 1865 record de Villiers as owner, it is not known whether the property was owner occupied.

There were three occupiers recorded in the Almanacs during the years 1865 - 1867; John Frederick Bourne, Colonial Railway Engineer and the Widow Elizabeth Tyers (possibly neé Parkes) and G.W. Tyers.

Captain Robert Granger bought the property in 1867, but it is uncertain whether he lived in Bertram Place as he had a house in Mouille Point. He died at Southampton in 1870 and Bertram Place was sold from his deceased estate in 1871 to Esau Harrington, a draper and haberdasher. During the four years of Harrington’s ownership the property was run as a board and lodging house.

Robert Granger is remembered at Granger Bay following his heroic rescue of nine men from a schooner which capsized in heavy seas in February 1857.

In 1875 Bertram Place was sold to James Wiley, an ironmonger, who established a prosperous family business and was a wealthy property owner. He lived at Bertram Place between 1875 and 1884 before moving to a house in New Kloof Street. Wiley retired to England and died in 1898. He let Bertram Place to T. Hill, then to Captain Francis Rennie and finally to Tiberias Benjamin Kisch, (1840 - 1913) the first Jewish professional photographer in the Cape.

In 1885 James Wiley sold a section of the property of Bertram Place measuring 663,17m2 to his son Robert, who built a house known as Bertram Cottage.

In 1891 Robert’s wife Sarah bought the lots originally known as lots 4 and 5 from the estate of Isaac Lewis. The couple owned the Victorian house called Bertram Cottage and a smaller dwelling facing Rheede Street known either as Perivale or Oakdale Cottage. In 1893 they bought James Wiley’s property Bertram Place, comprising 512,45m2 thereby amalgamating the separate lots. From then on the property formerly known as Bertram Place was called Bertram House in the municipal records.

In 1903 part of the Wiley property was sold to the South African College and in 1929 the remainder was acquired by the University of Cape Town, successors to the South African College. The land on which Bertram House stands was transferred to the Union Government in 1930 and used by the Department of Health. The building was declared a National Monument in 1962 and transferred to the SA Cultural History Museum in 1976 for use as a museum commemorating the English contribution to life at the Cape. It was unofficially opened to the public as a museum shortly afterwards. Major restoration of the house was carried out by Revel Fox and Partners, Architects, during 1983 - 1984, funded by the Department of Community Development. Bertram House was officially opened as a late Georgian house museum on May 12 1984.