Records of owners of the property known as Bertram House, situated in Government Avenue, date back to 1794 when Andreas Momsen was granted the site by the Dutch East India Company. Details regarding dates of the construction of the present Georgian town house and the identity of the author remain a mystery. Nor is there conclusive proof that John Barker, owner from 1839 until 1854, was the builder, although it is generally agreed that the house was named in memory of his wife Ann Bertram Findlay, who died in 1838.

In 1841 Barker, an attorney and notary public, requested permission to place a door in the Government wall around the gardens to have easier access to the public gardens. He left an estate consisting of a "dwelling house, coach house, garden and vineyard" as well as "a cottage and a piece of ground".

During the century that followed, Bertram House was used for various purposes ranging from that of family home to boarding house. Later it became part of the South African College before being transferred to the Union Health Department and declared a National Monument in 1962.