Brougham Castle is built next to the River Eamont on the site of an old Roman fort, Brocavum. A square keep with its entrance on the first floor and access stairs protected by a forebuilding was constructed by Robert de Vipont in the early 13th century. He also added a hall block (now destroyed) and various other service buildings all enclosed within a timber palisade.