After many years of Scottish raids in the region, William Strickland, who later became Bishop of Carlisle, was granted a licence to crenellate his property in 1397. It started as a pele tower and expanded into a square castle built around a courtyard. Much of the development of the castle was carried out by Ralph Neville, first Earl of Westmorland, who inherited the property in 1419. Further improvements were made by Richard, Duke of Gloucester, during his time as Lieutenant of the North, before he ascended the throne as Richard III in 1483.