The Comyns, Scottish knights, were established in Tynedale in the 12th century and the manor of Tarset was granted to William Comyn in around 1222. Licence to crenellate was granted to John Comyn in 1267, and Tarset Castle was built half a mile south-west of the present village, of which only grassed-over remains can now be seen. Two miles to the south-west on Birks Moor are the remains of the fortified house known as Dally Castle, believed to have been erected by David Linsey in his manor of Chirdon.