They were the descendants of northmen (hence the name), but spoke a form of French. In addition to the existing answer, Anglo-Saxon England was originally occupied with relatively few Normans to subdue a population of millions. Castles were essential to protect these small bodies of men in times of (frequent) revolt , and allowed them to control the surrounding country in more peaceful times. The Normans were used to building castles to retain their possessions elsewhere (in Normandy, Brittany, Sicily). Many of the Saxon aristocracy fled as political refugees and with their loyal followers joined the Varangian Guard in Constantinople (Istanbul now), capital of the Byzantine Empire. Five years after Hastings, Saxons were fighting the Sicilian Normans in the Balkans, trying to ward off another Norman takeover bid.

Source(s):
Feudal Britain by J Barrow