The beginnings of settlements in northeastern Bohemia date back to the time of princ Břetislav (1035-1055). During his reign a network of administrative fortified settlements was established in favourable locations, some of them brand new. Jaroměř was very likely one of the places, named after his uncle or son Jaromír. Not until the second half of the 13th century did the colonisation process moved to higher altitudes. In 1360s the town of Trutnov was founded. Around the same time the settlers from Silesia were reaching the area and the localities of Hostinné and Bolkov were established.
Vizmburk itself was founded by Tas or Peter of Skalice. This noble house with the stirrup in their coat of arms was known already in the 12th century, when their members held the burgrave office in Kladsko. The Vizmburk estate was a fairly large one with the economic centre in the town of Úpice. It included the villages of Bohuslavice, Rubínov, Suchovršice, Radeč, Studenec, Kyje, Brusnice, Maršov, Libňatov, Havlovice, Slatina, Boušín, Rtyně, Bohdašín, Batňovice, Zálesí, Svatoňovice, Petrovice, Sedloňov and Žaltmaň with a fort.
Tas of Skalice was an prominent figure in the family history. He belonged to the circle of faithfuls of king Václav II. and held several important offices in the Czech crown lands and in Poland. He was a violent man with a temper and it is believed he was murdered in 1304. His son Jaroš inherited all the fortunes and also some personal characteristics of his father. He was known to have oppressed the neighbouring settlements belonging to the Zderaz monastery. The complaints of the monks went as far as Rome. Therefore it is believed that Jaroš was forced to sell Vízmburk estate. In 1330 the the lords of Dubá owned the estate. Jiří of Dubá supported the Hussites movement which resulted in frequent raids of his estate by Silesian troops. The princes and burghers of Silesia and Lusatia eventually sealed the faith of the Vizmburk castle. Instead of arms they used the power of money. In 1447 the castle was bought up and intentionally demolished.