Also, they had mayors and stuff, a VERY clear class division and were technically under the rule of Gondor after the Northern Kingdom fell but Gondor didn't really have the resources or urge to do any ruling over there. Once Aragorn was in they were essentially an autonomous zone under military protection in exchange for maintaining the local roads and bridges.
The timeline says they were "allowed" to move there by original settler-colony of Arnor, which suggests displacement since Tolkien is a known Western sympathizer.