There is ‘a small but very loud minority of extremists, who are very uncomfortable with the cultural change around them,’ a local advocate says