Do they use our roads?
Do they use our police?
Do they enjoy the rights they have gained to work and get paid well?
Do they like eating clean food and drinking clean water?
Then no, it is not robbing anything from them as they're enjoying the benefits of paying it.
Should those who are not paying for any of it, enjoy the same benefits?