Arabs are also Semitic!
Really? Well I'm a native American ... I was born here.
Are you classified by the U.S. Government as Native American? If not - you ain't a Native American.
I'm using your Semitic "logic." Since I was born here in America, I'm a native.
Now according to the dictionary, anti-Semite refers to those who are "hostile toward or prejudiced against Jews" and even comes with a pic of you: