It shouldn't be. It's a term that Christians came up with.
Xmas Abbreviation
Actually, this usage is nearly as old as Christianity itself, and its origins lie in the fact that the first letter in the Greek word for 'Christ' is 'chi,' and the Greek letter 'chi' is represented by a symbol similar to the letter 'X' in the modern Roman alphabet. Hence 'Xmas' is indeed perfectly legitimate abbreviation for the word 'Christmas' (just as 'Xian' is also sometimes used as an abbreviation of the word 'Christian').