People who deny the Holocaust are nearly as embarrassing and unserious as people who claim the Earth is flat.
However, we need to be careful about moving the goal posts and not accuse someone of being a Holocaust denier because they don't adhere to the 5-6 million number. If they acknowledge that the Nazis murdered millions of Jews but put the number at 3-4 million instead of 5-6 million, that does not make them a Holocaust denier. If they make clear that the Holocaust was an evil act and that at least 3 million Jews were murdered in it, I don't think we can fairly call them a Holocaust denier.
But, if someone puts the number of murdered at below 3 million, I think that's straying into Holocaust denial territory, because we have clear, indisputable documentary evidence of 3.5 million executions in the death camps.