Not necessarily while he is still on probation. When he has completed his probation or parole, and is restored to full free status, then I am in favor of him getting all of his rights back. Once he has truly “paid his debt to society”, as long as he stays out of trouble, he doesn't owe society any more loss of freedom.
I also, however, advocate a very harsh treatment of persistent or severe criminals. One serious enough crime (murder, possibly some instances of rape or child molestation), he should get the death penalty. Same with a persistent pattern of lesser crimes, sufficient to indicate that he will never be willing to behave like an actual human being, and to peaceably coexist with actual human beings. At some point, we need to declare a criminal to be unsalvageable, and not worth allowing to continue to exist, and to create any further risk to actual human beings.
Even then, once he's completed that sentence, he still has what rights he is left able to exercise. For example, if he wants to keep his guns, then they can be buried with him in his grave. At that point, I don't think we need to worry much that he'll use them to commit any further crimes, do you?