I only know that books can't be removed from high school libraries for political or partisan reasons. The protections of the First Amendment might go much further even with respect to high school libraries, and they may go much further with respect to public libraries. That was just the case that I found after searching for a couple of minutes. That was only meant to demonstrate that the 1st Amendment does limit the government even in government provided forums. I don't know what the precise limits are with respect to different forums.
People try to get books banned for all sorts of reasons. As far as I can tell they lose eventually because of the 1st.
For instance, a lot of the reasons on this list have nothing to do with political speech:
3 I Know Why the Caged Bird Sings by Maya Angelou
Reason for challenges: racism, homosexuality, sexually explicit, offensive language, unsuited to age group
18 The Color Purple by Alice Walker
Reason for challenges: sexually explicit, offensive language, violence
24 Fallen Angels by Walter Dean Myers
Reason for challenges: racism, offensive language, violence
31 Kaffir Boy by Mark Mathabane
Reason for challenges: homosexuality, sexually explicit
39 The Bluest Eye by Toni Morrison
Reason for challenges: sexually explicit, offensive language
42 Beloved by Toni Morrison
Reason for challenges: sexually explicit, violence
67 The House of Spirits by Isabel Allende
Reason for challenges: sexually explicit, offensive language
71 Native Son by Richard Wright
Reason for challenges: sexually explicit, offensive language, violence
75 Bless Me, Ultima by Rudolfo A. Anaya
Reason for challenges: sexually explicit, offensive language, occult
85 Song of Solomon by Toni Morrison
Reason for challenges: racism, sexually explicit, offensive language
86 Always Running by Luis Rodriguez
Reason for challenges: sexually explicit, offensive language
Other books written by authors of color challenged during this period include Roll of Thunder, Hear my Cry (offensive language) by Mildred D. Taylor and American Indian Myths and Legends (sexually explicit) by Richard Erdoes and Alfonso Ortiz.
http://www.ala.org/ala/oif/bannedbooksweek/bbwlinks/challengedauthorsofcolor.cfm