A challenge is an attempt to remove or restrict materials, based upon the objections of a person or group.
A banning is the removal of those materials.
Challenges do not simply involve a person expressing a point of view; rather, they are an attempt to remove material from the curriculum or library, thereby restricting the access of others. As such, they are a threat to freedom of speech and choice.