Lucy is right, feedback is always voluntary, but any feedback you do receive is counted in your total.
For me, I am more concerned about what I am buying, & the price I am paying for my item, or the quality of a item I am selling, instead of how it will help my FB score. I don't want to indicate that I don't think it is important, since it is how other people can judge the kind of buyer or seller you are, but it is not the most important thing to me when I make a purchase, or sell a item.
Thanks!
JJ