I'm too sensitive?

I'm reasonably lucky to have very sensitive nipples, which is great during sex. I love to have my husband play with them, it's almost as good as him touching me *elsewhere*.

The problem is that he likes to play with them other times too, like when we are just getting into bed to go to sleep. He's not trying to initiate sex, he just likes to play with them a little bit. The problem is that they are SO sensitive that unless I can relax and enjoy the attention (i.e. during sex), I really don't want them to be touched. It is very distracting when I really just want to lay down, cuddle, relax and go to sleep. When he's playing with them, I can't relax and it gets a little annoying/irritating.

I've tried explaining this to him but he thinks it's just a cute joke between us now, him trying to touch them and me pulling away. It was cute at first but I'm starting to dread going near him in my nightie because he'll try to grab at me when all I want to do is reach over him to turn the light off and go to sleep.

FYI - Our sex life is fine. He have sex 2-3 times a week and he has said that the frequency is fine. I've asked him straight out if he is trying to initiate sex when he grabs me and he says no, he just likes to play with my nipples for fun.

How do I nicely tell him to keep his hands off? I don't want him to not cuddle or caress me other places (my back, my arm, etc) and I don't want him to stop playing with them during sex. I don't know how to describe the feeling of it being too sensitive. It's almost like being tickled. It can be fun and enjoyable sometimes but only in the right circumstance and not all the time. Then it gets annoying. KWIM?