I am no expert but I would say this is still a huge issue and possibly the root cause. You were the victim he does not have the opportunity to just say "I don't want to talk about it". You are his wife and you deserve to know everything. Your lack of sex is probably because of this as well. Not sure how anyone could have sex with some after they cheated on them. Sorry but personally that is enough there to leave him...
Honestly, He probably met with an attorney or looked up the laws and information and found out that with his affair that you are aware of, he realizes that he will lose everything, most likely custody of the child. He will also lose all his money and probably have to pay alimony...so just suck it up and stick with it (possibly cheat or continue to since he got away with it once already and your still around) or lose everything he ever had and end up living in some ditch.
He knows you have the upper hand...honestly and this is just my opinion, once the affair happened this relationship should have been over. Especially after he just wanted to push it under the rug. Not cool...
Good luck with everything, in my eyes I feel hes stuck, and at this point hes just going to try and make himself love you. I wouldn't be surprised if you told him you wanted to go through with the divorce and he begged for you to stay...