OP, please see a psychologist to set your mind in the right place before your child is born; as you do not wish to seek therapy for your marriage. You're focusing on the OW, who is 9 years older than your husband because your marriage was in a rut. What then when you run into "ruts", especially that you are expecting?
Based on your observations, your husband leads in their relationship. He is no pushover as he dragged her upstairs to you & told her to get her own cigarettes. Their dynamics shows you that your husband is a liar & a cheat. See him for who he is. She is a cougar and a home wrecker. They are two of a kind.
I understand your reluctance on walking off as you are in the late stages of pregnancy. However, I recommend for you to tell your friends and family. Even if the whole town knows about their affair, you should tell. Other eyes will be on them and you would know your husband's whereabouts. As you can see the OW is so bold. I speculate that she already told others and is mocking you. A conquest of a younger man for this older woman is a notch in her belt.
I'm very sorry about your situation.