He is in an emotional affair, at the very least. This is encroaching upon your life and your marriage, and it's time he let this other woman go. It's gone beyond the good samaratain clause. It's looking pretty darn fishy, and something more is going on so don't let him try to pull the wool over your eyes and make you feel silly for thinking so. Just read up on all the other EAs going on on this site and you'll recognize the signs.