I think you have a right to speak with him about it. If you have not set any ground rules, now may be the time to have a straightforward and equitable talk about your feelings and needs. Remember he will need to be able to do the same. It can be a hard thing to want to share your new relationships with your spouse but as long as you both respect the others feelings and don’t modify any rules or agreements that you have made with each other without consulting each other then things will be easier. Note I did not say easy, just easier.