How hard for you. You do need to take the course that is best for your children, which ever that is. You are having an affair with someone who has taken more interest in you that your husband, but what happens if you do go and live with him (taking your youngest) and it doesn't work out? We fall madly in love, don’t we, thinking (feeling rather) that it will last forever and is enough for a serious relationship. However, it’s very hard to live on an emotional high for very long. Eventually we start to come down from the ecstasy, excitement, strong passions and desires, typically after around 18 months to three years (people vary of course). If couples are friends, discuss their mutual values, shared ambitions, interests, etc., in some depth (obviously there will be some differences, which help make relationships interesting), and make plans, work on their personal development, etc., this can develop into a long and wonderful relationship. If one party feels insecure or low in self respect, it can make for a difficult partnership. It's easy to behave at our best when in love, but marriage, for example, requires a lot of self discipline, sacrifice, compromise and flexibility. If a strong friendship is not in place, the relationship will probably peter out eventually - or worse. Quite often we fall in love because we are lonely and allow ourselves to be won over by anyone who takes an interest in us. Thus we give away control to somebody else if we are not careful. This is another reason for taking things very slowly, and really getting to know someone before committing ourselves or getting too emotionally or sexually involved. Sex can be emotionally bonding, which is disastrous if the other things are not there: strong friendship, similar values and standards, common interests, etc.
Marriage counselling? Good Luck whatever you decide.