Relationship counseling. I have issues!

Hi. I know a person from 10yrs. We got into relationship few yes after v met. We were together for 3 years. But we had many issues like i have issues regarding his past affairs...he is very practical and lil money minded. He needs logic for everything. He has anger issues...i told my parents about us. They didn't approve for various reasons like caste money status. I fought for almost for 1yr. Meanwhile we had many fights n I decided it wont workout between us. Though he tried hard I broke up n at da same time had too much pressure from parents so i got married to someone else. Now im married for 3 yes n have a one yr old son . I still have feelings for my ex. I still love him. I got scaredy-cats things wont workout n backed out. My husband knows about my past. He said if I still hav feelings we should divorce n I should get back with him. Im not able to decide. If i go out i might be happy but il be hurting many families. If i stay i might regret later. My husband is very gr8 person