Hi Dear This is a really sensitive topic and it's difficult to handle children at this age because they are very inquisitive to know things. The more you say words like "no" or "I'll tell you later" it will urge him more to know. Kids are very intelligent at this age, so if u try to make him understand he will. Try and tell him the positives about his father( it is better to know the whole scenario first). May be you can tell it in narratives( stories). Get a direct consultation or meet a psychologist so that he/ she will better understand your perspectives. Take care.

Tell him the truth.  Because,  you and your ex husband werein Disagreements . He child has all the right to Interact and meet his father. Let him understand his father in his own way.  Not the way you describe about him.