Hormonal imbalance or low testesterone level may cause erectile dysfunction , low sperm count and gynecomastia. It is treatable but it is important to understand the underlying causes for each condition. Your family doctor is sure about breast swelling which is actually fatty breast tissue and not gynecomastia. And if your family doctor has done thorough check up then it can also be fatty breast tissue. Some men and boys have chest fat that resembles gynecomastia. This is called false gynecomastia (pseudogynecomastia), and it isn't the same as gynecomastia. Please consult a specialist and do required test to rule out such condition.