Puberty for menpenis size. I consult?

Hi! I'm 20 years old. I have lots of pubic hair at the base of my penis and noticeable naval hair. And few thick hairs around my groin area. I'm starting to grow 5-6 facial hairs under my chin. I've stopped growing in height for the past two years. I smoke cigarettes and drink occasionally. My penis hasn't grown much in girth as it did in length. When erect my penis is about 4.5-5.5 inches long but when it's not erect it looks very small like a baby's dick. I'm scared that it will never grow. Is it because of the smoking ? I'm pretty sure one of my parent was a late bloomer. When I masturbate I ejaculate semen. But my dick hasn't grown in girth. Growing in length has also stopped in the past one year. I'm 20 years old and I'm turning 21 this December. Should I be worried ? If I should be, which line of doctor should I consult ? I have hairs on my testicles too. But I don't see any changes in the size of me penis. Please reply cause I've been really worried about this for a long time