You have post fracture stiffness , have you undergone ankle rehab? If not then please do. For 4 weeks your ankle was in cast so your muscles and facia have become tight when you will visit physio he may give you paraffin wax bath (wax therapy) which will reduce your stiffness and swelling, the same time therapist will give you some stretches (calf and hams ),then dorsiflexor muscle strengthening. Do this for 2 weeks you will have negligible pain stiffness later do balance board and trampoline training which will finish your rehab. There is a standard  protocol in physio literature. Find a good physio, you'll be surely get normal movement at ankle.

Hi put up your present Xray.

Stretch Calves Get a custom made insole. Hams Stretching. Sports Taping. Hot Fomentation.

There is nothing in wrong that.your ankle was immobilize 4 week, so mobilty flexibility and strength is decrease. Cosult your physio to work on this to improve range of motion of ankle joint. Slowely your pain will disapear. At home put your leg in hot water bucket will relive your ankle pain. This you can do 2-3 times in a day.

You need to provide the present x-rays to suggest you more. Pain on weight bearing suggest either fracture has not yet healed or may be due to stiffness. Kindly consult your treating surgeon.

Don't worry. Its Normal. But Don't be late for Normal Range Of Movements.