Why was the cast removed after 4 weeks and not 6weeks?Also since there is no xray to see, can't comment on the fracture type.
Next Steps
See your surgeon again if you have pain or take a 2nd opinion after getting a fresh xray.
Health Tips
For fitness the fracture has to heal completely. Your doc will answer u better and is beyond the scope of this practo query page.