Do not remove cast at 3 weeks as that is one of the possible ways of managing a fracture, just like operation.
Ideally bone unites in 4-6 weeks depending on fracture site, pattern and patients age.
The k-wires in your foot might be removed at 6weeks.
Next Steps
Share x-rays..
Health Tips
If you remove plaster by yourself, you might not be able to blame doctor later.
Consult your doctor.