I can understand your concern. Do you already have kids? Or you are trying for the first.
If you already have kids, then the question of infertility is put to rest.
If you don't, then I think you need to consult a infertility expert on this and get yourself as well as your husband checked out.
The problem could be in you or in his sperm. A sperm analysis will be required for him.
Better to not waste any more time and consult a expert on this as soon as possible because you have been trying from long.
I hope my answer was of help to you. All the best.
Dr. Anil Kumar Jain.