Anxiety and stress can affect your normal biological functioning, You need to assess exact cause of your infertility whether is it hormonal, anatomical or psychological.

There are many cases where the stress leads to inability to conceive. But with the help of evidence based Psychotherapies and relaxation techniques you can easily overcome this anxiety. 

Body and Mind goes hand in hand! Mind i.e brain or specific parts of Brain affect each and every concerned or respective body parts. For e.g. Reproductive harmones are also under the control of harmones secreted by related parts in brain. Basically all of these things can be summed up in one word...i.e 'Neurobiology'. So coming to your concern, yes, it can be said that Anxiety and stress can affect your reproductive system. But rest assured, its just 3 month you are trying to conceive. Definitely infertility can not be labelled in this short duration.

Infertility and stress can affect each other. Infertility and process of trying to conceive can cause stress. Additionally anxiety and stress have an effect on body and hormonal cycles.