With the limited information available to me about your personal history of habits and health, my answer would be more empirical. The kind of fear you have mentioned is common in people who abuse alcoholic beverages. Regular or frequent intake of alcohol (in any form), even in lesser quantities leads to sub-clinical dependence. Such people will be in a chronic state of withdrawal; they feel better after a drink otherwise they are on the edge... nervous and irritable. Memory also begins to get affected because alcohol, whether beer, wine, or cool drink mixed variety, is toxic to the brain cells. Added to this most people who drink also smoke, which worsens the condition.
If the above is not applicable in your case, you might be suffering from depression; anxiety and impaired memory could be secondary to depression. You have not mentioned whether you are single or married, and whether you live with family or alone. All these factors play an important role in a person feeling stable and secure, or insecure, lonely, fearful and depressed. Lack of sleep also contributes to anxiety.
These apart, vitamin and mineral deficiencies, sub-clinical hypothyroidism, early stages of diabetes should also be ruled out.
Please consult a psychiatrist in person and discuss your case. You may also request him to get your blood tested for the above or any other medical conditions that he may suspect.
In the meantime please practice pranayama and meditation under the guidance of a yoga master. Learn the art of yoga. Start walking half to one hour every day. Keep company of good and sober people; not the drinking and drugging type. Stay a hundred miles away from drinking and smoking. Avoid late nights. Sleep 6 to 8 hours every day. You will soon start smiling and enjoying your life.