first of all the technical words you use (anxiety, panic attacks) may be misinformed. please allow professionals to do the labeling and diagnosis part. and therefore please avail support and advice from qualified professionals.
ones confidence shouldn't be a result of acknowledgement in one aspect of life (acceptance/approval by others), it should be a balanced consequence involving all aspects of your life. for example you are not including the success you have in your area of work in your self confidence index. and therefore when one focus too much on just one aspect of life, one tends to feel imbalanced,
relationships requires certain skills which can be learned and enhanced. it also requires positive thoughts about self which needs to incorporated and also acceptance of ourselves and our attributes and building up on our positives and working on our under developed areas and skills.
more you avoid, even though its an effective short term solution to feel safe, in the long term it will only diminish your confidence and make your poorer at your social skills which will in turn lead to more social discomfort and disappointments which will lower your confidence and contribute to more anxiety. so break the cycle and skill up and accept who you are and what you are irrespective of changing variables around you.
Next Steps
meditation will help you contain anxiety; start socializing more with small groups of friends. dont avoid social encounters