Swelling on the upper lip. To worry?

So exactly a year ago, I got a button-like swelling on my upper lip. I was stressed and had eaten a lot of chocolates and nuts before that so everybody around told me that it was probably an allergic reaction. The general practitioner had given me anti-bacterial and it cleared up after that. Although none of the doctors that I consulted suspected it to be oral herpes, I still wonder if it was that. I've never gotten tested for it. I had a sexual encounter 1.5 years before that swelling erupted and I've not had any outbreak as such a year after that as well. Even when that swelling happened, it wasn't painful, itchy, or anything like that. Neither did I experience flu-like symptoms. Do I need to worry? I recently read that herpes can stay dormant in the body for years and it really concerned me. I'm attaching a picture from a year ago of the scar when it was healing. It's on the upper lip and button like.