This is limited details and more precise information is needed about your skin condition which can help in planning a treatment. A thorough history, clinical examination and appropriate laboratory tests will be needed to solve the mystery. A photograph would have been helpful. Consider seeing a Dermatologist for proper diagnosis and treatment plan.You can try chemical peel for salvage of the present lesions but it will not address the core issues and new KP lesions may erupt
Your dermatologist will decide a treatment plan for you after a consultation.