It speeds up your anxiety. Caffiene is a stimulant — and that can be bad news for someone with anxiety. Caffeine's jittery effects on your body are similar to those of a frightening event. That's because caffeine stimulates your "fight or flight" response, and studies show that this can make anxiety worse and can even trigger an anxiety attack.