Past should not be disturbing your present. It's good to have a fair and transparent relationship . But doubts spoil everything . Be open and ask questions once for all . Don't keep them in your head and create stories. There is no point in keeping a relationship with someone you don't trust. Just once sit down, clarify and forget it.