Whenever me and my partner have sex It hurts is this normal?

I have been with my partner for many years now we've both done STI tests and we don't have anything however whenever we have sex with him it hurts.... sometimes during and most of the time after. We use lube when we need to but i find i am allergic to a lot of lubes out there which used to make the problem worse. The pain doesn't stay for very long after we have had sex just for about 5 min or so and then it goes away. Sometimes the pain is quite bad but could this just be because it was ruff? I am just wondering whether anyone has any advice because i don't feel it is something i need to go to the doctors about because it only happens during sex?. If anyone could help me i would be truly grateful.