Yes, you “should look into things.” It is not normal to have repeated bouts of fever. RSV in infancy is common and may predispose to asthma or frequent lung infections. You should ask his doctor if there is any sign of lung damage from the RSV. The good news is that, in most instances of RSV resulting in frequent bouts of croup or other respiratory infection, that frequent lung problems with fever are eventually “outgrown.”
Definitely seek answers to your questions from his doctor and, if not satisfied with the responses, request consultation with a pediatric lung specialist.
I strongly suspect that, with the passage of time, your son will experience better health.