Yes and no. :-) By four years of age, certain male behavior is already established. An unaltered male has had plenty of testosterone in his system so he's probably developed a dominant personality. Not all unneutered males are dominant, but the likelihood is higher. Do not confuse "dominant" with "aggressive". Those are two entirely different behaviors.
Still, neutering will help your dog calm down and you also won't have to worry about testicular cancer later in life. If he constantly gets out of your yard when he smells a female in heat, neutering will help that as well.
Exactly what issues are you having with him?
Do you mean make him less agressive or become lazy?