Hi there. Four year olds CAN be hard! I had a tough time with both of my boys when they were four. Remember that they have immature emotions, still working on appropriate ways to communicate, and have a natural instict to push the boundaries.
It sounds like your boy is not handling frustration, anger or sadness very well. I would go to the library and check out books on emotions. There are a ton for kids and they give words that kids relate to and understand to express how they are feeling. Then encourage him to "use his words". There is another book that I highly recommend called "hands are not for hitting" which drives that point home. Give him other things he can do when he is frustrated, angry or sad. He can tell a grown up, try to tell the other child how he feels, go to a 'cool down' spot that is just for him and no one can bother him there while he 'gets himself together', he can count to 10, he can take deep breaths, he can open and close his fists tightly/firmly, he can use helping hands (use some finger pain and have him put his hands in it and press them against a piece of paper to make a hand print. Then hang it on a wall. When he gets mad or upset, have him go to his helping hands hanging on the wall and push against them. Very calming.) Even act these things out for him. Sometimes that is very powerful as kids see what you are saying in action. In fact, you can act out a whole scenario of something frustrating happening to you and go through the steps of handling it appropriately in an exagerated way. It does really help.
I would do your best to stay patient but very firm. I am a pretty laid back mom but hurting other kids was a deal breaker. I never spanked my kids but there were consequences for such actions! I tried to stay with the natural consequences--- if you throw a toy, that toy goes bye bye for a while. If you have a fit when out, then we leave immediately. If you are yelling, I would tell them "I will talk to you when you use your normal voice because I can't understand you until you do." and then ignored them until they used their normal voice. I used time out of either them or prized possessions.
I did rewards as well. When he is nice to brother, heap on the praise. If he uses his nice words, give him a hug and commend him for it. If he works out an issue in an appropriate way---- high five and maybe a nickle for his piggy bank.
One other thing with two boys I ran into is the fighting over toys. We had a rule that you couldn't take someone else's toy unless you traded it. So instill that rule.
good luck
May I ask when this behavior started ,was it when little brother was born, as this could be jealousy , he sees his brother getting the attention he used to get and it hurts .Make sure he is getting one to one time with you and his Dad get Dad involved playing sports ,ball games and having fun with him ,he feels left out. .Focus on his positive side and praise him when you see him doing something right .good luck