Dear Stephen,
With any display of physical aggression, time out should be employed immediately, coupled with solitary play for a period after the time out has been completed. If you need ideas about how to implement time out, use the SEARCH function on this site and you'll find a number of replies that address the issue. Over the years, when I've heard parents comment that time out 'doesn't work', I usually learn that it's the implementation that is the problem, not the technique itself.
Now, it's not unusual for a four-year-old to display some aggression - one of the tasks at this age is to learn to manage anger and other emotions. This is a process, and it will take some time.
Your technique of taking a toy away permanently is not going to be useful. Children at four years don't have the capacity to store the 'message' in a way that allows them to learn from it. Their life is made up of more immediate concerns.
If a child of four mistreats a toy, the toy can be removed for a time (i.e., no more than one day). This can help a child learn to take care of the item, and it conveys the message that you won't permit mistreatment of property.
If you examine some of the replies to past questions, you'll see that I frequently recommend a book called S.O.S.: Help For Parents (by Lynn Clark, published by Parents Press). You will find in it a reliable, practical approach to behavior management, ideal for addressing problems of noncompliance and aggression.