You're probably letting your ex be a symbol, in you dreams. Dreams are visual and fast, so the brain uses visual shorthand in them. If someone or something (say, a dream of your old family house) means a certain emotion, what you are dreaming about is that emotion, not particularly the house. Your mind just cooked up a fast dream symbol (the house) to get you to that emotion quickly.
In your ex's case, it sounds like he represents the idea that something has been lost to you before you were sure you were ready for it to be THAT lost. Processing old loss when its only 3 years ago is not too surprising.