Take up running 3 times a week and that may help your memory. If it doesn't, see a doctor for lab work. Maybe you are anemic or something.
Masturbation in limit will not harm your memory. It may be that you are worrying too much about the masturbation and this worry can cause any type of health problem. It is a natural habit.Worrying about it harms you more than the act.