Yes you need to be worried. This could represent serious disease of the brain, heart, blood vessels or eyes, it might be a TIA transient ischemic attack. Best thing it could be would be eye migraine (which don't have headaches often)
JCH MD
See an Eye MD soon. Find one at www.aao.org
JCH MD