See an Eye MD soon. Find one at www.aao.org
JCH MD
Yes you need to be worried. This could represent serious disease of the brain, heart, blood vessels or eyes, it might be a TIA transient ischemic attack. Best thing it could be would be eye migraine (which don't have headaches often)
JCH MD