Testing for Psoriatic Arthritis

I have had psoriasis since I was maybe 6 or 7 years old.  I'm now 24, and it covers about 20% of my skin.  Four years ago, I started struggling with back pain. Walking down the stairs was (and still is) rough on my ankles.  I have two big sausage toes and my ring & middle fingers curve together (it's painful and looks weird, especially with the little dents/holes in those nails).

I went to a rheumatologist once, when I was 20.  He sent me in for X-rays (results: abnormal neck curvature), then an MRI (results: degenerative disc, scattered points of inflammation) and he did blood testing.  I was negative for rheumatoid factor so he said it was probably just depression.  I was sort of offended because ... it sort of makes me sad that I have these lesions, and it doesn't help that he repeats to me how dreadful they look... but I just want a solution for the ache.  He asked me back for more scans, but I didn't go.  

My decision to up the low-impact exercises and eat a healthier diet hasn't worked, so I'm considering further testing.  

Does this sound like psoriatic arthritis?  Does the negative rheumatoid factor mean that I DEFINITELY don't have it? I'm frustrated.  What should I do?