Welcome to the forum! Your usage is not considered A LOT, but you still may want to wean down to make it safer and less uncomfortable.
There are many tramdol "warriors" as they call themselves that have started a thread of their own by using a journal. They are on Part 55 now and could be really helpful to you. Here's the link:
http://www.medhelp.org/user_journals/show/571061/-Tramadol--Ultram-Recovery-Room-Part-55
I took Trams for back pain at one point in my pain journey, and the med guide said 50 mg 4x/day (max 200 mg). That's probably why your dr doesn't think you are on a high dose. Nonetheless, many drs don't even advise their patients that it IS addictive. Hope you'll find a way that works for you. Because it is a synthetic opiate AND an antidepressant, it has the possible w/d effect of a seizure. Doesn't mean it will happen to you, and your daily dose is "considered" low, but if you want to get off them completely, you may want to wean yourself. The tram warriors are a great bunch and they all have fabulous support and experience.
Blessings to you~