TSH is affected by so many things that it is not adequate as a diagnostic, only as an indicator, to be considered along with more important indicators such as symptoms and also levels of the biologically active thyroid hormones Free T4 and Free T3. Your TSH is high enough that it should cause the doctor to do additional tests. Specifically you should be tested for the possibility of Hashimoto's Thyroiditis. With Hashi's the autoimmune system erroneously identifies the thyroid gland as foreign to the body and produces antibodies to attack and eventually destroy the gland. As this proceeds the output of thyroid hormone is diminished, resulting in hypothyroid symptoms, and the pituitary increases output of TSH in an effort to stimulate more output from the thyroid gland.
To test for the possibility of Hashi's Thyroid Peroxidase antibodies (TPO ab) should be tested first, and if that is negative, then Thyroglobulin antibodies (TG ab) should be tested. Along with tht you should always be tested for both Free T4 and Free T3 every time you go in for tests. Note that those are not the same as Total T4 and Total T3, so be sure they test for the Frees.
Also, hypo patients are frequently deficient in Vitamin D, B12 and ferritin. So you need to get those tested and then supplement as needed to optimize. D should be at least 50, B12 in the upper end of its range, and ferritin should be at least 70, and some sources say 100.
The most important thing for you is to make sure you have a good thyroid doctor. By that I mean one that will treat clinically, for symptoms, by testing and adjusting free T4 and Free T3 as needed to relieve symptoms. Symptom relief should be all important, not just test results, and especially not resultant TSH results when taking thyroid med.
What was your doctor's reaction to your TSH test result?