There are a lot of reasons for low immunity. Sometimes it can be genetic where you were born with a mutation that affects your ability to make enough of certain blood cells etc. or if not genetic it can be due to a lot of things like stress, malnutrition or lack of certain vitamins like A, E , zinc copper,Iron, selenium. But these usually can be found in a multivitamin because you only need the zinc etc. in trace amounts. If you are not absorbing nutrients right through the intestines that also possible. But, usually you see floating , fatty or glossy stools.
Another cause can be anemia. Have you had a complete blood count done? This can check to see if you have the right amount of blood cells used to make your immune cells.
Do you have any health problems other than these infections?
Well, the high IgE could be due to allergies or autoimmune disease. Not sure if the IgM is related to the same. Perhaps you should get some further testing done. If you have an autoimmune disease maybe it can lower your immunity and give your frequent infections. So perhaps see a Rheumatologist to see if you have one. It makes more sense than an allergy.