The ear drum is one concern. Another is the how the normal sound is transmitted to the inner ear, and by moving the drum, even near unperceptive amounts, will damage that system too.
The drum moves in and out. this drags the Malius bone with it along with the Incus and then the Stapes. The way they work is through leverage to move the round window more than the drum did, something like 10:1 or so. By touching the drum with a q-tip, you will move the drum 100 times more than the loudest sound you have ever heard. Now the round window is moved thousands of times more than it can handle and now you have surgery to look at, all from the tinniest touch of the drum.
Doctors say not to use a Q-tip any further than in the outside of your ear (the rule is "never put anything in your ear that is smaller than your elbow,") but everyone breaks that rule. I think the main concern is that someone could puncture an eardrum, but avoiding infection would be a good secondary reason. Ear infections are more often caused by swimming, colds and sinus infections, and baths.