Going vegan is great for the skin! Cutting out grains will help a lot too! Just try to get as many vitamins as possible and drink lots of green smoothies!
Some dermatologists do advise eliminating all dairy, so maybe for at least a trial period of a month or two eliminate all dairy - at least for milk there are a lot of decent substitutes with rice milk, soy milk, or even coconut milk. For supplements: take a good antioxidant blend, take a MAX EPA (fish Oil) daily, and some capsules of natural beta-carotenes (from carrots, squash, etc.). Wash with a nice mild soap, and get a little bit (15-30 minutes) of sunshine) on your face daily - the lesser amount if you have very fair skin. Be sure not to use any sun screen unless you have to, and be careful not to use sunscreen with PABA, which causes blemishes.
Skin problem are generally due to liver problems. What's affecting your liver is hard to know, but if you look at natural skin herbal remedies they're all a combination of liver cleansers. So yes, eliminating foods that are bad for you will certainly help, but toxins come from other sources as well. You might try a gentle liver cleanse as well as cleaning up your diet. Chili depends on what's in it -- the main ingredient is beans, which is fine. White flour turns to sugar quickly in the system, and so that's a good one to eliminate. Yoghurt is probably fine, but most people do have a problem with dairy. Fried food depends on how deeply it's fried and what it's fried in -- there's sauteing, and then there's deep frying, so all frying isn't the same. It's more complicated than you're seeing it.