If you are about 170 pounds and have muscle and strong bones then you are at a healthy weight for your height. You probably don't need to lose the fat. My best advice is work on your body image issues and self esteem instead of trying to lose fat that probably isn't there. It is healthy for women to have a natural fat layer on their bodies for pregnancy.
The reason the "weight" doesn't stay off is probably because your body needs it to be healthy. Just focus on your health. Continue doing yoga and cardio because you love doing it, not for aesthetic reasons. Lift weights to develop more muscle, which will increase your metabolism, which allows you to eat more food. You need to eat more than fruits and vegetables for optimal health. Follow a well balanced healthy mealplan that consists of healthy carbohydrates, protiens and fats from all the main food groups. 5 pounds is nothing in a week, it can just be water weight or weight lost from a bowel movement alone. You won't lose fat that quickly, you have to give it more time. If you aren't eating enough carbs you will be losing muscle and water weight, not fat.
Are salads, vegetables and fruit all you're eating? What about lean protein, dairy, healthy fats? You aren't getting nearly enough nutrients that your body needs. You could actually use some protein and healthy fats more than you need the fruit, which has a lot of sugar in it, even if is natural. Yogurt is chock full of, both protein and healthy fats, as are eggs, lean meats, fish, avocados, peanut butter, etc.
If you've lost 5 lbs in a week, that's too much... safe weight loss is considered to be 1-2 lbs/week. Most people who lose more than that, don't keep it off long term.