If your hands are getting thin in a way that is noticeable, probably that simply reflects the fact that your entire body is getting thinner, and does not have anything to do with what you are doing sexually 4-5 times a month.
The way to put on weight in a healthy fashion is to continue to eat a healthy diet but just to eat more of it than you are now (meaning, now when you are losing weight). In the rare case where the issue is that the person is so active that he is burning off all of his food, it might also help to be slightly less active. But for most people who lose weight not on purpose, having an excessive activity level is not the problem. (Exceptions would be people who get obsessive about exercise, like ultra-long-distance runners or exercisers who do it compulsively, or someone whose job is so physically demanding that they can't eat enough to keep the weight on. But those exceptions are rare.) Generally, since being physically active is great for the body and the key to health, reducing activity level is not the best way to gain weight. That leaves eating more as the way to rebuild body bulk. Because you are not fat, this suggests that you mostly eat food that is healthy for you, so just ramp up the quantity and you should see the lost weight return. You can also probably find a lot of advice online or from your doctor about healthy diets for weight gain.
In case this makes no sense to you because you already eat a great deal, see your doctor and be sure you don't have a physical condition, ailment or even parasite, that might be the cause of the weight loss.
Your sexual life will not be the cause of weight loss. Having orgasm with your girlfriend represents significant life events for you, but 4-5 times a month is not enough to affect a person's weight AT ALL. The only time someone's sex life would make them lose weight is if something about the sexual activity makes them so upset that they get an eating disorder like anorexia. That does not sound like what is going on for you.
Good luck with your diet for weight gain, and enjoy being the envy of a lot of people who can't lose weight.