Calories don't make you fat. Eating food that stores as fat does. Bananas are very nutritious, but with fruit generally you want to eat moderately as it does contain a lot of sugar, which although slower metabolizing than cane sugar still will store as fat if not burned off fairly quickly. But not eating bananas at all would eliminate the incredible nutrients and fiber they have. Moderation is the key.
One 8-inch banana contains 120 Calories, which is about the same as one 40 g slice of white bread.
Bananas are soft and sweet, so it can be tempting to eat them few in a row. So, it depends how many of them you eat.