Better than a dumbbell use a plate!
First, are you doing full sit-ups or crunches? Almost everyone now advises against full sit-ups because of the stress it puts on the back. But it sounds like you've answered your own question -- pressing something hard (the dumbbell) into your chest might be bruising it. Try switching to something you can hold away from your chest, or do crunches or sit-ups if you must on an incline, using your body as your added weight, and see if that doesn't help after a time.