It would probably be safe during this last part of your pregnancy, as far as your baby is concerned, but you should definitely clear this with your obstetrician before using the drug. You should also ask for suggestions regarding other non-sedating antihistamines. Benadryl is definitely contraindicated while breast feeding.
The risk to you and the baby is that this drug can make one drowsy and problems could arise for both of you in a number of circumstances such as driving, walking down stair etc.