This is a short-acting drug, so it wears off quickly. Taking a benzo for sleep on a regular basis is probably not the best way to handle sleep -- this is an unwanted side effect of the med to make you tired. Not that it doesn't work for people, but you become dependent on it instead of solving your sleep problem or using something gentler that might work for you. These drugs tend to interfere with REM sleep, so they can make over time more anxiety. But it is a low dose, and a panic attack is you triggering a lot more angst than trying to sleep.
.25 mg is a fairly small dose, so that might be one reason. Xanax and Ativan have a short half-life (around 1.5 hours just like you wrote) so it should be no surprise that the symptoms might return after that length of time. You really shouldn't be afraid of taking Xanax or Ativan as needed. I used to take Xanax on occasion when I was younger and now take Ativan as needed. Both are very commonly prescribed and usually very well-tolerated.