Buspar isn't an addictive drug. To be labeled addictive, a drug must meet certain conditions. But withdrawal has nothing to do with addiction. Some people who take addictive drugs won't have any withdrawal problem, some will have a whale of one. But antidepressants, for example, are not addictive drugs but pack a wallop of a withdrawal for some people, a mild withdrawal for others, and no withdrawal for a few. So don't confuse withdrawal with how addictive a drug is. It's a good idea to not get addicted to drugs if you can help it, because they're almost always hard to stop taking, but withdrawal, again, is something that can happen with drugs that aren't addictive. The biggest problem with Buspar is that it actually has never done well in its clinical trials, and has been a drug looking for a condition to treat for decades. It is, however, considered generally easier to stop taking than, say, benzos, but mostly because it usually doesn't work, most likely. Nowadays it's mostly used in conjunction with an antidepressant. Here's the thing with drugs, especially those that affect the brain: nobody can tell you how you'll react as brains are very individual things and not very well understood. There is a reason, though, that many doctors are getting more reluctant to use benzos, especially in England. They are addictive, and the gov't is once again getting all heated up over addiction. This is a cyclical condition of people who like telling us what to do and not do for personal moral reasons and for genuine health reasons. And another thing about benzos which is most pressed by researchers in England, benzos are believed by many researchers, when they're used regularly for a period of time, to interfere with the brain's ability to adapt to stress, even after you stop taking them.
Meds affect everyone differently, so what you are hearing from people who take it is just anecdotal and may not apply to you.
Additionally their opinions may be subjective and not reliable, so I would place more faith in a drug trial result you get from a manufacturer of the drug because that has to be reviewed by the FDA. People who take medicines are mainly not medical professionals, so may have trouble coming up with scientific analysis anyway.
The counselling is a good idea so stick with it.