Klonopin and valium have longer half-lifes than xanax and some other benzos. That does tend to make the withdrawl process somewhat more tolerable. The fact you are more stable makes starting the withdrawl process more favorable as well. I had to get weaned off of valium by using seroquel which did help. It can be done, it takes patience but the good days will eventually outnumber the bad ones. When that happens, you're on your way. Good luck.
There's no way to know how it will affect you. Some have very hard times of it, some have very easy times of it, some are in the middle. The only way you'll find out is if you do it. The most important thing in quitting any of these meds is to have a psychiatrist who understands withdrawal and how to handle it if it happens. It sounds like you have that. People here can tell you how discontinuing it was for them, but that won't necessarily apply to you since you're a different individual. I think since you're forewarned about what might happen and have a shrink that will go down this path with you, you're probably already well ahead of most in making this decision.