I know I am still new here and have not earned my place yet, but SL is simply stating a different perspective and thinking towards the question at hand. One that I don't think warrants a lynching... Juss my two cents...
As for where we stop, I agree with sfem. Although, I can also see growth becoming somewhat of an addiction, so the question still stands. This is where I agree with SL. You need to analyze your intent and find the root cause for your need to grow, and your intent and variables needed to be considered will reveal the point where is best to stop. I don't think it is as simple as a big red button for stop and go either. This is something where while you gain something, you lose others. Sacrifices are made, and you need to find the point where the cause is worth the sacrifice. The comfortable spot. I don't think anyone can point that out for you, and even you will have a hard time seeing or knowing this "stopping point" if and when it comes. Your goals may even change at said point.
It's a great question, but I think the answer wont really reveal itself unless you already know. The best thing you can do is set goals for yourself, and when you reach said goals, re-analyze, and go from there. If you have outside perspectives (friends/wives) to help analyze for you, your judgement wont be as clouded by your own mind. If you feel the need to continue at that point, by all means. Happy growing!