Based on my experience seeing friends and colleagues stop full-time work, the biggest challenge is to remain relevant and not lose contacts. While many relish having more time for other activities, everyone needs to be ready for the question, "What are you doing now?".