He doesn't argue that he does in fact have somewhere to be. He tries not to feel guilty for abandoning the team, but he feels it anyway. Even knowing he'll be much more use to them if he takes the time to heal.

"A spaceship?" Ted all but yells excitedly, before Sherlock has even finished saying 'within reason'. Get used to it, Sherlock.