Ted shrugs, but not in a way that implies 'why not' or 'I guess', but rather that he actually doesn't know.

"I don't want him to scare me out of going," he answers. "And I want to be there for Rebecca." He pauses thoughtfully. "And I guess... I do kind of want to look him in the eyes."

He doesn't want to let him know that he knows. But he wants to see his reactions. And maybe... just maybe there's a little bit of spite in him too.