Rapunzel is spot on. For all of the isolating and hiding feelings that Inigo does, he doesn't actually want to be alone. He sucks at being alone. Being alone only makes him feel worse, although he probably doesn't dare to admit as much even to himself.

So he hesitates. He knows he should just tell her to go. So she can do something more fun than just sitting here with some guy who's feeling sorry for himself.

But he wants her to be here.

And that latter part of him slowly starts to win out. It's soft - almost as if he doesn't dare to even let himself hear it, since it'll just make him feel guilty. But he does say: "I.. think I would like it better if you stayed."