He's torn. Part of him feels like he should just let Yuki go if he wants to go, but on the other hand, even just hearing him saying he's going to leave instantly makes Inigo feel bad. Did he do something wrong? Was it something he said? Considering how awful he often is at properly conveying himself he feels like that has to be the reason for Yuki's sudden odd behaviour.

But one thing is clear. He can't just quietly let him leave. He'd feel awful. He already feels pretty panicked as-is.

"Hey, Yuki--" He makes sure to hold onto and squeeze the other's hands as he says it. "I.. Of course you can go if you want to? Or if you're not enjoying the party?" He wouldn't want to stop Yuki if he's just bored or something, but still.. "But I just want to make sure you know-- I want to. I really want to go to your world with you. You know that, right?"

Inigo's gaze is so desperate, worried that somehow he made it seem like he doesn't want to.