It's not the judgement he's worried about as much as he just doesn't want anyone to feel sorry for him when they know the full extent of his circumstances.

Explaining that though feels like even more of a chore than explaining said actual circumstances, so after a moment of hesitation he looks away from Prompto, instead staring ahead of them into the empty space.

"My father was killed when I was very young. After that the kingdom completely fell to ruin and destruction. My older sister at least tried to salvage things, but I've been completely useless. Maybe things wouldn't have come as far if there had been someone more competent in my place."