I feel like there's two ways we can look at student behavior.
One is that they are doing it on purpose and it's directed at us.
The other is that they are struggling with someone/something and it has nothing to do with us.
Asking a student to join me for lunch/recess may be viewed as a 'punishment' for some but I look at it as another opportunity to understand them.
I love asking a lot of questions to better understand their perspective, home dynamic, daily routines, and what they enjoy about life right now.
I've learned that while students are upset they are missing time with friends at recess, they quickly forget about it and seem to appreciate someone asking about their life.
After all, who doesn't like feeling appreciated by someone asking about how they're doing and what they're interested in?
Use the opportunity to build a relationship, and your credibility.