The *NSYNC singer thought the diss was about the whole band until his phone started ringing incessantly

Chris Kirkpatrick knew boy band life meant sometimes being the butt of a joke.

But the *NSYNC member never expected one of the most popular rappers of the early 2000s to target him directly. However, that’s what happened in 2002 when Eminem released his single, “Without Me.”

“We did a show in Miami and we were coming back and I started getting all of these text messages, ‘Have you heard the new Eminem song? He makes fun of you in the Eminem song,’ ” Kirkpatrick, 53, recalls during his appearance in CW’s The ’90s Boy Band Boom.

The pop singer says he reasoned, “I love Eminem and *NSYNC rhymes with everything. *NSYNC, stink, you know. It’s really not that hard.”

“And they’re like, ‘No, no, no, you. Chris Kirkpatrick.’ And I’m like, ‘Chris Kirkpatrick? That doesn’t rhyme with anything,’ ” he continues.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

“And as it was coming out of my mouth, I’m saying, ‘This is Eminem.’ And then I hear, ‘Get your ass kicked,’ and I’m like oh, yeah.. I guess it rhymes with that. That makes sense.”

He continues, “And at that moment, I was like, ‘What does this mean? Do I have beef now? Is this, are we beefing? Man, that sucks.’ ”

At a time where it became popular to poke fun at boy bands, Kirkpatrick says it gave the industry an opportunity to dismiss the success of the group, which also included JC Chasez, Joey Fatone, Justin Timberlake and Lance Bass.

“A lot of that pop genre didn’t get that recognition we deserved. We were up for a couple of Grammys. The one that kills me the most was when we lost to Steely Dan for Best Vocal Performance on a Record,” he recalls.

“We strived on being in the studio way longer than we should have to make sure that every detail of every song, every part was perfect. It was before autotune. We had to sing it right. We had to make it exactly what it sounded like. It almost sounded fake, we put so much effort and work into these sounds.”

When it came down to it, Grammy voters determined, “They don’t have the credibility. They’re a boy band,” Kirkpatrick recalls.

“And that hurts because that’s, in our business, the Grammy was always the coveted award you won in recognition for what you do,” he explains. “The term boy band doesn’t sound like anything that has credibility or anything that’s got a basis in maturity, in rehearsal, in effort that we put into things. It comes out as dismissive.”