Getting to be part of Ashton Kutcher’s prank show had its downsides.

On Thursday night, “The Office” star B.J. Novak appeared on Global’s “The Late Show with Stephen Colbert” and talked about working on “Punk’d” and making Usher “furious”.

READ MORE: B.J. Novak’s Photo Is Being Used On Random Products Worldwide

“That was my first job, on camera. He changed my life with that show,” Novak said of Kutcher. “I was doing open mics at the time and I was like a dorky, khakis wearing kid and he was like the coolest person in the world. He wore a trucker hat and everyone in America wore a trucker hat. He was a trend setter and I wasn’t, in that moment, really getting anywhere or feeling like a cool kid.”

He recalls Kutcher calling him up regularly with last-minute prank ideas for different celebrities.

“It’s Ashton Kutcher, like, ‘I need you to be at the Van Nuys DMV tomorrow and know everything about the California Driving Code. Hilary Duff is taking her drivers test and you’re the instructor,'” Novak said, revealing that many of the celebs got “terribly mad” at him when the cameras were revealed.

“I’m meeting all these celebrities for the first time, right? It’s thrilling for me. I’m meeting Missy Elliott, I’m meeting Usher, [but] it’s the worst day of their life,” he laughed.

 

“I got to meet Usher but my job was, I was a store owner on Melrose and his little brother had been busted for shoplifting — he was in on it with us — and the only way I would let the brother go is if he recorded a rap jingle for my store, which I rapped for him,” Novak recalled.

“And he was like, ‘First of all, I’m not a rapper. Second of all, why does it refer to ice?’ And I’m like, ‘Well, we wanted Vanilla Ice,'” he said.

READ MORE: B.J. Novak Reveals Whether Kathy Bates Or James Spader Was More Intimidating On ‘The Office

“He’s furious and then Ashton comes out, and he’s like, ‘Bro!’ And [Usher] gets like a huge hug. And I’m like …” Novak said, with his arms outstretched. “And he’s like, ‘No, no, no, no.’ Like, your first impression of someone sticks, you know.”

Despite Kutcher putting him in some awkward positions, Novak cast him in his new directorial feature debut “Vengeance”.

“The Late Show with Stephen Colbert” airs weekdays at 11:35 p.m. ET/PT on Global.