Prince Harry says his relationship with his father isn’t perfect.

In an excerpt from the royal’s leaked memoir, he alleged that King Charles III used to joke about who really fathered him with his mother Princess Diana.

“Pa liked telling stories, and this was one of the best in his repertoire. He’d always end with a burst of philosophizing … Who knows if I’m really the Prince of Wales? Who knows if I’m even your real father?” he recalled, via Page Six.

READ MORE: Prince Harry ‘Spares’ No Details, Shares Cheeky Story From Prince William’s Wedding

“He’d laugh and laugh, though it was a remarkably unfunny joke, given the rumor circulating just then that my actual father was one of Mummy’s former lovers: Major James Hewitt,” he continued. “One cause of this rumour was Major Hewitt’s flaming ginger hair, but another cause was sadism.”

Diana had a five-year-long affair with Hewitt after meeting him in 1986. He was dubbed “Britain’s biggest cad” after attempting to sell love letters between himself and the princess.

Harry believed tabloids latched onto the idea of his illegitimate parentage due to jealousy.

READ MORE: Prince William Called Meghan Markle ‘Rude’ In Heated Exchange After Her ‘Baby Brain’ Comment About Kate Middleton

“Maybe it made them feel better about their lives that a young prince’s life was laugh-able. Never mind that my mother didn’t meet Major Hewitt until long after I was born,” he added.

Other revelations from Spare included an alleged fight between Prince William and himself over Meghan Markle, as well as his attempt to recreate his mother’s tragic accident.