George R.R. Martin had a lot of inspiration for A Game of Thrones and A Song of Ice and Fire — the novels that inspired the HBO series — but one thing from his childhood inspired him most of all.

RELATED: George R.R. Martin Promises ‘Game Of Thrones’ Prequel Will Not Interfere With Finishing His Next Book

The bestselling author appeared on Global’s “The Late Show with Stephen Colbert” Wednesday night to talk about his new Westeros-set prequel epic Fire and Blood.

Colbert opened the interview by complimenting the “purple turtle” brooch Martin was wearing on his shirt and asking what it meant.

“I love turtles,” Martin said. “That’s how my writing career began, with turtles.”

The 70-year-old explained: “I lived in Bayonne, New Jersey, the federal housing projects there. We were not allowed to have dogs. We were not allowed to have cats. So the only pets I was allowed to have were turtles, little dime store turtles.

RELATED: George R.R. Martin’s Next ‘Game Of Thrones’ Novel Won’t Be Out In 2018

“The thing is about those little dime store turtles is that they die very soon,” he continued. “And I couldn’t figure out why they would die, but it wasn’t my fault, so I decided that they were competing for the Turtle Throne. They were killing each other to determine who would be the Turtle King. So that was my first fantasy, Turtle Castle, it preceded Game of Thrones by many years.”

Tune in to Global’s “The Late Show with Stephen Colbert” weeknights at 11:30 p.m. ET/PT, and watch full episodes online here.