Jason Isaacs was surprised he was allowed to improvise on the ‘Harry Potter’ movies

British actor Jason Isaacs joined Jesse Tyler Ferguson for a meal and a chat on the latest episode of Dinner’s on Me. The pair met at the London vegetarian restaurant Tendril (Isaacs is vegan, but Ferguson said he loves a challenge) and they discussed several contours of the White Lotus star’s career.

Naturally, it’s tough to speak to anyone involved in the Harry Potter franchise without bringing up The Boy Who Lived, and Isaacs was eager to reflect on his time as Lucius Malfoy. The character appears in five of the eight films (six if you include photographs), the first being Harry Potter and the Chamber of Secrets, directed by Chris Columbus.

Of the experience, he said, “The thing that was very surprising is how much leeway we had to improvise and change things.”

He continued, saying, “Jo [J.K. Rowling] recognized they were films and not books, and it’s a different medium. She wasn’t on set. I didn’t meet her for years…and on my very first day, I was improvising lines. My very first shot was walking out of Dumbledore’s office from the end of Chamber of Secrets, and where he just dismissed me and I leave and I went, ‘I don’t know, would this guy leave without saying something?'” 

Recalling the scene, he said, “Harry Potter’s there, [and Lucius] is a very proud, arrogant man. Surely, he’d say something trying to get status back in, and so we’ll try something. And I made up a line, and Daniel made up a line back. And I thought, ‘We’re allowed to make s— up in Harry Potter.'”

Sign up for Entertainment Weekly‘s free daily newsletter to get breaking TV news, exclusive first looks, recaps, reviews, interviews with your favorite stars, and more.

About his onscreen foil, Isaacs had nothing but kind things to say about Daniel Radcliffe.

“I always knew he was smart,” Isaacs said. “I thought he would, and I still think, he’ll end up being a director as well as an actor. He was always so grounded and so mature and humble, and he’s continued to be ambitious, really ambitious, and take risks as well. He’s like, I’ve been famous. I’ve been rich. I’ve done all those things. So now I can do interesting independent films.”

Radcliffe, who lives in New York City, has deliberately avoided franchise films since the conclusion of the Potter series, appearing in several oddball movies like Horns, Swiss Army Man, The Lost City (a masterpiece, by the way), and Weird: The Al Yankovic Story.

More than that, Radcliffe continues to work on Broadway and Off Broadway, in well-received productions like revivals of the musicals How to Succeed in Business Without Really Trying and a Tony-winning performance in Merrily We Roll Along, plays like Equus and The Cripple of Inishmaan, and London productions of Rosencrantz and Guildenstern Are Dead and Endgame. He has, however, yet to take a seat in a director’s chair.

For more with Jason Isaacs and Jesse Tyler Ferguson, pull up a chair and join them on the latest episode of Dinner’s on Me. (You just have to cover the tip.)