Downton Abbey’s Hugh Bonneville hasn’t let period-drama fame go to his head.

“He’s very much a costume I put on,” says Hugh Bonneville unsentimentally of his alter ego, Robert Crawley, 6th Earl of Grantham. We’re discussing the disappointingly 21st-century state of the Bonneville grooming. There’s a T-shirt under his blazer. There’s facial hair, in preparation for a gig playing a hobo, that would have poor Bates sharpening his cut-throat razor were he not currently languishing at His Majesty’s pleasure, seemingly auditioning for the lead in Les Misérables. Amid the luxury of an Auckland hotel lounge set aside to receive the press, Bonneville looks as defiantly underdressed as Downton Abbey’s rebellious chauffeur-turned-in-law, Branson, when he joined the affronted Crawleys for dinner in his Irish-radical-hothead tweeds.

Post-Downton, people expect certain standards to be maintained above stairs. They want the slightly clenched look of a man whose stately oasis of privilege is about to be run over by the Mack truck of the modern age. They want a suit, a tie. They want a stuffed shirt. “I think they’re disappointed when I turn up and I’ve got a shaggy beard and jeans,” Bonneville muses. “They expect me to have a Labrador permanently at my side.” Downton’s faithful if under-disciplined Isis would never make it past border biosecurity. But apart from looking more like a teddy bear than a toff, Bonneville is walking the Downton walk. His itinerary includes a high tea to raise money for Starship Hospital and a royal progress to the provinces – well, Palmerston North. It’s a bit like a visit from the Queen, if the Queen was looking a bit rough.

He is somewhat posh in real life, is he not? “I suppose I went to a boarding school. My father’s a surgeon,” he explains with slightly exaggerated patience. “If that makes me posh, then I’m posh.” It’s almost a relief to locate a little defensiveness beneath Bonneville’s implacably charming exterior. His bio does have its rarefied interludes. His godfather was former Governor-General of New Zealand, Sir Arthur Porritt. “He won a bronze medal in the 1924 Olympics in the 100 metres in the famous Chariots of Fire race and went on to become a great teacher of surgery. Taught my dad everything he knew.” And Bonneville studied theology at Cambridge. He went in an atheist and came out agnostic. “I had great respect for them,” he says, of his teachers, “but I haven’t made the leap of faith myself.” He was in little danger of becoming an academic. “I was a rotten student at Cambridge because I spent all my time doing plays.”

Bonneville is amiable, loquacious and reflexively self-depreciating. Asked by the Guardian how he’d like to be remembered, he suggested this Eeyore-ish epitaph: “Oh, him. Oh well.” Yet Downton’s writer, Julian Fellowes, has said that he thought of Bonneville because he has “a comfortable self-esteem”. Mention of this has Bonneville assuming the look Robert Crawley gets when he’s about to lose Cora’s fortune in a dodgy Canadian railway investment. “I dunno,” he sighs. “Julian Fellowes is a Tory peer in the House of Lords. He has an enormously liberal sensibility but with a sense of tradition and place and structure and order. I can see that in Robert, definitely. Whether it’s in me as a person, I don’t know. I certainly don’t rule the roost as someone like Robert does.”

He doesn’t entirely buy the idea that Downton appeals to nostalgia for more socially sedate, if more feudal, times. “I think it’s far more complex than that, and in one way, much more simple. It’s just good telly.” Certainly, Downton operates on many levels, some quite mad. There’s the cult following that has seen the bracing pronouncements of Maggie Smith’s Lady Violet – “Don’t be defeatist. It’s very middle class” – starring on YouTube as “Sh!t the Dowager Countess Says”. And British actors are unassailable when it comes to delivering such vintage endearments as “I love you, hard as it is for an Englishman to say the words” with a straight face. “Ha, not all the time,” says Bonneville. There’s considerable cracking-up on location at Highclere, apparently. Well, isn’t part of the appeal of Downton that it’s all a bit … silly? “There is something inherently silly about acting, let’s face it, be it period drama or set in a spaceship,” says Bonneville, giving the question a deft swerve. “It’s all daft. It’s all just trying to pass a bit of time and hopefully pay the mortgage.”

Fair enough. You wouldn’t want to come across as too earnest about your craft when you are about to play a character called Mr Stink. A BBC version of David Walliams’s children’s novel accounts for the beard. It must be said: he doesn’t look like anyone’s first choice to play the homeless guy at the bottom of the garden. “Well, it’s not quite all that it seems. You discover things about his past, which, I think, is why they thought of me.” Possibly, Mr Stink turns out to be the Earl of Grantham on hard times. Not that Bonneville doesn’t take the day job seriously. He will elegantly about the finer points of Downton until you make his stop. The visual language, the muscularity of the dialogue – “You tinker with Julian’s script at your peril!” It’s all quite stylised, he agrees. “You don’t see Maggie [Smith] stuttering and umming and ahhing during the middle of her lines.”

All that loving period detail. Costume drama porn, some say. “Yeah, absolutely. Exactly.” He sees the funny side. “Apparently there’s a drinking game that whenever the Dowager has a put-down, there’s a drink. I think there should be one every time someone says, ‘Can I have a word?’” Irony is fine by him. “The only level you shouldn’t watch it as is a social documentary or social history. Some people really like to sharpen their knives about how x, y and z never would have happened in that era.” Indeed. Viewers roared when Ethel, the maid, dropped in a very street “I’m just sayin’” on a bemused Mrs Patmore. “Oh, really?” says Bonneville. That one got by him. “We don’t want to get into ‘whatever’ and all that,” he says. Equally, he has no patience for pedants. “Funnily enough, there weren’t HD cameras in 1912, either, but we have them in the room,” he says. “We’re making telly.”

Which is not what Bonneville set out to do. He joined the Royal Shakespeare Company in the early 90s, but after a year wasn’t asked back. “I was in seventh heaven, doing Hamlet with Ken Branagh, playing Laertes …” he says wistfully. “I was dying to stay on but it didn’t come to pass and I was pretty devastated at the time. I thought ‘I’m never going to work again’, and, ironically, that’s just when TV things started to happen a bit.” There has been film and television, from Doctor Who to Lost in Austen via The Vicar of Dibley. He’s made a very successful career out of being not-quite-Colin-Firth. His most overused phrase, he told the Guardian, is “I didn’t want the part anyway”.

One he did want was in a drama called Legally Mad by Boston Legal’s David E Kelley. “It was to play a singing, dancing head of a Chicago law firm who played the ukulele, none of which, strangely, are on my CV.” They made a pilot in LA. “Driving through the gates of Warner Bros every day, I thought, ‘This is great.’” NBC didn’t pick it up. He also did a BBC sitcom, Freezing, one of three times he has played the husband of his Downton wife, Elizabeth McGovern. It was hilarious – think Extras meets Curb Your Enthusiasm – but it was canned after three episodes. Such vagaries have no doubt helped him develop that brooding, vaguely vexed look Lord Grantham has made his own. No one can bark “Nothing’s the matter. What should be the matter?” when something is clearly the matter like Bonneville.

Downton has certainly upped his celebrity. “I suppose, yes, recognisability. People tweeting ‘Hugh Bonneville has bought a load of pineapple in the supermarket. Isn’t that fascinating?’ On the other hand, you get invited to come to New Zealand and chat to you guys.” The British press can be vicious. “I think we’re all cautious what we say about the show because there are certain media outlets that like to ruin a show because they despise it or because they want to spoil it for an audience,” he says gloomily. “The British aren’t comfortable celebrating success, put it that way.” That’s what Americans are for. When we meet he’s just flown in from the Emmys with his wife, Lulu (10-year-old son Felix, stayed home). “It was a hoot. No one celebrates their own industry like the American film and TV industry.”

Transatlantic Downton mania earned the show 16 nominations, up against Mad Men, Breaking Bad and Homeland. The show won three, one for the brilliant Maggie Smith. Though she has competition in series three, screening here now. It’s ostrich feathers at dawn when Shirley MacLaine, as Cora Crawley’s mother, Martha, goes through Downton like a dose of Epsom salts. This is casting out of left field, or possibly outer space. “Oh, it was sensational. To have a proper Broadway and Hollywood legend coming to join an already legendary Maggie Smith …” says Bonneville. “We persuaded her to do a bit of Sweet Charity and she told wonderful stories, everything from reincarnation to the price of cheese.”

She’s still into all the woo-woo stuff, then. “Oh, absolutely. She has no side about it. She’ll happily talk about a glass of water and in the next breath about a former life she might have had. And the ghosts of Downton, which she could feel.” The ghosts of Downton. They must haunt Bonneville, too, these days. “The impact of the show is happening elsewhere. One is just in the middle of getting on with the work and getting on with one’s life at home,” he says, with the practiced ease of a survivor of the unstoppable, global, slightly unaccountable Downton Abbey juggernaut. In the end, as he says, it’s just telly.