On 10 January 1999, the first episode of a drama about a New Jersey gangster with panic attacks debuted on the US cable channel HBO. The Sopranos ran for six seasons, the final episode being broadcast on 10 June 2007. Across eight years and 86 episodes it came to represent a golden age of TV, when moral complexity, deep characterisation and unprecedented authenticity were common features of a bold new form of televisual storytelling.

It’s this “peak era” that Peter Biskind, the cultural critic and film historian, both celebrates and to some extent mourns in his new book, Pandora’s Box: The Greed, Lust, and Lies That Broke Television. Biskind is best known for his eye-opening account of US cinema’s own peak era of the late 1960s and 70s: Easy Riders, Raging Bulls. That book became a word-of-mouth hit, partly for its wealth of lurid anecdotes about Hollywood figures, featuring epic misbehaviour with all the sex-and-drugs trappings of starry success.

The screenwriter Robert Towne once told me that he thought Biskind should be “horsewhipped” for the stories he told in that book, not least about Towne himself. When I mention this to Biskind – on a Zoom call from New York – he laughs.

“That’s what you get when you interview somebody’s ex-wife – she had pretty much nothing nice to say about him. But I try to be fair and paint the good and the bad. Very few people are all bad. I’ve written about Harvey Weinstein, and even he had his good points,” he says.

Biskind, who lives in upstate New York, is a sprightly 83 and has spent much of his life documenting the business and mores of the film industry. A former editor of Premiere magazine, he is a longtime contributing editor of Vanity Fair. Down the years he has developed a wealth of Hollywood insider contacts. Yet he admits that, after Easy Riders, Raging Bulls came out in 1998, interviewees became much more wary about speaking to him. Aside from its scurrilous tales, that book also detailed the triumph of young auteurs such as Francis Ford Coppola and Martin Scorsese, who rode a wave of countercultural disruption to create a new kind of film-making that supplanted the dated formats of the out-of-touch studio system.

Something similar happened in US television at the turn of the millennium, but the catalyst was technology rather than social upheaval: first the move from the free-to-air big network companies – ABC, NBC, CBS, Fox – to subscription cable, such as HBO and Showtime, and then to the streaming companies such as Netflix and Amazon Prime. These shifts led to glittering opportunities in TV for ambitious writers and directors.

It’s this milieu that Biskind explores and, along with it, the corporate power struggles that first enabled a period of radical creativity before, he contends, bringing it to an end. It’s a story of dizzying success, followed by hubristic overreach and cautious retreat. Although not as lurid as Easy Riders, Raging Bulls, the book is full of scenes of difficult people behaving badly, and of the exuberance and excitement that propelled US television at the turn of the century.

“The networks had pretty much reached a dead end,” says Biskind. “The business model had been sponsor-based, in which advertising paid for the programming. And advertisers didn’t want their Buicks and their drug advertisement adjacent to sex and violence.”

Known for their largely bland output, the networks had enforced what he calls a “rigid 1950s morality” that “dropped their wet blanket over programming”. As the last century came to a close, a generation of frustrated writers began to find creative liberation on cable subscription services such as HBO that didn’t have to appease advertisers. The Sopranos wasn’t the first HBO drama to break with convention – It’s Garry Shandling’s Show (a satire on American talk-show culture) and Oz (an iconoclastic prison tale) predated it. But it was the first to gain serious critical attention, both in the US and abroad, and a large audience.

The creator and showrunner of The Sopranos was David Chase, a veteran TV writer who had built up decades of bitterness and resentment about the humiliations he had experienced at the hands of network TV, which duly rejected his idea about a gangster in therapy.

In the book, Biskind depicts Chase as a difficult and controlling figure. A depressive, he is seen as jealously guarding his hard-earned reputation as a creative genius from everyone else on set, particularly his writing team, a number of whom were dIspatched without ceremony. It’s an unflattering portrait, but it fits with a visit I paid to The Sopranos set at the height of the show’s success, when I interviewed a dispirited Chase who had scarcely a good word to say for anybody.

One scriptwriter is quoted as saying: “As David became more secure, his temper tantrums got worse. He demonised people. The objects of his hatred would change – the wardrobe person, the casting director, his secretary. And then he would go back on his Prozac.”

Biskind sounds a more sympathetic note now when I bring up his characterisation of Chase. “He had suffered so long in the network system,” he says, “that, when he got his own show, he was determined to run it the way he wanted and to fend off anyone who he thought challenged his vision. And as The Sopranos became more and more popular, he felt more and more pressure, so he didn’t lighten up. If anything, he got more difficult to deal with.”

There were many clashes with the show’s lead, the late James Gandolfini, a hitherto minor character actor who quickly grew into the starring role, as the role in return served to expand him – not just in range and earning power but also in girth. Biskind writes that Gandolfini would sometimes go missing for days on drink- or drug-fuelled breaks from the pressure of being Tony Soprano.

The conclusion of the show coincided almost exactly with the departure of HBO’s chairman and CEO Chris Albrecht. He had been arrested at 3am in Las Vegas for assaulting his fiancee after they had watched a boxing match. Although she refused to press charges, Albrecht was ousted when it emerged that he allegedly assaulted another woman 16 years earlier.

That kind of behaviour was much more prevalent in the period dealt with in Easy Riders, Raging Bulls, when alcohol and drug use was rife and women were expected to play a passive, and not infrequently mistreated, role. A decade on from Albrecht’s exit, after Weinstein’s exposure and the #MeToo movement, several executives lost their jobs for using inappropriate language to women lower down the hierarchies. As much as this appeared to be a step forward for women in TV, Biskind suggests that it also proved a convenient means of removing high-ups in internecine power struggles.

But in Albrecht’s case it seemed to be a matter of enforcing principle over corporate self-interest. He had overseen not just The Sopranos, but The Wire, Six Feet Under, Deadwood, Band of Brothers and Sex and the City. In the UK, we saw these shows on different channels, although they all came with the quality-guarantee imprint of HBO. In the US, they were all broadcast by HBO. It had become the go-to home of the critics’ favourite television.

Albrecht’s ousting, says Biskind, was a turning point. “They replaced him with four people and none of them had any real programming experience, and HBO went into a long decline,” he says.

At the same time, a month after The Sopranos finished, the first episode of Mad Men, created by former Sopranos writer Matthew Weiner, aired on rival cable channel AMC after HBO had passed on buying it. That decision was made during the Albrecht era, but the timing looked symbolic of a channel that had lost its way.

In its Sopranos pomp, everybody wanted to be on HBO, and thus the channel could commission pilots from the cream of writers and film-makers. Those pilots didn’t necessarily make it to the screen or become a series. That is the long-established American pattern of TV production. The networks had always hugely overcommissioned and, flush with success, HBO began to follow suit.

It passed on Breaking Bad, and writers began to take their work elsewhere, with Homeland going to Showtime. To make matters worse, several of the major productions that HBO managed to action were embarrassing disasters. Who remembers Vinyl, a music-industry drama boasting Mick Jagger and Martin Scorsese in the producer credits, that was cancelled after one season? If the conjoining of those two cultural giants was a mistake, it pales beside the misconceived Luck, a horse-racing drama that galloped straight to the knacker’s yard. Starring Dustin Hoffman, Michael Gambon and Nick Nolte, it was co-produced by David Milch, the mercurial creator of NYPD Blue and Deadwood, and Michael Mann, the celebrated and fiercely self-assured director of Heat and The Insider.

No one could seem to agree on what the story was about, as agreement was not something that either Milch or Mann had spent much of their careers seeking. The two producers barely communicated with each other, according to Biskind, and neither appeared overly-concerned about the spiralling costs. “Michael Mann restraining Milch was like the fox guarding the henhouse,” said Allen Coulter, one of their directors.

In the end the show was cancelled owing to animal welfare concerns (three horses had died during filming – one reviewer wondered whether it was from boredom), although Biskind implies that it was a pretext for closing down an expensive mess.

Much of Biskind’s book is devoted to the big executive players behind the scenes, the huge corporate buy-outs and the CEO merry-go-round. For most British readers, the names may not mean very much, and their dog-eat-dog, master-of-the-universe attitudes can grow tiresome. Yet what is notable, not to say shocking, are the amounts of money involved. David Zaslav, for example, the head of the Discovery Channel (or Warner Bros Discovery as it was renamed), took home $246.6m (£199m) in 2021, mainly in the form of stock options. The following year he had to make do with a measly $39.3m.

No wonder the writers and actors who went on strike earlier this year weren’t sidetracked by the claims of the TV companies that they lacked funds to meet their demands. Did Biskind ever ask one of these executives if they thought they were worth the phenomenal sums they earned?

“Well I never asked Zaslav, for example, if he was worth the money he was getting because I didn’t want him to hang up on me. Your perspective gets totally distorted when you’re in that sort of stratosphere of money and power. Do they wake up in the middle of the night and think, am I really worth all this money? I don’t think that happens.”

The great TV the US produced, particularly in the first decade of this century, may have stemmed from a change in financing – from advertising to subscription – yet it should never be forgotten that the prime function of TV companies is not to entertain but to make money.

Netflix, for instance, was set up when its founders were casting around for a new home delivery market after Amazon had taken care of books. The answer they came up with was DVDs, sent out to renters and buyers. As the business evolved into a streaming platform, Wall Street investment came rushing in. Flush with that support, the platform’s first original offering was House of Cards. The showrunner was David Fincher, director of Zodiac, The Social Network and Gone Girl, and the star was Kevin Spacey, and all 13 episodes of season one were released on the same day – 1 Feb 2013.

With that introduction, it looked as if big-budget quality entertainment was going to be Netflix’s stock-in-trade.

“Because these dramas are so expensive,” says Biskind, “the way that they’re greenlit is they green light a pilot, and if the pilot goes well, they go to a series. But Ted Sarandos [managing director of Netflix] gave Fincher a $100m two-season deal, which was totally unheard of. They used their famous algorithms and the algorithms said, go ahead. And it put Netflix on the map.”

Once on the map, a diet of masterworks didn’t quite materialise. Instead, Netflix filled up its library with masses of old network TV, a number of more commercial commissions, such as Orange Is the New Black, and such a vast bank of shows that it’s nigh on impossible to distinguish the gems from the dross. Huge sums of money have been routed towards original marquee films such as Scorsese’s The Irishman, which was said to have cost $160m to make, but in recent years the platform’s once ever-increasing number of subscribers has stalled or even fallen, although a recent crackdown on password sharing to stem the losses has had some success.

For all its promise to redefine television, Netflix has always been saddled with enormous debt – at last count $14.5bn. For a while, at least, it was a streaming monopoly, which promised great wealth. Now the market is crowded with competitors, including Amazon Prime and Apple TV+ with their almost bottomless pockets.

“When Netflix was in its adolescent growth spurt and adding subscribers hand over fist,” says Biskind, “Reed Hastings, who was essentially the founder, said that Netflix’s only competition was sleep. That’s no longer true.”

Sometimes the boardroom machinations and clash of billionaire egos that Biskind records is reminiscent of what was famously termed a “shit show at the fuck factory” in HBO’s finest show since its glory years: Succession. Was that series a sign that the golden era continues?

“Succession is already done and gone,” Biskind responds. “It was a great show but where TV is going now doesn’t look so great. David Nevins, a studio and [former] Showtime executive, said to me: ‘They’re all racing for the mainstream.’ They’re all going in the direction of the old networks with broad-based programming that appeals to everyone. And that’s a recipe for blandness.”

Both Netflix and Disney have recently launched cheaper advertising-supported plans, and Biskind sees the return of sponsors as a likely step towards discouraging controversy or moral complexity in drama output. Perhaps this is the nature of golden eras, they burn brightly for a short time and then succumb to the prevailing winds of economic forces. In the 1970s, the young Hollywood auteurs became increasingly self-indulgent, with films such as the massive loss-making Heaven’s Gate leading studios to reclaim control of their product.

At the same time, some of the celebrated young directors turned their talents to blockbusters such as Jaws. The combined effect was a death knell to the personal film-making that had flourished for a decade or more.

In the same way after the success of The Sopranos, cable TV threw great sums of money at talent, and got Vinyl and Luck. But sometimes a seemingly limitless budget is the enemy of creativity. Biskind believes that between them, the cable and streaming companies will continue to produce acclaimed work every now and then, such as The White Lotus, but increasingly a kind of consolidation will take place that will look a lot like how TV used to look in the second half of the last century.

“Eventually we’re going to be in a situation where instead of [US] TV being dominated by three or four networks, you’re going to have TV dominated by three or four big streamers. A lot of network shows are being streamed. Even from the beginning Netflix was showing Seinfeld and Friends. That may just increase. [Mad Men showrunner and Sopranos writer] Matt Weiner was quoted as saying it’s like Mad Men never happened. It’s not a pretty picture.”

Perhaps. But before we lament the passing of a great age of television, we ought to appreciate that it changed ideas of what was possible, for both creatives and commissioners, across the globe, and helped turn TV, internationally, into the most exhilarating medium for drama. It’s hard to imagine that the small-screen revolution in which Denmark produced The Killing and France the brilliant The Bureauwould have happened without HBO’s and other US cable stations’ redrawing of television’s boundaries.

As Succession showrunner Jesse Armstrong has pointed out, high-quality TV existed before this golden era, and doubtless there will be more of it afterwards. But it may be a while before the mega-corporations hand over quite so much artistic freedom to the talent again.

• Pandora’s Box: The Greed, Lust, and Lies That Broke Television by Peter Biskind is published by Allen Lane (£25). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply