"Bad Pharma: How drug companies mislead doctors and harm patients." by Ben Goldacre - Courtesy of Observer online

Ben Goldacre: 'The world would be a better place if doctors were less enthusiastic about adopting very new drugs.' Photograph: Linda Nylind for the Guardian

The blindingly obvious inference of the extract from your book published in the Guardian– as of so many others you once commendably wrote in your Bad Science column – is that this is an industry totally unsuited to being run on profit-maximising lines by conventional shareholder companies. Given that, and the tremendous level of subsidy the industry already receives from governments around the world, why not spell out the vital necessity of locating it within publicly owned/non-profit organisations where there need be no obstacle to full transparency?

Top of Form

Harry Shutt, via email

I am a realist about this. I don't want a central-command state economy. In general, drug companies are reasonably good at developing new treatments and there's also a lot of good in the industry. The point of my book is that it's possible for good people in badly designed systems to perpetrate acts of great evil completely unthinkingly. I don't think any of the people I write about would punch an old lady in the face, but they would inflict the same level of harm when they are abstracted away from the outcomes of their actions.

This is made easier, I think, because in general, most drugs do work better than nothing: it's just that we may be misled into using, for example, an expensive new drug where an older, cheaper one is more effective.

Overall, the problem is we don't have a competent regulatory framework that prevents things from going horribly wrong. If companies are allowed to hide the results of clinical trials then they will, and that will distort clinical practice. Doctors and patients will be misled and make sub-optimal decisions about what treatment is best for them.

Similarly, if you can get on to the market by making a me-too copycat drug that represents little or no therapeutic advance and is even less effective than the drugs that it copies, then you will. And you can get such a drug to the market because regulators approve new treatments even when they've only been shown only to be better than placebo…

The Observer: Rather than the best current treatment available?

Exactly. When that happens, then there's less drive to innovate new medicines. So a lot of people think they know how you test a new drug, they think it's a placebo-controlled randomised trial, and that's kind of true. However, for most situations we already have some kind of treatment, so we're not interested in whether your treatment is better than nothing; we're interested in whether it's better than what we're already using. But you can get on to the market without ever comparing your drug against the best currently available treatment, and then you can bamboozle doctors and patients into using it by distorting the clinical evidence with all the tricks I describe, and then by giving a biased picture of that distorted evidence to doctors and patients through your marketing activity. All of that means, crucially, you can turn a profit by producing drugs that aren't very innovative.

So at the moment our regulatory frameworks do not represent a good set of incentives for innovation. When people say, "Oh my God, it's dreadful all these companies have had to pull out of developing new drugs for Alzheimer's", on the one hand Alzheimer's is a very difficult problem, on the other hand we haven't created a regulatory framework that sufficiently incentivises people to take serious risks, and research entirely new treatments.

I have been in healthcare marketing communications for more than 30 years (flogging drugs to doctors) and can confirm that much of the sharp practice you describe is caused by the pressure exerted on researchers by marketing departments. When a new drug entity makes it through Phase I, the pharmaceutical company's marketers carry out market research studies to find out what doctors expect and desire from such a product. The results of these studies are used to develop "guidance" for researchers designing Phase II and III trials. One issue is the speed with which some doctors ("innovators" in marketing speak) adopt new medicines. There is a drug surveillance system in this country that requires doctors to report all adverse events occurring in patients taking new drugs. So if doctors were a little less enthusiastic and prescribed new products to no more than a couple of patients at first, most problems would come to light before large numbers were affected.

arghbee, via web

I agree, the world would be a better place if doctors were less enthusiastic about adopting very new drugs. When I was taught how to prescribe by my clinical pharmacology professors, as an undergraduate in medicine, we were told: don't prescribe new drugs unless they are spectacularly superior to what you already have; let other doctors take the chance. And that's what I teach students myself. Regard the whole of the rest of humanity as unpaid stunt doubles. Because, when you prescribe a new drug, often you are prescribing something that has only been tested in a few thousand people for a very short period of time, perhaps only six months, and that's not long enough to know whether there are any medium- or long-term side-effects. It's also too few recipients of the drug to find out about rarer side-effects. On top of that, when drugs come to market they've only often been shown to be effective on short-term outcomes, and on surrogate outcomes, weak outcomes, theoretical outcomes. So a new drug might have shown to affect a blood test result, which we hope is theoretically associated with a real-world outcome, but not tested against real-world outcomes such as pain, suffering and death. On top of that, while doctors and patients sometimes fetishise the new, the evidence is that overall, new drugs are often no better than the older ones they replace.

If there is time for shared decision making, when we're prescribing a new drug, then I think first doctors could maybe say to their patients: "The drug I'm prescribing has only been tested in a few thousand people for less than six months, and it's only been shown to improve blood tests, not the real outcomes in your disease. If you prefer, we also have a tablet that has been around for 10 years, has been taken by millions of people, and that we know has a positive impact on real-world long-term outcomes – which would you prefer?" Because I think these are reasonable issues to factor in, when you're making a treatment decision.

Obs: And the point about marketing departments influencing trials?

This is a really interesting area and it's a good illustration of how there is good and bad in current marketing practices. I think it's really good that somebody out there listens to what doctors and patients want and need, in order to try to address those needs. There's a scandalously brief and interesting history of research on this question in academia. It was only very recently that people went out and formally asked patients: "What's the most important outcome to you?" The classic study was one in a rheumatology outpatients' clinic – they said: "What kind of trials do you think need to be done?" And the patients said: "We don't think you need any more trials comparing one pill against another, we want trials to tell us if physiotherapy improves outcomes, because doing physio is a massive drag and requires a lot of effort from us." The other thing they said, to everybody's complete astonishment, was: "We don't think that pain is the most important symptom, it's actually stiffness." And that amazed everyone. So good quality, systematic, mixed methods, qualitative and quantitative research, to find out this sort of stuff is really great and important.

Obs: Are pharma marketing departments doing that?

Well, they're probably doing some and a mix of other stuff. The real problems come when you look at the interplay between market research and the whole process of publication planning, which is something that many doctors and academics are completely unaware of. Certainly, I think, the public have very little knowledge of it. We all like to imagine that the academic literature is composed of worthy papers by independent academics exploring things on the basis of interest. We don't, for the most part, realise that there is often a hidden hand guiding this process. So when a new drug is being brought to market, especially one that addresses a problem that hasn't been addressed before, you will often get an elaborate sequence of covertly planned marketing activity in the academic literature, without any declaration that this is what's happening.

For example, companies will set about paying for and planning, and in some cases ghostwriting, papers saying that the disease their new drug treats is an underdiagnosed problem, it's much more prevalent than we thought. You might also start to produce lots of literature about how current interventions aren't very good, downplay your own side effects, promote off-label uses of your drug, and so on. That's all before you get anywhere stage-managing the literature about the actual benefits of your treatments.

So the interplay between marketing departments and research departments, I think, is inevitable. It's a mixture of good and bad, but the thing that's most striking is how ignorant most people are about these things, and how huffy and upset drug companies get when you start to talk about it. If people really think it's OK to have commercial medical writers writing papers instead of academics, then they should put their names clearly and proudly at the top of the papers. If people really think it's OK to stage-manage the whole programme of academic journal articles behind the scenes up to the launch of your drug, they should just mention that in the papers at the bottom: "This is part of the programme leading up to the launching of drug X. The idea for this paper came from the marketing department at Y." If they think all this activity is OK, then that's fine: they should be happy to declare that publicly, and we can decide for ourselves.

In your book you appear to be describing situations where companies knowingly suppress information that shows the products they are selling are ineffective or worse. My doctorate is in engineering and if we had done that we would have been liable to a prosecution for fraud. This is quite irrespective of the regulations pertaining to a specific industry, though we were required to report any significant deviation to the regulator as soon as we were aware of it. So, quite simply, why are these pharmaceutical companies not being prosecuted?

Joseph Cullen, via email

Well, it's a huge cultural blind spot. No one, with the exception of the Faculty of Pharmaceutical Medicine, which is a very small organisation, not one of the medical and academic bodies have stood up and said: "Selectively withholding unflattering trial data is research misconduct, and the ultimate end product is a biased picture of the effectiveness of your intervention."

If I deleted half the data points in one study to make my treatment look better than it really is, everybody would say that was research misconduct. But for some reason when I delete half of the trials from my clinical trials programme and only publish half of them, that is not regarded as research misconduct.

It's a cultural blind spot that comes about, I think, because the misconduct arises from a slightly diffused network of failure. So in some cases there will be obsessive, evil people at the centre, with full panoramic knowledge of everything that's happening, stage managing it. In other cases, you'll have individuals in organisations, maybe quite junior, they've got a trial with a negative result, nobody's enthusiastic about putting it out there, and to that one researcher, not publishing a paper intuitively doesn't feel the same as deleting some data points in a study, even if the end product is the same, in terms of the impact on doctors' and patients' knowledge of the risks and benefits.

That's why I think it's really disappointing that nobody, not the Royal Colleges, the Academy of Medical Sciences, the British Pharmacology Society, the British Medical Association, none of these organisations have stood up and said: selective non-publication of unflattering trial data is research misconduct, and if you do it you will be booted out. And I think they really urgently should.

I have to say, for a lot of the things I cover in the book, I honestly believe this stuff has been protected from public scrutiny for many years by a wall of tedium, or if you like, by a wall of modest scientific complexity. It's behind closed doors but not locked doors, it's all hiding in plain sight. I think it's very likely that this will turn out to be like MPs' expenses or phone hacking journalists. The people involved in these small communities have all convinced one another that what they do is completely normal and fine. But it's not. I think that when the public come to see what has been going on, they might be appalled.

I work for an institutional shareholder of many large pharmaceutical companies. Do you think there are any pharmaceutical companies that stand out in terms of transparency of trial results? And at the other end of the scale, are there any companies that consistently fail to publish negative trials?

Laura Foll, analyst, Henderson Global Investors

No. I have no reason to believe that any one is any better than the other. If you have bad regulations, incoherently enforced, then everybody does what they have to do, to succeed in the marketplace.

Considering we have to discuss and explain population-based risk to the individual we are treating, and the way we discuss risk biases the response, how do you discuss risk in simple terms, for example, in primary prevention?

Tariq Hussain, GP

This is an amazingly interesting area, and I think it is the next horizon in medicine. If you put me in charge of the medical research budget I would cancel all primary research, I would cancel all new trials, for just one year, and I would spend the money exclusively on making sure that we make the best possible use of the clinical evidence that we already have. This means first devising better ways of communicating risk to patients, and engaging in a process of shared decision-making wherever appropriate. Second, I would put the money into a better information architecture for evidence-based medicine. I would put the money into researching and implementing better systems to get the right information to the right doctor at the right time, to collate and then to disseminate the information that we have.

Obs: We had quite a few questions from readers who, faced with difficult decisions about changing drugs or trying a new drug, were reading trials and studies themselves…

That's a real mistake, actually. As I say in the book, I don't think it's a good idea for individuals to try to pick and choose their drugs, or stop their drugs. I think it's a very dangerous business. I don't say that because I want to protect any special status in the medical profession, I just think it takes a really long time to acquire the skills and knowledge to do that. People often say: "What do I do for me? For my treatment decision right here and now?" And the best advice I can ever give is to find a good doctor and talk to them about it, and anybody who tells you that they can give good advice about your medical problems in a newspaper article, or on the internet, or in some silly magazine, should be regarded with infinite suspicion, no matter how big a bouffant or how deep a perma-tan they have – I'm not thinking of anyone in particular, I don't really know that scene. I don't do readers' health advice and I think these things are best discussed with your doctor.

Patients often resort to saying: "Doctor, if it were you, what would you do?" Or "If it were your child…?"

I think that's a reasonable question to ask. Often decision making is a complicated business, weighing lots of risks and benefits against one another, often it can't be operationalised. But I think the important thing is that doctors have all of the information that they need in order to make informed decisions. First, because it's not hidden from them, and second because it is competently disseminated to them, and patients should have the choice to engage with that, but there's nothing compulsory about it. I think data on how good your local school is should be available to you, but equally I recognise that the overwhelming majority of parents never look at it, and that's normal. I wouldn't judge them. I just think the information needs to be available.