There is no shortage of scientists, or the research they produce. But how does a lay person – a non-scientist – judge what is good research and what is bad research. What should they believe and what should they be sceptical about?
It is now accepted that a very significant proportion of all published research, in all fields of science, is flawed to various extents. This situation is often called the ‘reproducibility crisis’. And if it is flawed it might not be repeatable. Society should not base its policies on flawed information. Why has this situation arisen, and what can be done to improve it, so that people can believe the science they read about?
The situation has arisen mainly as a consequence of the pressures scientists are under: pressure to publish their findings and pressure to bring research funding (money) to their institutions. Surely if a scientist achieves in those two endeavours then he or she is a good scientist? Possibly, but possibly not! In fact these pressures can easily lead scientists to act in ways that I consider unethical. For example, because the number of research papers a scientist publishes is often considered a measure of his or her standing as a scientist – the more the better philosophy – scientists have learnt to play a variety of games to maximise their output of papers. Two of the major tricks are premature publication of preliminary results, before it has been established that they are repeatable, and exaggeration of the significance of results. The latter trick begins with the title: make it as sexy as possible. The scientific journals that publish the research papers are likely to favour publication of sensational results, and agencies funding research might favour funding novel, exciting research, rather than confirmatory research. Negative results, such as ‘chemical A does not affect species B’ are unlikely to get published, leading to bias in the scientific literature.
These, and other, problems with scientific research as it is presently published are now widely recognised to be pervasive. But correcting them will be very difficult, and take a long time. Probably the most effective way to improve the current undesirable situation is to very significantly improve the training of scientists. That needs to begin with the young scientists who are currently learning their trade, so that as time passes not only will they do better research than today’s established scientists, but they will pass on their higher standards to the young scientists that they will train in the future. Unlike many professional occupations, such as accountancy, medicine, and law, there is currently no official training programme with an agreed curriculum for scientists. A scientist does not need to pass a series of examinations, that assess his or her competency, in order to call themselves a scientist, start doing scientific research, then publish the results of that research. I consider that they should have to demonstrate that they are competent before they can practice as a research scientist. They should have to take exams covering such topics as experimental design, hypothesis testing, dealing with bias, data analysis and interpretation, and presentation of results. Those examinations should be mandatory for all research scientists, and the results should be placed into the public domain. Only scientists who pass all the examinations should be eligible for positions as research scientists. Many of the topics young (and older!) scientists need to be better informed about are discussed in the recently-published book ‘How to be a better scientist’, by Johnson and Sumpter.
Until the current culture within scientific research changes, and does so very substantially, the reproducibility of published research will remain unclear to all except the experts. Hence I will be difficult, if not impossible, for the vast majority of people to know what research is reliable and what is unreliable. Unless we improve the training of scientists, the reputation of science as a profession will continue to decline.
Reported by:
Tim Pilgrim,
Media Relations
+44 (0)1895 268965
tim.pilgrim@brunel.ac.uk