How to read a paper. Navigating the maze
1:45
Welcome to the webinar on how to read a paper and navigating the maze. Tonight’s webinar is going to be presented by Prof Paul Glasziou. Before we start, I would just like to let you know my name is Gail Roberts, I will be hosting this evening’s webinar, I would also like to acknowledge the people who are the traditional custodians of the land on which we are all joining this webinar tonight, I wish to pay my respects to their elders, past and present. You will see at the bottom of the slides there is your Control Panel. We don’t have chat tonight, but we do have questions and answers, which we will be looking at very frequently, so, please feel free to put your questions and answers in there, you will have the opportunity to ask questions throughout the webinar and at the end for about 20 minutes. I would like to introduce our speaker for tonight, we are very lucky to have Prof Paul Glasziou. Paul has been a very long member of our college and we are very grateful for that. Paul is a professor of evidence-based medicine at Bond University, was a part-time GP until fairly recently and for many years and is the professor and director of the Centre for Evidence-Based Medicine at Oxford University from 2003 until 2010. Paul’s key interests include identifying and removing barriers to using high-quality research, which we will learn about tonight, in everyday clinical practice. His research has influenced numerous guidelines and clinical policies and practice, cardiovascular disease, management, screening, clinical monitoring and antibiotics stewardship and guidelines for reporting research. We are very pleased to have Paul talk with us this evening about this key aspect in the research process, how to actually read about research and understand if it is good or not. Paul, welcome and thank you.
3:41
Thanks very much for that introduction Gail and welcome to everyone who has been able to make it to the session at night. I always know it is tempting at 6 o'clock in the evening to find better things to do, so I am glad so many people could actually join us here. So, just a little bit more background about me. My main interest has been in evidence-based medicine but I was, as Gail said, a part-time general practitioner for over 20 years, and in that time I really spent a lot of time grappling with how I could better use research evidence to assist with patient decision-making, sometimes with individual patients or sometimes with category problems that our practice had. So, in that time I have learned a lot of little tips and tricks, some of which I will show you tonight about just how to make it easier and quicker to read a paper. So, I am just going to try screensharing and hopefully in a moment you will be able to see some slides. Can I just check … can someone tell me that that’s actually up. (Pause). I will just assume it is then. So, what I am going to talk about is how to read a paper about the anatomy of a research paper, the IMRaD structure and where you find things and also about three acronyms that you can use, the PICO, RAMBO and FAITH, which, if you understand those will actually guide you through the reading of almost all research papers, and these are things that try and make it easier for you. So, this is one of my GP colleague’s daughters reading the evidence-based medicine journal, and if she can read it at the age of, I think she is about 3 there, I am sure you guys can do even better. This is really a sort of overview, introduction and orientation to the issues, which will help guide you to learning how to do it, but the workbook that I think you have been circulated, the Evidence-Based Practice workbook, is a more detailed guidebook and also has got a lot of practice exercises. So, what I am going to orient, which is a bit like telling you about the black and white keys of the piano and what the pedals are for and explaining what a scale is. But to actually learn how to play the piano or to read a paper you have actually got to do the practice and that would be going through the workbook. So, you will get some tips tonight that would help, but really to learn how to do it is quite a skill. So, I just want to test, we are going to poll you a few times during the night and I just wanted to test that that is actually working. So, here is the first question, which parts of a research paper do you focus on when you read a paper? The abstract only, the introduction and conclusions, the methods and results or you never read research papers. I will give you about 20 seconds for that. (Pause).
7:06
Okay, can you see the results?
7:08
No, I can’t, so if you can….I am just going to stop sharing to see if I can see the results. (Pause). No, I still can’t.
7:15
Okay, so, 21% said abstract only, 54% introduction and conclusions, 21% method and results, and 4% never read research papers.
7:31
Okay. That’s very helpful. So, one of the things I am going to teach you is that actually reading the introduction and conclusions can be quite dangerous, but most people avoid the methods and the results part of it because they seem harder to read, so I am going to talk about how you can read them a little more easily. I’m just going to bring up my screen again. (Pause). Okay. So, the steps in reading a paper is first of all to find and understand what the research question is. Hopefully some of you at least have heard of the PICO before, but I will just walk you through some different variants of PICO, the population, intervention, comparison and outcome, and I usually mark that up on the paper as I am reading it because it is such a useful initial orientation. And once you have understood the research question then you can work out what the study design they could have used versus the study design that they did use. So, is this really the best type of research to answer this type of question, and different types of questions require different types of research. If it turns out that it is actually a good design you still need to look at the internal validity of the study, and there’s some very long questionnaires with you know 10, 20, 30 questions in them and I am just going to talk to you about a very simple method, which is to use the so-called RAMBO, which just looks at the three most crucial components to look for, and once you are skilled in that you can do it in probably less than 30 seconds. And then for systematic reviews there’s an alternative that you would look for, and that’s the FAITH. And then I will teach you a little bit about reading the results and that’s looking at the clinical implications of that. So, let me start with a simple example, this comes from the Australian Journal of General Practice, it is about placebos in general practice, the national survey of physician use and their attitudes and beliefs. So, the structure that they have here is the usual IMRaD structure, which most papers now have. This did not exist a hundred years ago, it’s been an innovation of the last century, to break it up into the introduction, methods, results and discussion. But it is not always called that of course, in this paper it is called the background and objectives is the introduction and then the methods and then the results and then the discussion. And there are some journals that put the methods at the end, for example, it is not always in this order, but usually you can find that the components are there, but they are just maybe named differently or they are in a different order. So, a couple of tips about this, reading a paper you shouldn’t read through the whole paper, what you want to learn to do is to hunt for the various key things that you want. One tip is that the question is usually at the end of the introduction or in the last paragraph of the introduction. Sometimes it is the first sentence of the methods but most often it is described in the end of the introduction. So, usually the introduction is structured as a sort of background, like why we are interested in this problem in the first place, but the question is at the end, that’s sort of the hourglass structure of gradually narrowing down the introduction to get to the point of the question. And then step 2 is to check and appraise the methods and that’s by hunting for some key items and then if they are okay, reading the results, and again you don’t need to read all the text, I would generally just go to the figures and tables first of all and then use the text to help me read those figures and tables. So, for those that are not familiar with PICO, this is the general anatomy of a researchable question so you could ask a question like, does vitamin X, which is a non-existent thing, lower mortality. And so, what you are going to do is you got to get a populational or set of group of patients and then you give some of them the intervention, the vitamin X, there has to be a comparison and that here might be a placebo for example, and then you have got the outcomes such as death. So, the question here actually only specifies two of those, vitamin X and the mortality, the intervention and outcome without saying what the population and the comparative would be. So, sometimes you will get that in the title of the paper, but then you need to work out the other components and you really need to find all of them to understand what the question is. And I put time in brackets because sometimes the question will intrinsically involve time like, is this short-term mortality or long-term mortality, for example. So, let’s go back to our paper, The Placebos in Australian General Practice, and indeed the question is at the end of the introduction. The aim of this study was to examine rates of use and beliefs about placebos in Australian general practice. I have to think slightly because the question is sort of implicit, that the population here is general practitioners. The outcome is the use and beliefs about placebos and in this one there is actually no intervention and no comparators. It is what I would call a PO question, because it is not about trying to change something or looking at what the risks are of something, it is just about a simple frequency question. So, I am going to give you some practice PICOs just to play with. So, these are some titles from a single issue of the BMJ and I want you to see if you can work out from the title, what the population, intervention, comparator and outcome is, starting with this first one, and realising that some of these it is actually incomplete. So, I might just give you about 60 seconds to think about that yourself to see if you can work those out and work through as many as you can. (Pause). Okay, well, let’s have a look at some of them. I have colour-coded the answers in this next slide here, so, blue is for the population, red is for the intervention, green is the comparator and purple is the outcome. So, in that first one they specify the population, mild to moderate, severe community-acquired pneumonia, the intervention is just continuing antibiotics at day three, the comparator is versus day eight, actually it might have been the other way around, maybe the three days was the … no actually the comparator is usually the standard thing, so eight days versus the standard and this was doing it earlier, so we would consider that usually to be the intervention. And they don’t actually specify the outcome here, they say effectiveness, but not the effect on what, so, if you wanted to know the effect on what, you then have to go through and find that somewhere else in the paper, not in the title. And it is actually pretty rare to get everything, all of the PICO in the title. So, it helps to try and work it out, you start with the title and go, well what am I missing here and let’s work it up. The colour of bile, vomiting and intestinal obstruction in newborns, so, here the population is newborns, the outcome is intestinal obstruction. This is a diagnostic question and that’s what they are trying to predict, and they are looking at the colour of the bile. So, actually the colour is the different possible colours and so both the intervention or exposure in the comparator group are both hidden within that thing of the colour of the bile vomiting. And the last one, the effect of off-pump coronary artery bypass surgery, that‘s clearly the intervention. The outcome… I am sorry I should have put this in purple… clinical angiographic, neurocognitive and quality of life outcomes. The population is not specified, but it is pretty clearly going to be people who are having coronary artery bypass surgery, that is people with coronary artery obstructions. And they don’t specify what the comparator is, that may be a self-comparison or it may be with a group who is not having it or is having off versus on pump. So, that’s a couple of example questions and as I said I usually, to help orient myself, when I read a paper I underline or mark up with a highlighter what are those four elements, the PDI, the C and the O. But as I said there are some questions that are just P&O questions where we are interested in the frequency of something. So, for example, a question might be, what outcomes are patients with rheumatoid arthritis concerned with. So, the population would be patients with rheumatoid arthritis and the outcome is their degree of concern for different outcomes. Now, sometimes it is qualitative research where you don’t even know what the outcomes are that people might be concerned about. You are trying to ascertain and that is why I have got the question marks for. What the outcomes of interests might be to patients. And I put up this specific example because it is a really interesting one in terms of the history of assessing rheumatoid arthritis because a group called ______ was putting together ways of assessing trials for rheumatoid arthritis, for drugs in particular but for other therapies, and eventually they asked a group of patients to come in and ask them about the questionnaire that they were using, The patients said whilst it is great that you are trying to standardise, but you have left out one of our main concerns … one of their main outcomes. So, I have illustrated here, pain and deformity, which people would commonly pick up, but they had raised the issue of fatigue as one of the major concerns of patients with rheumatoid arthritis. The _____ group then subsequently did the survey to assess the frequency of that and indeed fatigue wasn’t number one, but it was actually very high up in terms of the concerns of patients. Okay. So, in the book there is the thing that takes you through the basic study designs and I am going to talk about how those designs relate to the critical appraisal. So, on the left is a table of some of the designs, the randomised controlled trial, a cohort study, a case-control study and a cross-sectional study, that I hope you have at least heard of before. So, how would you appraise those, so with a randomised trial you are looking for random allocation. With a cohort study, there will be non-random allocation, but it is the same structure. So, the right-hand side shows you what Rod Jackson in New Zealand says, there’s only one study type because all studies go like this no matter what they are named. You can think of the structure as being the dotted triangle is the group of potential patients, the ones in the solid triangle are those that actually got in. And then they are either allocated or you measure the intervention or exposure group and the control group, that is we sort of split that group into two halves. And then we follow them up over time and then we measure some outcome and that creates our 2 x 2 table, and so, all studies do that, and if it is a cross-sectional study the timeframe is much shorter that is, it is almost immediately afterwards, and a survey would be similar. Whereas cohorts and randomised controlled trials take a bit longer to do. Case-control study is slightly different because it starts at the other end, we identify the outcomes and then work backwards, we find cancer patients and noncancer patients and work back to see if they have been exposed to smoking, for example, which was the first type of study that was done to detect the association between smoking and lung cancer and then they later did a cohort study as well. But it is actually conceptually it is the same structure, it is just that the data has been collected in the reverse order, so, commonly also called a retrospective study. And then for a cross-sectional study there’s just no middle bit, all you are doing is measuring some outcome like attitudes towards placebos as we saw. So, this one as we said is just a PO question. And what type of question is it, is it treatment, prognosis or frequency, it is clearly a question about frequency where wanting to know the frequency with which people used placebos in general practice and what their attitudes are. So, I have highlighted some of the bits we would look for, but I am going to just talk about what the best evidence would be first of all and then I will come back to critically appraise that study briefly. So, the best evidence depends upon the type of question, if it is a treatment then you want a randomised trial, but if it is a prognosis what happens to patients with rheumatoid arthritis over time, or are they at greater risk of cardiac disease. For that you want an inception cohort. And if you just want to know about the frequency of smoking in the population or GP’s use of placebos then that’s done by just a survey. Now, I have left out some of the question types here, there’s a few others such as diagnosis, but it is the same sort of principle and that’s spelt out in the workbook, that the best evidence depends upon the type of question. That’s what we would call the level 2 evidence, the best would be a systematic review that look for all the randomised trials or all the inception cohorts or all the surveys of that particular question. Okay. Your turn again. So, just going to test your ability with study designs and if you are not familiar with study designs there is a section in the workbook which takes you through this and works you through some examples. So, I will give you the simplest one first, the best study type to assess the effect of a new treatment would be… and you have got a choice of A to G there, which one would you use, and I will give you 20 seconds to do the poll. (Pause).
23:26
Paul, can you see the results?
23:27
No, if you can read them to me.
23:29
Yeah. Cohort study was 0%, case control study 0%, and randomised controlled trial was a 100%.
23:43
Oh that’s wonderful. Okay. Let’s give you a tougher one. Okay. The best type of study to determine the prevalence of cataracts. Another 20 seconds. (Pause).
24:21
Okay. The results. Cohort study 14%, case control study 3%, randomised control trial 0%, population survey 76%, consecutive sample of patients 7% and nested case control study 0%.
24:45
Okay. Okay, that’s very good. So, actually the number one study would be some sort of population survey where you would need to ask people and then actually check vision for example, or actually just get people in to check their vision. You could incidentally do that through a cohort study, so, there was a cohort study in the Blue Mountains, which was a representative sample and they got the prevalence of cataracts at various points in time, so, you may be able to extract the information from a cohort study, but if you only wanted it for current prevalence once, that’s really just a population survey. So, the other answers are not completely wrong that people said, but the usual thing would be to say, if you just want to know the prevalence or frequency of something, what you want is a sample from the population and then do the measurement on them. Okay. I am not going to do the other ones because we have got limited time tonight, but this was really just to give you the idea that actually all of the study types have some purpose and some relevance to them, but it is understanding which study types, which basic designs you want to use for different types of questions. So, that’s one of the steps in going through the process is to identify the study design. Okay. Even if they have got the right study design they may have not necessarily conducted the study well enough to avoid major bias, so we also need to do a rapid critical appraisal and here is where the RAMBO and FAITH acronyms come in. You might think that just because it has been peer-reviewed that it must be okay. That is not true. Even top medical journals can make mistakes, like the Lancet did about at the beginning of the pandemic by putting in the Surgisphere observational study about hydroxychloroquine, which was actually fraudulent but was also a rubbish study as well even if it had been real data. So, good journals can make mistakes, but the lower tier journals often have very poor quality studies in them, so you do need to look at what the quality of the study is, but you don’t have to do that in huge detail, there’s a couple of key things that you want to go through. So, this diagram again, the participants intervention comparison outcome in Rod Jackson's GATE framework of the population, the allocation and then the measurements. And what we are interested in for the validity is those three letters. So, if we are talking about treatment, is it randomised, but if we are talking about cohorts or frequency questions the focus is really of the R is the representative. Is this a complete or a random sample of the population. Then you want to know how much attrition there was, how many people for example in the survey did people lose or in the trial did they not manage to follow up. That is the A. And then the measurements, were they blinded or objective measurements. And these three things cover the major biases. In a sense there are only three types of biases, selection bias, attrition bias and measurement bias. So, the questionnaires cover a lot more things than that, but these are the three key domains that if you understand them will cover most of the issues that you need to look at in critical appraisal and it covers all types of studies, but just that for the trials you are going to be using randomised as your R and for surveys, diagnostic or prognostic studies you are going to be using the representativeness. Okay. So, for treatment trials, we are looking at random allocation, attrition, we are usually asking for greater than 80% follow-up and for measurements we want them blinded or objective, it doesn’t have to be both. So, if you have got an objective measure like mortality, then it doesn’t need to be blinded. You can still do the blinding but you want one or the other of them as a minimum. And for observational studies it is pretty much the same thing, the representativeness, it is usually using a random selection. The attrition again, you basically want an 80% response rate, like the 80% follow-up, and the measures again should be blinded or objective. Okay. So, let’s try applying that to our non-treatment question, which was the placebos in Australian general practice. So, is it representative? Well, they looked at a random sample, so, that’s great, from a national database. So, they get a tick for that. What about the attrition, well 18% took part, 82% did not. So, here is the first limitation of this paper is, that it has got a very poor response rate, actually 18% is pretty typical by the way in national surveys with general practitioners. Now, the measurements, were they blinded or objective? That’s a bit hard to say, let’s just see what their measurements were. So, one of the major questions was the frequency of use of either inert placebos or active placebos. Did you never use, used at least once a week, once a month, once a year, less than once a year. So, do you think that question is blinded or objective? (Pause). I will let people put that in the Q&A. (Pause). No one is willing to have a go at that? Okay. It is relatively objective, but you are still relying on people's recall. So, it is certainly not blinded because they know the purpose of the question, so, no one said it is blinded, that’s great to see. The question is whether it is an objective question. And I would suggest that you could have done something that is more objective than that, but it might have been difficult like you could have them recording as they did consultations, what the actual placebo that they prescribed was. Here we are relying on their recall of it, and so it is not completely objective. So, I think that’s the minor limitation, the big limitation would be the attrition bias, that is that only a select number of people got there. So, what should we do with this, would we just throw away the paper completely? Well, I want to ask you about what the likely direction of bias here is and we will get this again as a poll for you to do this. But just to see what the results were first of all, so, 61% of people said they never used a placebo, 39% said they had ever used a placebo and once a week was 4%, once a month was 10%, once a year was 8% and less than once a year was 16%. So, that’s an interesting set of frequencies. What I want to ask you is what’s the likely direction of bias … let’s focus on the 39% who have ever used a placebo. Do you think it’s likely to be an underestimate, an overestimate or neither? That will be fairly close. Again 20 seconds for the poll on this. (Pause).
33:21
Okay. We have got 56% underestimate, 15% overestimate and 30% neither.
33:34
Okay. So, that’s probably pretty reasonable folks. It is hard to predict the direction of bias here. I could argue in multiple different ways about which way the bias might go. The question you might ask is who is more likely to answer a questionnaire such as this, a person who has never used a placebo or a person who has used a placebo. That’s the one you have to think through. I could argue either case, a person who has used placebos may go, well I am really interested in a questionnaire about placebos because I use them. Or you could say I have never even contemplated that, that’s a really interesting question, I wonder what other people think about this. And so, I think you could do either. So, unfortunately, we probably can’t predict what the direction of bias is, but I wanted to raise the issue because sometimes the direction of bias is predictable and you could then say, yes, well, this is the estimate but it is likely to be an underestimate, or, yes this is the estimate but it is likely to be an overestimate, but here it is actually hard to predict, we just know for a sample of 18%, that about 39% say they have used it at least once. Okay. So, we are reading the numbers there and by the way next to it has been the confidence interval, so, that 39% they calculated the confidence interval as going from 31% to 47%, a measure of uncertainty. But in reading those numbers I wanted to put up what I call the fundamental equation of error, and that is that that measure, like the frequency of the usage of placebos is a combination of three things, the truth, how often people really used placebos, bias in the way that we have done the study and random error. Okay. If I as a researcher and trying to reduce that, I want to use a good study design to reduce bias and I want to use a large number of people in order to reduce the random error. But you are reading a paper, so, from a reader's point of view what you are doing is first of all critically appraising the design, is it the right design and here’s my RAMBO, was it representative, what was the attrition, what were the measurements. Because that is looking at the bias component and then there is the random error component which is the confidence intervals and P values, but you have to interpret those confidence intervals and P values with regards to the bias. If there is a high degree of bias then the random error isn’t enough, there is another factor in there that’s distorting the truth and that is the bias involved. So, just because something has got a very low P value it doesn’t mean it is true. It may be all due to bias, so, you can’t read the confidence intervals and P values of a paper, the statistics without a consideration of doing the RAMBO first, that is looking at the bias, which is why we teach you in that order by the way of thinking about the bias. Okay. I am not going to walk all the way through a treatment study because I want to give people time to ask some questions as well, but let’s just have a look at one study that was published I think this was last week in the New England Medical Journal. It was a large randomised trial with 87% follow-up and they used a placebo, so it was blinded, so it gets ticks for all three. And what were they looking at? They were looking at total fractures. So, does vitamin D vs placebo in the general midlife to older adults, as a supplement, lower the rate of fractures, and the answer to this was no. You can see that in the graph here, there’s a figure here of the total fractures. You can see that the cumulative fractures for the vitamin D group and the placebo group pretty much overlap with one another, but they have also measured it through this thing of the hazard ratio. Now, what does that actually mean? Well, if they had been identical rates or identical hazards, the hazard ratio would be 1. If vitamin D was protective it would be less than 1 and if vitamin D was dangerous it would be greater than 1. The confidence interval here, because we have said okay we can read it so there’s minimal bias, so we are able to read all of these numbers, then that 95% confidence interval here goes from below 1, that’s 0.889 to above 1, 1.08, so it overlaps with the no effect, which would be 1 for a ratio, and that corresponds to this P value of 0.70, that’s a non-significant P value and this confidence interval also tells you that because it is basically straddling one that confidence interval. Okay. I am going to pause now, I will leave that up for a moment. Are there any questions that people have about interpreting any of those numbers or this trial? If you can just post those in the Q&A. I am going to move onto just a couple of other slides, but then I will come back and I just want to make sure we have captured any questions about this that people wanted to ask. (Pause). Okay. Let’s move on. Just two other things to talk about, one is, all we have been talking about is in the workbook that’s been sent around, but in great detail, and there are various exercises to work through those. So, if you are not familiar with the basic studies, designs, there is a section here on basic study design. We will go through the PICO again and there’s some examples that you can work through trying to work out what the PICOs are. There is critical appraisal for systematic reviews, which I haven’t done yet, and for trials and the critical appraisal and the two mnemonics is the FAITH and RAMBO that we have just been talking about, but the workbook will cover those in more detail. I just wanted to mention… I am going to go straight to it… I haven’t gone through the FAITH, but if you are talking about doing a systematic review, which would be the ideal thing, what you are looking for is the FAITH, which is, did they find all the studies, did they appraise them that would be using a RAMBO process, and did they include only the good ones and for treatment that might be the randomised ones with a blinded outcome for example. Did they total them up that is combining the studies in the so-called forest plot meta-analysis and did they look at the heterogeneity, were these things that you could validly pool together or was this statistical heterogeneity. So, I am going to stop the presentation there and just check to see if there are any questions on that or on anything else that you would like to ask. Okay. So, someone has asked me how do you determine bias. There’s the bias that comes from measurements, there’s bias that comes from confounding from not using randomisation. And we can quantify some of those biases, so, people have done studies that have looked at the same question being answered in a blinded and an unblinded study, and if it is a subjective outcome of symptoms, for example, the unblinded studies always do better than the blinded studies. That is the bias, the measurement, because you haven’t used an objective measurement, or a blinded measurement by using the placebo. And roughly speaking, that bias is about a 50% improvement that is attributable to bias rather than a real effect. The randomisation is harder to predict because the confounding can go in either direction. All you know is you can’t trust the results of unrandomised studies about treatment. So, I hope that has answered the anonymous question, but there is more about the biases that are in that workbook. Okay. So, for time poor GPs, is there a website, blog or podcast or other social media site we can go to to summarise the multiple studies that come out each day of the week? That is a very good question. There are processes that do that, like the ACP Journal Club for example, which scans many of the major journals to try and find out the good quality stuff, but I am not aware of anything for GPs that currently exists at the moment, but a group of us who have been running a research study about this is looking to try and raise that possibility. Does anyone else have an answer to that for things that they have found, that would be helpful. Otherwise, I will try and send you some of the ones that I am aware of. Is the workbook being emailed to us? My understanding was that it should have been emailed to you before this, but we will certainly make sure we can re-email it to anyone that didn’t get it. I will leave it to Gail, if you want to answer that question.
44:25
Thank you. The workbook has been sent out to most registrants; however, it will also be posted on the RACGP research webpages and I will email everyone here tonight to let you know where you can find that on our research pages.
44:50
Someone has asked me, is Cochrane too pessimistic? That’s a good question. I think it is reasonable for most treatment questions to look at the Cochrane results, but that’s their results, their conclusions are often more research is needed, it is pretty rare that I will say that the question has been completely answered, but it is useful to read through the results section of a Cochrane review to see what the results actually show, what the quality of the current evidence is and what the size of the effects are. Someone's mentioned the monthly podcast that does _____. Yeah, the ____ group in the US who have been running now for about 20 years I think actually do pretty good quality stuff, they start with the … it’s the patient-oriented evidence that matters, so their first filter is whether the outcomes that have been measured in the study are actually things that would be important to patients and that is great. And secondarily they look at the quality of the study. So, I think that would be a good one for people to actually have a look at. It doesn’t cover, I think, a huge range of studies, but it is good for finding out some current things that are relevant to family practice, which is what general practice is called in the US.
46:29
While we are waiting for someone else to ask a question, Paul, if you are a GP who is interested in research and learning more about being more analytical about your practice, what sort of things can you do in your day-to-day practice, like, for example a journal club. What are some other things GPs could do to hone their skills?
46:44
Okay. I am just going to put up my slides again, for your Dorothy Dixer question (laughter). Here are the things that you might want to do. So, we have just had a question about, get an evidence-based update service, I mentioned the ACP Journal Club, but that’s really for internal medicine only, but is a fantastic resource, or we have heard about the ____ as well, which is a pretty good resource as well. The other thing I suggest is keeping a logbook of your own questions, that is, things that have come up in practice, either in discussion or that a patient has raised, I used to keep a little… that’s what I have got here, a little spiral-bound notebook on my desk, because you often forget the questions, people forget about two-thirds of the questions that they actually have that have arisen in practice. Now, you are not going to be able to find the evidence for all of those, but you might ask colleagues and when you do ask them, do you know what the evidence for that is when they give you an answer. But some of them you will try and answer yourself, you might look up guidelines, you might try and learn from the workbook about how you can search for the evidence yourself. And the other thing you can do is, it’s really good fun to run a question discussion group. Journal club is a bit of a technical name for it, actually a lot of it is just about discussing some clinical issues and the picture here is from my practice in Oxford, we used to do it every Friday morning before the clinic started and we would have a cup of tea, we were actually doing a blind tasting of margarine, different types of margarine versus butter by the way, because it was one of the questions that came up in our thing … and we usually had one topic of the week that somebody is leading and a paper on that, but most of the discussion is after we briefly look through the paper, is about the clinical issues that arise out of this. And it doesn’t have to be a research paper, it can also be going through a guideline and checking the evidence and information that is in a guideline as well. And it is a great place to bring up the things that you are worried about in practice, so, people used to come along on their day off because they enjoyed it so much, I highly recommend trying to do it yourself. Frequency might be about between once a week and once a month. If it's less than once a month, you start to run out of steam. They are some of my suggestions, but if people have others please put them in the Q&A.
49:32
We have got another question Paul. Are there any GP-led trials in Australia?
49:40
Oh, yes, there are …
49:42
When do you start (laughter)?
49:44
(Laughter). Yes, most of the university departments would be running some sort of trial or something in general practice. So, for example, we ran one recently on shared decision-making for antibiotic usage, we have also run one on top tips for weight loss, Nick Svar who is a GP, but head of my faculty of medicine, has an MRFF grant to do one on COPD and the community for example, a trial looking at that. A group in Melbourne University are running ones on depression, a whole series of ones on depression, so yeah, there are lots of GP-led trials in Australia, some of which the college funds, most of which have been funded through NHMRC, National Heart Foundation or the MRFF Medical Research Future Fund.
50:42
And it is worth mentioning also that the RACGP now has a new noticeboard for people who are conducting GP research so that they can recruit through that, so, we will put that address up for you in case you are interested in participating in someone else's research or if you are running your own research and want to get some people as participants.
51:04
Okay.
51:05
Thank you.
51:06
Hormone replacement therapy. Oh, that’s a difficult one to get into. The massive trial that was published in ____ about… it must be nearly two decades ago now, they actually stopped it early because of a lack of benefit and apparent harms. I think it has been the only definitive study ever of hormone replacement therapy. People have said since that they got the wrong doses or the wrong hormones they used, etc, and have done smaller studies, but nothing of the scale of the women's health initiative trial, which would actually definitively answer the question. So, I think at the moment it is probably the most solid piece of evidence we have, but clearly there are questions that have been raised since then about what the appropriate age groups are, but that’s not one that I am completely up-to-date with, so, I think you might have to do your own… find a good systematic review. And Gail has typed something into the …
52:10
I have just put in the address for the noticeboard, Tanya has kindly given that to me at short notice. You have got some more questions. I have got one for you Paul. Everyone loves systematic reviews and they are held up as the gold standard, but they are not all equal, are they? What should people look for if they get a systematic review to read?
52:38
Okay, that’s FAITH. So, the problem is not all systematic reviews do a good job of the process. In fact, there are many more poor reviews than there are good-quality reviews, there have been a couple of meta reviews that have looked at the quality of systematic reviews, and John, I and Edi have just done one of those studies which says that only about 1 in 10 of them actually lives up to the high enough methodological standards. Cochrane’s reviews 95% of them I think are trustworthy, occasionally there’s a Cochrane review that slips through, but because they have a fixed protocol they get a review of that protocol, they have methodological guidance on the searching, they have statistical guidance, etc, the Cochrane reviews are generally pretty trustworthy, but non-Cochrane reviews can do a good job too, particularly the ones from health technology assessment agencies, but there are a lot of pretty rubbishy things that are even called systematic reviews that you just can’t trust. And so, that’s where the FAITH comes in. Did they do a good job at finding everything, did they do an appraisal to look at the quality of studies and surprisingly some don’t do that. Did they include only the high-quality ones and did they total them up in a valid way, and again, in your workbook there is guidance that takes you through how to do that with an example. One thing I should warn you is, these PICO RAMBO things will take you quite a while the first time. It might take… when you do the workbook it might take you an hour to get through that. Second time, half an hour. Third time, 15 minutes. Fourth time 5 minutes. And then eventually you can get it down to being able to identify the question of a paper and appraise it in less than a minute, but you can’t do that the first time. Just like you can’t play Beethoven at your first attempt at playing the piano, I wouldn’t start there.
54:57
We are still waiting on some more questions. People, this is a great opportunity, you can be anonymous. While we are waiting, if people want to learn about P values, what should they do, is there something that they can read or some exercise that will help them understand P values and confidence intervals.
55:19
Again, there are things in the workbook, there’s a very nice video which I will just try and find for you to post in the chat link, by Steve Woloshin from Dartmouth, that’s about 5 minutes of his walkthrough of forest plot that explains confidence intervals and P values extremely readily. He is a great teacher. (Pause). Okay, let me post the link in here.. oh, actually I can’t do it I realise.
56:06
If you send it to me Paul, I will send it out to everyone.
56:08
Okay.
56:09
Yep.
56:15
Okay. Other questions.
56:19
Oh, good, we have got a couple. Hazard ratio is the same as odds ratio, that’s a question, and stats are boring. How can I generate interest in the same (laughter).
56:31
(Laughter). Yes, I agree. Stats seems pretty boring, but you actually have to understand them for the good of your patients, because you need to know … is this going to reduce your odds of getting COVID by 10%, 50%, 95%, that makes a difference to your decision-making. The question about hazard ratios and odds ratios, for all intents and purposes you can ignore the difference between a hazard ratio and odds ratio and a risk ratio or relative risk. They are all about the same and one of the magic things is anything that is a ratio, the magic number is 1, alright? And anything less than 1 is beneficial and anything higher than 1 is harmful. If you have got a treatment group divided by the control group and they are equal, the number is going to be 1, whether it is a hazard ratio, an odds ratio or a risk ratio. The only other mathematical operation you can do to compare two numbers is the difference. So, you also have a risk difference for example, or a hazard difference.
57:51
Some more in there in the question and answer.
57:54
Is there a research project on making great diagnosis and avoiding bad mistakes. Oh, that’s a whole separate webinar in itself about how you can use the right diagnostic processes and about safety netting as well, but that’s a long process, there is a useful book called Evidence-Based Physical Examination, which teaches you what bits of physical examination work, but the other thing I would suggest looking at is learning to use the appropriate clinical prediction rules for general practice, like, the Ottawa ankle rule for example when you have got a sprained ankle, or the _____ sore throat rules when you are worried about whether someone's got beta-haemolytic strep. Can you please advise about how to get … GP can get into research or an academic general practice. There are academic registrar positions that you can apply for at most universities, which is once a year, and Gail, you will probably able to say how you actually get hold of the information about those.
59:05
Actually Tanya can probably talk to you about that, people can email Tanya.
59:10
Okay. That’s if you are really serious about this, that gives you some time in an _____ to learn about research, but most academic groups also have a practice-based research network. And so, the first thing I do is find out whatever your local university is, like Monash, Melbourne, Sydney, et cetera, find out about from the department what their practice-based research network is, most of them have regular meetings as well that you can go along too, and that’s a way of getting connected into research processes.
59:46
And we have a list of PBRNs, the practice based research networks from all over Australia and also all the Australian universities that have a GP department or discipline on our website and our research pages if you type in RACGP research or email me if you don’t have any luck, but you should…
60:05
I think Tanya typed that in. Okay. Suggestions of books that a GP can access that have basic statistics and epidemiology and any university courses you can recommend. The first thing I would do is read the workbook, and once you have understood the workbook, come back to me and we can make some recommendations about further reading that goes beyond what’s in the workbook, but the workbook works through the absolute essentials in statistics and epidemiology that you must know as a reader of research. If you want to become a doer of research there’s more than you need to know in the design stuff, in fact to be a good doer you should first be a good user of research and know what a good research study looks like, so, doing some critical appraisal practice is really a first step on the wrung towards becoming a doer of research.
61:02
Thank you Paul. We are just about at the end of our time. I would like to thank you very much for your time Paul. You have worked really hard getting this together so people can look at it afterwards as well as live, which was very helpful. The audience know that we will send out all the information we have talked about to you by email and we will also post it on our webpages and we will let you know where those webpages can be accessed. We have got another webinar coming up on the 14th September, which will be run by Professor Steve Margolis, but Paul, just back to you, what you have said tonight is hopefully inspiring people to think about research and it sounds like it from some of the questions, so thank you so much, we are very grateful and I will speak with you about a possible other webinar next year that might be an adjunct to this one.
61:58
Okay, thanks very much Gail.
62:01
Yeah, thank you so much.
62:01
Great to see that 44 people stayed on right to the end.
(Laughter)
61:08
So, I am going to finish quite abruptly I think, so, good night everyone and thank you for your interest and we will be in touch. Thank you. Good night Paul. Thank you everyone.
62:16
Good night.
61:17
Thanks. Bye
END