Lessons in Orthopaedic Leadership: An AOA Podcast
Lessons in Orthopaedic Leadership: An AOA Podcast
The Future of Orthopaedic Surgery Research Unveiled
Curious about how the future of orthopaedic surgery scholarly publications is shaping up? This episode features a deep dive with Marc F. Swiontkowski, MD, FAOA who provides an authoritative look at the challenges and opportunities facing the field. From the unintended consequences of open access mandates leading to a surge in subpar journals and paper mills, to the crucial rise in retracted manuscripts due to poor peer review, Dr. Swiontkowski explores every avenue. Also covered is the role of hybrid open access models and the complexities surrounding preprint servers, especially in the wake of the COVID-19 pandemic. This is an unmissable conversation that underscores the need for rigorous peer review and global cooperation to maintain the integrity of orthopaedic research.
In another compelling segment, the pitfalls and limitations of using large clinical databases and registries in medical research is assessed. Learn how missing data and skewed populations can skew your conclusions and why it’s crucial to approach these resources with a critical eye. The misuse of statistical methods by some researchers to chase positive associations without a clear research question, and how this can dilute scientific rigor is also discussed. Finally, how academic journals may inflate their impact factors and the transformative—and sometimes troubling—role of AI in academic publishing is explored. This episode is packed with invaluable insights for anyone vested in the future of orthopaedic surgery research and publications.
Welcome to the AOA Future in Orthopedic Surgery podcast series. This AOA podcast series will focus on the future of orthopedic surgery and the impact on leaders in our profession. These podcasts will focus on the vast spectrum of change that will occur as the future reveals itself. We will consider changes as they occur in the domains of culture, employment, technology, scope of practice, compensation and other areas. My name is Doug Lundy, host for the podcast series. Joining us today is Dr Mark Swinkowski.
Speaker 2:Dr Swinkowski received his medical degree from the University of Southern California School of Medicine. He completed his internship and residency training at the University of Washington and then went on to Davos, switzerland, where he completed a research fellowship at the Laboratory for Experimental Surgery. He began his work as associate professor at the Vanderbilt University, where he helped establish the state of Tennessee's first level one trauma center. Dr Swierkowski then moved to the University of Washington as professor of orthopedic surgery and chief of orthopedic surgery at Harborview Medical Center in Seattle from 1988 to 1997. From September of 1997 through October of 2007, he held the position of professor and chairman of the Department of Orthopedic Surgery at the University of Minnesota. Dr Swinkowski has also been president of the Orthopedic Trauma Association, president of the Mid-American Orthopedic Association and president of the American Orthopedic Association, in addition to serving on the board of directors of the American Board of Orthopedic Surgery. So, dr Swankowski, welcome to the podcast series, sir.
Speaker 3:Thank you, Dr Lundy.
Speaker 2:Happy to be here. And then also, I did not mention that you also served in the role as editor-in-chief of the Journal of Bone and Joint Surgery until very recently. Isn't that true that?
Speaker 3:is true, a 10-year term from 2014 to January 1, 2024, where I handed the reins over to my esteemed colleague, Dr Mohit Bhandari.
Speaker 2:Very good, so I can't think of anybody that would be better suited to talk about the future of our academic journals and the work that goes into these journals and how this will impact our profession. So thank you for talking to us about this.
Speaker 3:My pleasure.
Speaker 2:Now we had briefly discussed ahead of time. A great way to probably start this out is to talk about the issues with open access, and I know we talked to him. I heard you speak at the American Orthopedic Association meeting very recently and St Louis about this. What are your thoughts on open access? How has that affected our profession and where do you see that going?
Speaker 3:That's a huge topic, one we could spend an entire hour or two on it as a mandate, initially in Europe and then across the pond in North America and Canada, from federal governments who were trying to make their funded research accessible to all researchers worldwide. In other words, if individuals didn't have access, through a paywall, to a specific journal, they couldn't access the research. So it became a funding mandate that all of their publications should be available freely and immediately after publication, to acknowledge the contributions of their funded research, but also to speed up the exchange of information and make it more fluid. Unfortunately, the whole concept became hijacked, if you will, as so many forward thinking and good ideas do, by nefarious motivations, and it has resulted in a huge number of journals that are really not providing adequate peer review or oversight and paper mills, and resulted in large batches of published manuscripts being retracted. There's a group that's formed that's called Retraction Watch, and they provide charts which shows the near 45-degree angle escalation of the number of manuscripts being retracted on an annual basis because of falsified information that hasn't undergone peer review. It's all about cash for publication, and it has contaminated the literature with faulty information, and so the whole open access initiative has been sidelined and is under extreme scrutiny now by knowledgeable individuals worldwide who really are seeing, unfortunately, the downside being higher than the upside.
Speaker 3:Now all scholarly publications of reputation now have opportunities where you can have any article that's published in so-called hybrid model where an article can be immediately available. So CORE and BJJ and JBJS we all have mechanisms where an individual, or particularly an individual with sponsored research, can pay an extra couple thousand dollars to have it immediately available. Hybrid open access be the model for open access, rather than having specific open access journals that only have article processing charges or APCs or an individual pays to have the manuscript be open access. It would be a way to eliminate these shadow journals and paper mills etc. But that's going to take near global governmental support and it's a slow process to educate people about the problems with open access.
Speaker 3:I know I'm taking a long time to answer this first issue, but the one thing that's possibly worse than open access is the phenomenon where manuscripts are put up prior to peer review and they're citable and what that has resulted in is dual copies of manuscripts being available to individuals worldwide.
Speaker 3:Often these are called preprint servers. Often the original posting on a preprint server, even though proven in subsequent peer review to be fallacious or in error are never taken down. Oftentimes the most, I guess, outrageous and headline-grabbing research findings are picked up by the lay press and then brought out to the public without appropriate peer review, and it has resulted in all kinds of problems, the most notable of which is in the area of the early treatment paradigms for COVID, where false research claims small cohort series are picked up and published as truth and they're never really taken down after exposed to rigorous peer review and found out to be false. So it's a combination of the open access plus preprint servers which are really degrading the overall scholarly publication world and are of grave importance to orthopedic surgery and the future of our specialty.
Speaker 2:I was a little surprised when you brought this up at AOA, in that I always saw those solicitations from these open access journals to be almost like internet scams. Almost it was like, hey, we want you to publish in here and, by the way, if you send this check, we'll publish anything. It almost seemed like get away from me. This isn't legit. But what you're saying is this is a real threat. That's actually. A lot of people are submitting their work to yeah. Absolutely.
Speaker 3:And you're an educated consumer of scholarly work. And there are many young individuals who are feeling under pressure to have huge increases in the number of citations in their CVs and they will jump at these particularly when they can get around a really rigorous peer review to gain a publication. And that leads us to perhaps another topic and that's the misappropriation of drive towards having CVs that have numbers count, the numbers kind of approaches, which is unfortunately motivating a lot of our younger colleagues who feel that's what matters, when it really shouldn't be what matters. What should matter is the impact of your work and participating in high quality research designs that can actually favorably impact care in a way that undergoes the most rigorous peer review. And that's a hard thing to get through to our younger colleagues that it's not about numbers. It should be about doing quality work with fewer numbers that are actually going to impact care work, the fewer numbers that are actually going to impact care.
Speaker 2:But are the committees that work on academic advancement? Are they contributing to this motivation of the younger physicians, or is this all intrinsic to them?
Speaker 3:I have been interested in this issue for a long period of time and I serve on our medical school's promotion and tenure committee and I will point out that it is a problem in most medical schools around the United States in having an adequate number of surgeons to serve on these committees. The fact of the matter is that an academic orthopedic surgeon's life is a very different life than an academic pediatrician's life. Many of the pediatricians I hate to pick on those my daughter is a pediatrician, she's a very good one happens to be a Peds GI person but their academic life dominates over their clinical productivity life and, as all of our listeners know, that's not the case in our specialty. All of us in our field are quite busy clinicians and it is more difficult for us to produce adequate time to focus on producing scholarly work. So that's one thing.
Speaker 3:The other thing is that a very positive thing is I think our field of orthopedic surgery has advanced the other surgical fields in terms of the quality of clinical research we publish in terms of prospective trials, randomized trials, multi-center trials, collaborative work, work research registries, etc. And it is hard for other groups of people practicing in other fields to understand the difficulty and the amount of time that is involved with participating in a multi-center trial, resulting in a publication that may have 24 authors to it. So, all of this, I hope that my plea reaches some years of more senior members of our community that will actually seek out opportunities to participate in the peer review process of these dossiers that are coming forward and really serving to protect and advance our surgical colleagues as they go through the academic process, because it does really result in a numbers game for particularly basic science researchers and other fields, when that's a totally different world. So you're right, some of the promotion and tenure pressures are contributing to this notion and we, as a community of horsepeed leaders, need to help dispel those misconceptions.
Speaker 2:That's very helpful. Now, when we were at the AOA meeting and you were presenting this information, you also brought up the idea of these large administrative databases that seem to be generating significant and I think you used the word that these are shortcuts to getting a lot of manuscripts submitted. Can you talk about that a little bit?
Speaker 3:I'd be delighted to, because in my decade as editor-in-chief we saw from very few publications into massive numbers of publications using these administrative databases, to the point where we made it the target of a retreat. We did and we published a supplement which I think I have back here on my desk.
Speaker 3:It is Sorry, doug, that's not what I have written in my tongue, but we produced in 2022, a supplement from a day and a half symposium that we did at the Academy's headquarters in Chicago on the use of these administrative databases, listing in a very detailed manner the limitations of every single one limitations of every single one as well as we listed comprehensively all of the registries around the world and their individual limitations to try to inform the research community about what those databases can do and what they should not attempt to do. There are deficiencies across the board in every single one because of the lack of specificity on important clinical characteristics of the patients that are included in those databases and registries, which limit the value of any conclusion that they come to. There is simply no way to control for the factors that influence clinical decision making and unfortunately again, I hate to bring up our younger members of our community, but many members- see this as a
Speaker 3:quote quick and dirty unquote unquote way of getting a publication because you can sit in your barcode lounger and churn through data to produce some finding. That is no way should it or could it impact actual care For anybody that's listening. Please do not embark on such work unless you understand the limitations of the database you are trying to use and what it can and cannot do. And please understand that if you try to submit these to higher impact journals, you're going to be successful in the case of JBGS, in single-digit percentages because of these deficiencies, and I believe that's true for the other more higher-impact journals in our field.
Speaker 2:So this includes stuff like the ChemZeta, the Medicare claims databases and things like that.
Speaker 3:Correct, correct, they're. Often. What happens is a younger investigator who has access to these files will start a phishing expedition looking for statistical associations and then publish it as a so-called finding. Asked a focused question in advance and started using statistical methodology to literally fish for positive associations.
Speaker 2:And the registries like HARR and the other more advanced registries could do the same thing.
Speaker 3:Absolutely. There are profound limitations, not only with lack of clinical characteristics affecting decision-making, but also missing data, and many of these databases do not have information across all socioeconomic groups, they do not have information across all modes of employment, et cetera, et cetera. They're skewed populations and it is risky to make any sort of conclusion. I would say that on the upside, they are reasonable ways to generate hypotheses to be tested with higher level research design. So it's not all bad. Nothing is, or very few things are really all bad, but unfortunately they're so attractive because it's so easy to sit in the comfort of your living room next to a refrigerator and do work instead of doing the hard approach of actually formulating a research question, collecting pilot data and testing a hypothesis, which takes much more work but has much more profound impact on the progress of our field.
Speaker 2:Now, as I've been more associated with our journals you and I are both in trauma so we both read the same things become more accustomed to the whole concept of impact factor. How do you feel impact factor affects our journals now and then? Where do you see this moving in the future?
Speaker 3:Well, the impact factor is generated by a company called Clarivate and it's a simple formula of you take a published manuscript and you assess, you measure how many times it's been cited over a two-year period of time and then you do the multiplication to generate an impact factor. You have to remember that orthopedic surgery, even though we're an influential group, we're a relatively small percentage of the overall physician community less than 3%. So there literally is no way that any orthopedic journal is going to generate an impact factor that's going to come close to a general medical journal like the New England Journal or Lancet or DJJ, one of those we're. Just we're too small a group. So that's factor number one.
Speaker 3:Factor number two is, unfortunately, many of our colleagues in our field have learned how to game the system, and you can game the system by publishing editorials that cite the research that's published in your journal, and that is one thing that we never did at JBJS and I hope we would never do that. So you can artificially inflate an impact factor by gamesmanship. You can also inflate the impact factor by limiting the number of publications you have. So if you're really interested in producing a high number we take the example of JBGS, which I know the best would never publish an ethics manuscript, because ethics manuscripts simply do not get cited, similarly with educational related manuscripts very rarely cited. So you can game the system in multiple ways to produce a bigger number, but that really does not favorably impact the progress of our field by any way, shape or form.
Speaker 2:And so it is a fact of life.
Speaker 3:It is a calculation used to generate an age factor, which those of our listeners who are in academia will understand that the dean's office in any medical school knows your H factor, which is basically a citation indice, and it is a metric that is commonly understood and used. But I'm hopeful that perhaps by this podcast we might educate our community a little bit more that it should never be the end-all be-all, because it's gameable, fungible if you will, and it really doesn't tell the whole story about how an individual site, a public manuscript or a journal impacts the progress of a field. So I don't think it's going to go away anytime soon. There's a lot of pressure on Clarivate to innovate better ways to measure the impact, but we need to all understand a little bit about how it's gained and what it is and what it isn't.
Speaker 2:And many times we hear from the pundits about the issues with artificial intelligence and how they can be used to mislead the public in what certain candidates may or may not say or what they may or may not have done. We also know this is extensively pervading the academic spaces as well, and I know that y'all wrote a editorial you and Seth Leopold and Mr Haddad, I believe, and was there somebody else too? I can't remember.
Speaker 3:Yeah, linda Sandel, from the Journal of Orthopedic Research. Yeah, that's right, I'm sorry.
Speaker 2:Wrote an article about, or an editorial about, ai. What are y'all's thoughts about where AI is? Where's it going? How's it impacting our journals?
Speaker 3:It was a rather straightforward and limited editorial. We basically agreed on two basic principles One is that an AI engine cannot be an author, and two was that when individual authors use these tools large language model tools being the most prominent they need to state in the manuscript where and how it was used.
Speaker 3:We did an experiment which you may have heard me talk about, where I worked with some colleagues at Harvard and asked them to generate a couple of RCTs, which they did. Each one took two minutes, using ChatGBT and unbeknownst to our editorial board. We sent them through the process and one of them got through the peer review process, which was quite alarming to all of us. We used the opportunity at last year's editorial board meeting to go through this and we have learned some important issues about the weaknesses of these tools to date, being that they do perseverate around, particularly the bibliography. There can be journal names which don't exist, the citations are presented in nonsensical ways. So we've not only educated our editors, but we've educated our staff to really comb through the bibliography to look for hints that there may be something going on, so that we can direct a query to the author asking them to state whether or not such a tool was used.
Speaker 3:There is a arms race, if you will, going on amongst the very energetic and well-funded AI community between those generating new and innovative tools and those generating the ways to identify the use of those tools, so it is a rapidly evolving field. I think there is potential for huge positive impact in the use of these tools Because, as I'm sure our audience knows, where the weak spot is what is the information being used to inform the tool. So, if you can inform an AI search engine with only high quality, highly peer-reviewed information, you have tools that can be used to be quite useful at point-of-care decision-making, etc. But it's all about the data that's being fed to the tool.
Speaker 3:And if we can control those processes and have vetted curated information being added into the tools. I think there's a very big upside, but myself and my successor and those who also are editors in the field, we're really concerned about the potential for just flat out falsified information getting through, about the potential for just flat out falsified information getting through, which has happened as you know, highly publicized cases in other fields and I just hope and pray it never happens to us in orthopedics.
Speaker 2:But you do see this as becoming a greater issue, even though the policing efforts of this has increased. On the flip side side of this, the penitence of ai into this work is going to just only go get greater and greater correct.
Speaker 3:Both the negative and the positive are escalating at a rapid race that's a bit gloomy.
Speaker 2:Now I do have my copy of jbjs. I just got this morning here on the desk and I know that y'all and Journal of Orthopedic Trauma, jaos the big ones that I go through all the time are starting to move, or have moved for quite a while, to more of the virtual format. The paper edition has become thinner. I think I've heard y'all say that the advertisements or the marketing efforts within the journal are becoming more difficult. Where do you see that headed?
Speaker 3:Yeah, that whole business of the print ad revenue declining happened during my decade as editor-in-chief. We had a two-year period of time where the revenue from the print ads declined by two-thirds. Revenue from the print ads declined by two-thirds and that was a major revenue source for all scholarly publications. And if you look now, there's very few pages in any of the print journals that you see.
Speaker 3:So the thought was early on that we'll make it up on the ads on the websites and that really hasn't turned out to be anywhere close to replacing the revenue during print ads. And it's for multiple reasons consolidation among our orthopedic vendors you know now only really five major implant manufacturers, etc. And the feeling of these groups the manufacturers that they have direct access to their surgeons for innovation and decision-making regarding what's to be used, et cetera, so they don't have to go through journals or scholarly societies or whatever to have access to the more limited number of people making decisions about products the more limited number of people making decisions about products so it places real economic pressure on scholarly publications to find ways to break even.
Speaker 3:Jbjs is sound fiscally through the hard work of the editorial board and the trustees and the innovative, creative people that work at the journal outstanding young individuals who are thinking every day about ways to improve the product and find ways to generate revenue. So it is stable but the pressures are huge.
Speaker 3:It has been long predicted that the print will go away, but yet every time we've done surveys of our readers. The last time I saw data and I know we're serving again very soon but the last time I saw data would have been three years ago and it was still. The majority of surgeons under the age of 45 preferred the print version and that's because they can fold it up in their lab coat and if they have five minutes while the patient is getting the spinal, they can open it up without logging on and they can look at an article and read an abstract in much faster time than they can to log on and look something up. I don't think it's going to be completely gone in my professional lifetime. Could be wrong about that and maybe AI tools will supplant that. Hard to know, but it's not going to be gone soon.
Speaker 2:Fascinating. We've seen this big change in our career with any other aspects that you see in the next 10, 20 years or it's very difficult to predict that far out in terms of where our scholarly journals will be headed or changes. If we were to suddenly open a journal in 20 years, what it may look differently from today. Any Any other thoughts?
Speaker 3:I think that the ability to integrate more precise data collection efforts into the electronic medical record are going to increase in the next five to 10 years so that you will be able to prospectively include data that's important for patient decision making so you can deal with the limitations of the current administrative data sets.
Speaker 3:I think they're going to increase making the use of huge data sets of actual patient care information to be analyzable I don't think that's a word To be able to be analyzed in a positive way to get more detailed information on huge populations of patients.
Speaker 3:So I think that is going to impact our ability to advance knowledge to a much greater degree. There will always be a need for randomized trials because that is the most efficient way that you can deal with bias, and I think that we are going to get smarter about where we wish to invest those precious dollars to answer appropriate questions. I've been involved, as have many of my colleagues in orthopedic trauma, in many multi-center trials and, quite frankly, the ones that we have had funded I'm not sure they're the ideal clinical questions to. I believe that the leadership of NIH and NIAMS, which is our agency, would welcome processes to help focus topics which would have greater clinical impact than the way we do it now, which is it's really about the quality of the clinical question decided upon by a study section rather than a large group of peers. So expanded utility of actual patient records.
Speaker 3:Continued use of control trials in a more focused area and I think ways to add outcomes data to registries which should prove to get patients incentivized to provide dollars would be the third one. I have offered suggestions about how to get patients incentivized, like if you'll provide a validated survey to us every year that we send to you via text, you will get a 10% off of your deductible. Ways that you can incentivize patients to give us that information that would be the third one.
Speaker 2:That's fantastic. Lastly, on a personal note, I've heard you say this before and you're, you've been held. You've, your self-discipline has held you to this. You were chair at Minnesota for 10 years, right.
Speaker 3:Yes.
Speaker 2:And you were editor of JBJS for 10 years and I've heard you say this before that you feel that a 10 year, 10, 10 year tenure is, in your and your wisdom, was an adequate use of your expertise and your energy. And after 10 years you felt that I'm putting words in your mouth, that you felt that would wane and you should step aside and let somebody else take over, that you've held down. Did you say that? But you've actually done that. I remember hearing you say that it's just inspiring to actually you've actually said it and then you actually did it. I'm just opening the floor to that.
Speaker 3:Yeah, it takes a lot of energy. You've led big organizations. You're no stranger to how much energy it takes to lead physicians, and whenever you get a new position you get resources I call them bullets and you spend the bullets on initiatives that are important to you and I do not know of any experience I've ever had where you can say after I think it's actually eight years that you have many bullets left. You've spent the goodwill, you've driven the things that are important to you and it is just worked out in my experience that at beginning, at eight years, you ought to be trying to help the organization select your successor, and organizations are always improved by having a different perspective and a new leader at the table. I think that many people hang on to these positions because they're kind of afraid to leave something that they know and can control and venture into something different. But I myself and I know many of us in orthopedic surgery I've always felt the best part was taking care of patients. So I never had any fear of actually going back and being what I call a real doctor, because that's still the best part. You know, sitting around a table and listening to colleagues whine about the donuts in the doctor's lounge is quite energy draining and it's always been very reinforcing to have the fallback position to be what I enjoy the most.
Speaker 3:Yeah, I think that the 10-year rule is good. It's good for an individual. It's also great for an organization. Many deans don't want to change at that and I had to. I actually had a signed letter when I took the position that I wouldn't do it for more than 10 years. I made the dean sign it and I had to use it on the on that dean successor to to get the process done. But in many organizations are improved by an internal selection, which is number one, less expensive. Number two the successor is going to get an infusion of resources which benefits everybody in the group and it's low risk because everybody knows the person and they know what's going to happen tomorrow. So I'm a huge advocate of the 10-year rule and I wish more people in our field would jump on board. But I haven't seen too many people willing to do it, but for me it's been very energizing every time I did it.
Speaker 2:Wow. Okay, it's been an absolute pleasure to discuss these issues today with Dr Mark Swinkowski, who is the immediate past editor of the Journal of Bone and Joint Surgery and past AOA president. Dr Swinkowski, thank you very much for being on the podcast series, sir.
Speaker 3:Thank you very much, Dr Lundy, for the invitation. I've enjoyed chatting with you.
Speaker 2:Yes, sir, and y'all look forward to future AOA, future and orthopedic podcast series episodes. Thank you.