Can ChatGPT Do Your Next Systematic Review?​ The short answer is yes. However, not in the way you think. When trying to use the ChatGPT user interface for screening or data extraction for your systematic review, you run fast into problems. You cannot upload a PDF, so you need to copy and paste the text, and then you run into the limited prompt length. And OpenAI has yet to start thinking about how to verify the quality of GPTs’ work for systematic reviews.

It was discovered that ChatGPT and other LLMs have the potential to assist with some review tasks, like formulating a structured review question and creating an initial search query for PubMed. However, the authors also found that challenges remain in generating the entire usable search strategies and synthesizing multiple studies requiring expert verification. (Qureshi, R., Shaughnessy, D., Gill, K.A.R., et al., 2023). At Pitts, we have been working on integrating GPT into our web application for systematic reviews. In this first iteration of this integration, we have focused on GPT-assisted data extraction. And we are making it easy to verify the accuracy of the GPT data extractions.

At Pitts, we have been working on integrating GPT into our web application for systematic reviews. In this first iteration of this integration, we have focused on GPT-assisted data extraction. And we are making it easy to verify the accuracy of the GPT data extractions.

How does it work?
After you have uploaded your search results or retrieved them directly from PubMed via the Pitts web interface, you must first do the abstract and full-text screening. Unlike the ChatGPT tool, you can upload PDFs to the Pitts tool. After completing this for at least one study, you can use GPT for data extraction. Go to the review setting and add tabs and columns to your data extraction sheet. You can then go back to your review and start data extraction. A GPT prediction box is present, with a button to configure it if not done already.

In the demo review you don’t have to screen first, as we already have some studies included for you as well as set up the settings for the review. You can check out data extraction with GPT right away. Have a look here: app.pitts.ai

It’s worth noting that this integration with GPT comes with caution. The technology is still in its infancy and requires substantial development to fully apply to all systematic review tasks. In trials, there were clear areas where GPT’s output was valid for some data extraction tasks, yet there were also numerous instances where the information was erroneous and needed manual correction. Therefore, considerable expertise in the subject matter is still necessary to ensure the reliability and validity of the data extracted with GPT’s assistance. The “uncanny valley” in GPT’s current performance highlights the technology’s potential yet underscores the importance of expert involvement in the process. The number one priority at Pitts is to develop further the tool to allow users to validate the use of GPT for data extraction further scientifically. 

References:
Qureshi, R., Shaughnessy, D., Gill, K.A.R. et al. Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation?. Syst Rev 12, 72 (2023). https://doi.org/10.1186/s13643-023-02243-z