An AI-supported scoping review can use AI for data extraction while keeping analytical decisions under human control. A rigorous approach is to compare manual and AI extraction, reconcile disagreements against the original papers, and manually synthesise and chart the data. AI can act as a second pair of eyes, but it should not replace researcher judgement.

How can AI be used for data extraction in a scoping review?

Data extraction and charting form an important part of the analysis in a scoping review. When you are working as the sole reviewer, AI can provide an additional check on your extraction.

I used a three-step process:

1. Manual extraction

2. AI extraction

3. Reconciliation

The manual extraction involved reading through the 39 papers and extracting the information relevant to the review. I often included parts of the original writing or summaries where appropriate. This took considerable time because of the number of papers involved.

For the second stage, I used Elicit AI. It allows you to upload papers and ask questions about them. I used it to extract information from the papers, with additional prompting where the questions were specific to my review.

Elicit produced an extraction table containing data, direct quotations and the AI's reasoning. The AI-supported process was intended to make the extraction more rigorous rather than replace the analytical decisions I needed to make as the researcher.

How do you reconcile manual and AI data extraction?

In a typical scoping review with multiple reviewers, each reviewer conducts data extraction and the reviewers then discuss and combine their findings. As I was working alone, I needed to reconcile the differences between my manual extraction and the extraction produced by Elicit AI.

Where the information matched but the wording differed, I chose the more specific explanation. Where the two extractions disagreed, I returned to the original paper and verified the information.

This was often faster than checking the papers from scratch because Elicit provided specific quotations. I could use those quotations to quickly locate the relevant information in the original paper.

AI extraction was not perfect, but neither was my manual extraction. Elicit frequently got basic information such as authors, country and publication year wrong. It sometimes extracted editors and reviewers as authors, confused the country of publication with the country of the journal, and confused the year of publication with the year the paper was accepted.

Participant numbers could also be incorrect, with the AI sometimes confusing numbers or participant groups.

What problems can AI have when extracting data from research papers?

AI also produced problems with more complex extraction questions. For example, when I asked for references cited for the thematic analysis, Elicit sometimes reported that no citation was provided even when I had identified one during manual extraction.

The complexity of thematic analysis created another problem. Its nebulous nature meant that AI sometimes confused thematic analysis with sentiment analysis, particularly where both approaches appeared in the same study or where themes were generated at multiple stages.

However, AI performed better when producing narrative synthesis. It was able to make sense of long and complex descriptions of how AI was used within a study and often produced more detailed and succinct summaries than I had created manually.

This difference was important. AI was useful for helping to process and summarise complex information, but its output still required checking against the original research papers.

The comparison between manual and AI extraction therefore became a form of quality control. Neither source was assumed to be automatically correct.

How should you synthesise and chart extracted data?

Synthesis and charting required a different approach, and I completed these parts manually.

For example, I initially extracted participant counts together with basic descriptions of the participants. However, this created a large amount of writing within a single extraction column, making it difficult to identify and compare participant numbers.

I therefore created a cleaner column containing only the participant numbers. This allowed me to use filtering in Excel or order the papers by participant count, making the information easier to work with when writing the results.

I also established categories when patterns became apparent in the extracted data. For example, I categorised when AI was used within three different stages of thematic analysis.

For some justification columns, I created a checkbox system. This allowed me to filter the extraction table and check counts across the papers more easily.

These decisions were not always planned at the beginning. Sometimes the messiness only became apparent during the writing process. When that happened, I returned to the extraction table and reorganised it so that the data was easier to interpret and less likely to lead to confusion or mistakes.

What can you learn from comparing AI and manual data extraction?

Cross-checking manual and AI data extraction was valuable because both the researcher and the AI can make mistakes.

Comparing the two allowed me to identify and correct errors that could have damaged the rigour of the review. The process also showed that different approaches were better suited to different types of information.

AI was particularly useful for narrative synthesis and for identifying and summarising complex descriptions. However, it was unreliable for some basic factual information and struggled with the more ambiguous aspects of thematic analysis.

I would therefore be cautious about allowing AI to conduct data extraction without strict oversight. For students and researchers who do not have the resources to hire a second reviewer, however, AI can provide a useful second pair of eyes.

Used in this way, AI can support a solo researcher in checking their work and improving the rigour of the review while leaving the analytical decisions and final judgement with the researcher.

If you're applying this to your own work and want a second pair of eyes: https://www.drkenyanwong.com/start-here