VU Systematic Literature Reviews: Test your search strategy and monitor its development

Test your strategy

Once you have identified relevant keywords and subject headings and developed a search strategy, the next step is to run a test search. This is the process where you test whether your search strategy picks up a particular group of articles (otherwise known as a 'test set or 'gold set') you would expect your search strategy to retrieve. If your search retrieves all records, you can safely say it is a successful strategy. If not, the results that weren't retrieved should be examined to identify additional search terms (text words and indexing) or highlight potential limitations. Depending upon the complexity of the review topic, this process may need to be repeated several times until an agreed strategy is formulated. Furthermore, it is strongly recommended by Cochrane's handbook that any search strategies are peer reviewed before they are run (Cochrane handbook, Ch. 4.4.8). This includes the process of testing and adjusting the search as necessary.  

The following steps outline how to run a pilot search on a set of test records to see if your search strategy is successful:

  1. Identify a 'gold set' of articles that fit your research question well (this 'set' should be agreed upon by the reviewers, or selected from studies included in an earlier review) and are all indexed in the same database that you are running the test search in.
  2. Search the database for each of the test records and make a note of the unique record number for each one - in Medline and PubMed this is in the PMID field.
  3. Run your search strategy.
  4. Run a search for all the record numbers for your test set using 'OR' in between each one.
  5. Lastly combine the result of your search strategy with the test set using 'OR'.
  6. If the number of records retrieved stays the same then the strategy has identified all the records. If it doesn't, combine the result of your search strategy with the test set, this time using 'NOT'. This will identify the records in your test set which are not being retrieved.
  7. Examine the results not being retrieved to identify additional search terms and incorporate into your revised search strategy.
  8. Peer review and re-test revised search strategy.

Once you have refined your search strategy, you can begin recording your search results and remap your search to other databases. However, this does not mean that search refinements should necessarily stop. If new search terms are identified during the search process they should be incorporated into the strategy or supplementary searches should be carried out.

Make an appointment with a librarian for help testing your search strategy. 

References:

University of Reading Library (2022, Jul 13). Step 3: developing a search strategy. University of Reading. https://libguides.reading.ac.uk/systematic-review/search-strategy