Reuse Of Prescribing Assessment Items And Student Performance: An Observational Study

Background and aims: The General Medical Council requires that medical graduates are able to ‘prescribe drugs safely, effectively and economically’ (1). Prescribing assessments are now used by most medical schools to determine whether students have achieved these outcomes, either in the form of local examinations or through participation in the UK Prescribing Safety Assessment. Most prescribing assessments incorporate some items reused from previous years, so as to reduce the time and cost of developing the assessment and to allow comparisons of year-on-year performance on anchor questions. This approach assumes that candidates have no prior knowledge of the reused questions, an assumption that has recently been questioned (2). In this study we sought to establish whether candidates performed differently on items when they were reused, as compared to their first use.

Nature of the assessment: The St George’s, University of London prescribing skills test is a high-stakes, final-year assessment taken by approximately 300 candidates each April. It is completed on paper and comprises 10 free text prescribing questions (6 items each), 45 single best answer (SBA) items and 10 dose calculation items. The pass mark is set by a modified Angoff method. All items in April 2013 were new for that assessment. Seven free text prescribing questions and 24 SBAs from April 2013 were reused on one occasion, either in 2014 or 2015.

Methods: The facility of each item was defined as the proportion of candidates answering the item correctly. A paired t-test was used to compare item facility (or mean facility in the case of the 6-item prescribing questions) on use and reuse. A regression analysis was used to assess the association between reuse and facility after accounting for relevant confounders (year of use, question type and Angoff score).

Results: The mean change in facility on reuse, as compared to first use, was -0.01 (95% CI −0.09 to +0.07, P=0.755) on prescribing questions and -0.01 (95% CI −0.05 to +0.03, P=0.564) on SBAs. On regression analysis, facility was significantly associated with year of use and Angoff score. No significant association was found with item reuse, and no interaction was found between reuse and other variables.

Discussion and conclusion: In this local prescribing assessment we found no evidence that reuse of items was significantly associated with facility. Although it is difficult to determine what a meaningful change in facility would be, the confidence intervals are reassuringly close to zero. Given the advantages offered by item reuse, in terms of deliverability of the assessment and year-on-year comparisons, we have found no compelling evidence to depart from our current practice. However, we are mindful that this analysis was limited to a comparison of facility on a relatively small number of items, and that it provides only a snapshot. Ongoing vigilance and further analysis are required.

References:

1. Outcomes for graduates (Tomorrow’s Doctors). General Medical Council (2015).

2. Tonkin AL (2015). BMJ 351: h4014.