Why many medical schools don’t use advanced stats for assessment data analysis

By Jerome Rotgans


The analysis of assessment data is an important part of medical education since it allows educators to evaluate the effectiveness of their teaching methods and identify areas for improvement. Assessment data is also crucial for determining whether students have mastered the required knowledge and skills to advance to the next stage in their training. However, despite the general acceptance of the importance of assessment data in medical education, many medical schools may not be using advanced statistical methods to analyze their assessment data. Let’s take item-response theory (IRT) as an example of how it can improve the analysis of assessment data.

One of the primary advantages of IRT is that it provides more accurate estimates of student ability levels. Unlike classical test theory analyses, which assume that all test items are equally difficult, IRT takes into account the difficulty level of each test item and the ability level of each student. This allows for a more precise estimation of student ability levels, which can be useful in identifying areas where students may need additional support.

Additionally, IRT can help identify which test items are most effective in measuring student knowledge and skills. By examining the performance of each test item, IRT can identify which items are providing the most information about student ability levels and which items may need to be revised or removed.

Finally, IRT can provide more information about the overall quality of the assessment instrument. By examining the performance of each test item, IRT can provide a measure of the reliability and validity of the assessment instrument. This can help educators ensure that the assessment instrument is measuring what it is intended to measure and that the results are accurate and reliable.

Despite the clear advantages of using advanced statistical methods, such as IRT, for analyzing assessment data, it is surprising that many medical schools do not fully utilize these capabilities.

Here are a few major reasons why that may be the case.

One of the main reasons is a lack of expertise among faculty and staff. Advanced statistical methods require specialized knowledge and skills, and some medical schools may not have staff with the necessary expertise to implement these methods effectively.

Additionally, it is often assumed that implementing advanced statistical methods may require the purchase of expensive software or hiring of consultants, which may not be feasible for some medical schools due to budget constraints.

Time constraints can also be a factor. Medical schools may not have the time or resources to invest in implementing and analyzing assessment data using advanced statistical methods, particularly if they have limited staff or faculty resources, as it is the case in many medical education departments. Moreover, some medical schools may not see the benefits of using advanced statistical methods, particularly if they are satisfied with their current methods of analyzing assessment data.

Despite these challenges, some medical schools use advanced statistical methods to analyze their assessment data, particularly those with a strong emphasis on research or with specialized expertise in this area. However, for many medical schools, other factors may take priority over the use of advanced statistical methods when it comes to analyzing assessment data.

Here are some simple solutions how medical schools can get started with dealing with these challenges.

There are several solutions that medical schools can implement to overcome the challenges of using advanced statistical methods for analyzing assessment data. One approach is to increase the expertise in statistical analysis by providing training programs for faculty and staff or hiring staff with specialized statistical knowledge. Collaborating with external experts or partnering with other institutions to share knowledge and resources can also be beneficial. Once the expertise is available, many data processing and analysis processes can be automated and require less manpower. In addition, cost can be reduced by exploring open-source software or free statistical analysis tools, such as R Studio, jamovi and JASP. Collaborating with other departments within the institution, such as the statistics department, can help share the cost of purchasing software licenses.

By implementing these solutions, medical schools can make advanced statistical analysis of assessment data more accessible and maximize the benefits of these methods to enhance the quality of medical education. With the right approach, all medical schools can effectively use advanced statistical methods to analyze assessment data and improve the outcomes of medical education.