Towards a Contextualized Understanding of Scholarly Impact: The Development and Implications of the Contextualized Scholarly Impact Index (CSII)
DOI:
https://doi.org/10.70142/ijbge.v1i3.242Keywords:
Contextualized Scholarly Impact Index (CSII), Scholarly impact assessment, Qualitative literature review, Research evaluation, Academic contributionsAbstract
This qualitative literature review explores the development and implications of the Contextualized Scholarly Impact Index (CSII), a framework designed to address the limitations of traditional metrics in assessing scholarly contributions. The review highlights the need for a more nuanced understanding of scholarly impact that incorporates contextual factors, such as collaboration, societal relevance, and interdisciplinary engagement. Through a comprehensive analysis of existing literature, the review identifies key themes that underscore the importance of recognizing diverse forms of scholarship, including public engagement and community involvement. The findings indicate that the CSII has the potential to transform the landscape of scholarly impact assessment by promoting a holistic and equitable evaluation framework. This shift aligns with the growing consensus within the academic community for more inclusive methodologies that reflect the multifaceted nature of research contributions. However, the review also acknowledges limitations, including the subjective interpretation of literature, the rapidly evolving nature of the field, and the need for empirical validation of the CSII. Ultimately, this review lays the groundwork for future research and practice, advocating for a contextualized approach that enriches our understanding of scholarly impact in contemporary academia.
References
Abritis, A., McCook, A., & Watch, R. (2017). Cash bonuses for peer-reviewed papers go global. Science. https://www.science.org/content/article/cash-bonuses-peer-reviewed-papers-go-global
Aguinis, H. (2025). Research methodology: Best practices for rigorous, credible, and impactful research. Atlanta: SAGE.
Aguinis, H., Archibold, E. E., & Rice, D. B. (2022). Let’s fix our own problem: Quelling the irresponsible research perfect storm. Journal of Management Studies, 59(7), 1628–1642. https://doi.org/10.1111/joms.12766
Aguinis, H., Cummings, C., Ramani, R. S., & Cummings, T. G. (2020). “An A is an A”: The new bottom line for valuing academic research. Academy of Management Perspectives, 34(2), 135–154. https://doi.org/10.5465/amp.2018.0148
Aguinis, H., Dalton, D. R., Bosco, F. A., Pierce, C. A., & Dalton, C. M. (2011). Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact. Journal of Management, 37(1), 5–38.
https://doi.org/10.1177/0149206310382910
Aguinis, H., & Gabriel, K. P. (2022). If you are serious about impact, create a personal impact development plan. Business & Society, 61(5), 818–826.
https://doi.org/10.1177/00076503221107474
Aksnes, D. W. (2003). A macro study of the relationship between publication and citation in the social sciences. Scientometrics, 57(3), 391–406.
https://doi.org/10.1023/A:1021804415996
Alperin, J. P., & others. (2019). The role of institutional policies in promoting open access. PLOS ONE, 14(4), e0215333. https://doi.org/10.1371/journal.pone.0215333
Alvesson, M., & Sandberg, J. (2011). Generating research questions through problematization. Academy of Management Review, 36(2), 247–271.
https://doi.org/10.5465/amr.2011.65753334
Bakkalbasi, N., Bauer, K., Glover, J., & Wang, M. (2006). Three options for citation tracking: Google Scholar, Scopus, and Web of Science. Biomedical Digital Libraries, 3(1), 7. https://doi.org/10.1186/1742-5581-3-7
Beltran, J. R., Aguinis, H., Shuumarjav, Y., & Mercado, M. (2024). Putting scholarly impact in context: Implications for policymaking and practice. Academy of Management Perspectives, 38(3). https://doi.org/10.5465/amp.2023.0198
Björk, B.-C., Roos, A., & Lauri, M. (2015). Academic publishing 2.0: Developing a new model for academic publishing. Information Research, 20(1), 674.
http://www.informationr.net/ir/20-1/paper674.html
Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of Documentation, 69(3), 391–410. https://doi.org/10.1108/JD-11-2011-0090
Bornmann, L. (2014b). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8, 895–903.
https://doi.org/10.1016/j.joi.2014.09.006
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Booth, A., Sutton, A., & Papaioannou, D. (2016). Systematic approaches to a successful literature review (2nd ed.). London: SAGE Publications.
Denzin, N. K., & Lincoln, Y. S. (Eds.). (2018). The sage handbook of qualitative research (5th ed.). Thousand Oaks, CA: SAGE Publications.
Edwards, M. A., & Roy, S. (2017). Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environmental Engineering Science, 34, 51–61. https://doi.org/10.1089/ees.2016.0245
Enders, J., & Bornemann, M. (2015). The impact of research evaluation on research productivity and quality: A comparative analysis of institutional strategies. Research Policy, 44(7), 1295–1307. https://doi.org/10.1016/j.respol.2015.04.001
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. London: SAGE Publications.
Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews. Thousand Oaks, CA: SAGE Publications.
Hammarberg, K., Kirkman, M., & de Lacey, S. (2016). Qualitative research methods: When to use them and how to judge them. Human Reproduction, 31(3), 498–501.
https://doi.org/10.1093/humrep/dev334
Harzing, A. W., & Alakangas, S. (2016). Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2), 787–804. https://doi.org/10.1007/s11192-015-1798-9
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102, 16569–16572. https://doi.org/10.1073/pnas.0507655102
Kim, H., & Kwon, S. (2019). The impact of institutional support on research performance: The mediating role of researcher motivation. Journal of Research Administration, 50(1), 55-71. https://doi.org/10.2139/ssrn.3290272
Krefting, L. (1991). Rigor in qualitative research: The assessment of trustworthiness. American Journal of Occupational Therapy, 45(3), 214–222.
https://doi.org/10.5014/ajot.45.3.214
Larivière, V., & Sugimoto, C. R. (2019). The journal impact factor: A brief history, critique, and discussion of adverse effects. In W. Glanzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 3–24). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-030-17045-1_1
Lee, Y. S., & Bozeman, B. (2005). The impact of research collaboration on research output. International Journal of Technology Management, 29(2), 151-171.
https://doi.org/10.1504/IJTM.2005.006487
Moed, H. F. (2017). Citation analysis in research evaluation. New York: Springer.
National Academy of Sciences. (2010). Preparing for the future of the U.S. research enterprise. Washington, DC: National Academies Press.
National Institutes of Health. (2015). The impact of research on public health. Retrieved from https://www.nih.gov/
Priem, J., & Hemminger, B. H. (2010). Scientometrics 2.0: New metrics of scholarly impact. The Future of Scholarly Communication, 4(1), 5–11.
https://doi.org/10.2218/ijdc.v4i1.113
REF (2014). Assessment framework and guidance on submissions. Retrieved from https://www.ref.ac.uk/
Ramani, R. S., Aguinis, H., & Coyle-Shapiro, J. A. (2022). Defining, measuring, and rewarding scholarly impact: Mind the level of analysis. Academy of Management Learning & Education, 21, 470–486. https://doi.org/10.5465/amle.2020.0223
Robinson, J. A., & Moniz, D. (2015). The role of social media in the dissemination of research findings. Research Evaluation, 24(3), 234–245. https://doi.org/10.1093/reseval/rvv007
Salter, A., & Martin, B. R. (2001). The economic benefits of publicly funded research. Research Policy, 30(2), 247–262. https://doi.org/10.1016/S0048-7333(00)00092-6
Scott, P. (2018). Impact, metrics and the measurement of research quality: The changing nature of research evaluation in the UK. Journal of Higher Education Policy and Management, 40(1), 33-48. https://doi.org/10.1080/1360080X.2017.1368099
Swan, A. (2013). Policy guidelines for the development of research assessment. International Journal of Research, 8(3), 299–308. https://doi.org/10.2139/ssrn.2566342
Taylor, M., & Toffoli, L. (2020). The importance of scholarly impact: Recommendations for improving the assessment of research output. Research Evaluation, 29(1), 45–51. https://doi.org/10.1093/reseval/rvz030
Van Raan, A. F. J. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143. https://doi.org/10.1007/s11192-005-0008-y
Van Noorden, R. (2014). Online collaboration: The researchers who share. Nature, 512, 126–129. https://doi.org/10.1038/512126a
Wagner, C. S., & Leydesdorff, L. (2005). Network structure, self-organization, and the growth of international collaboration in science. Research Policy, 34(10), 1608–1618.
https://doi.org/10.1016/j.respol.2005.08.003
Woolley, J. K., & MacMillan, S. (2020). Research output and impact: A review of the literature. Journal of Research Policy, 49(3), 210–222.
