Development and content based validity evidence for the Social Thinking and Identity Diversity PSDI instrument for qualitative data collection in higher education
DOI:
https://doi.org/10.63688/aprendizaje.v3.i2.50Keywords:
Social thinking, identity diversity, inclusive education, content validity, higher education.Abstract
The aim of this study is to develop and obtain content-based validity evidence for the Social Thinking and Identity Diversity (PSDI) instrument, a structured guide for individual interviews and focus groups designed to explore representations, experiences, and positionings related to social thinking, inclusive education, and identity diversity in higher education. Using a sequential instrumental design, 34 expert judges from three areas of expertise evaluated 19 stimuli across two successive rounds according to their relevance, representativeness, essentiality, and five technical quality criteria. The first round identified four stimuli that required reformulation. Following their revision, all 19 met the established decision rules (Aiken’s V = .912–.971, CVC = .934–.978, and CVR = .588–.941). Agreement and interrater reliability increased substantially (Kendall’s W = .843; Fleiss’ κ = .803; Krippendorff’s ordinal α = .837), with no significant differences across areas of expertise. The results indicate that the PSDI constitutes a methodologically grounded and sufficiently refined qualitative guide for proceeding to pilot testing in university educational settings.References
Aiken LR. Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement 1985;45(1):131–142. https://doi.org/10.1177/0013164485451012
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. American Educational Research Association; 2014. https://www.testingstandards.net/open-access-files.html
American Educational Research Association. AERA code of ethics: American Educational Research Association approved by the AERA Council February 2011. Educational Researcher 2011;40(3):145–156. https://doi.org/10.3102/0013189X11410403
Ato M, López-García JJ, Benavente A. Un sistema de clasificación de los diseños de investigación en psicología. Anales de Psicología 2013;29(3):1038–1059. https://doi.org/10.6018/analesps.29.3.178511
Ayre C, Scally AJ. Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development 2014;47(1):79–86. https://doi.org/10.1177/0748175613513808
Bangdiwala SI, Shankar V. The agreement chart. BMC Medical Research Methodology 2013;13:97. https://doi.org/10.1186/1471-2288-13-97
Betancourth-Zambrano S, Martínez-Daza V, Tabares-Díaz YA. Evaluación de pensamiento crítico en estudiantes de Trabajo Social de la región de Atacama-Chile. Entramado 2020;16(1):152–164. https://doi.org/10.18041/1900-3803/entramado.1.6139
British Educational Research Association. Ethical Guidelines for Educational Research. 5th ed. British Educational Research Association; 2024. https://www.bera.ac.uk/publication/ethical-guidelines-for-educational-research-fifth-edition-2024
Calderón Jaramillo AM. El pensamiento social y los temas sociales relevantes: Una reflexión sobre la formación de los psicólogos. Nuevas Dimensiones 2018;7:2–20. https://doi.org/10.53689/nv.vi7.1
Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960;20(1):37–46. https://doi.org/10.1177/001316446002000104
Duran S, Parra M. Diversidad cultural para promover el desarrollo de habilidades sociales en educación superior. Cultura, Educación y Sociedad 2014;5(1):55–67. https://revistascientificas.cuc.edu.co/index.php/culturaeducacionysociedad/article/view/995
European Parliament, Council of the European Union. Regulation (EU) 2016/679 of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union 2016;L119:1–88. https://eur-lex.europa.eu/eli/reg/2016/679/oj
Fleiss JL. Measuring nominal scale agreement among many raters. Psychological Bulletin 1971;76(5):378–382. https://doi.org/10.1037/h0031619
Hallgren KA. Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology 2012;8(1):23–34. https://doi.org/10.20982/tqmp.08.1.p023
Harpe SE. How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning 2015;7(6):836–850. https://doi.org/10.1016/j.cptl.2015.08.001
Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 2007;1(1):77–89. https://doi.org/10.1080/19312450709336664
Hernández-Nieto R. Contributions to Statistical Analysis: The Coefficients of Proportional Variance, Content Validity and Kappa. BookSurge Publishing; 2002. https://books.google.com/books?id=yk_2PAAACAAJ
Hurtado S. Diversity and learning for a pluralistic democracy. In: Allen WR, Bonous-Hammarth M, Teranishi RT, Dano OC, editors. Higher Education in a Global Society: Achieving Diversity, Equity and Excellence. Advances in Education in Diverse Communities. Vol. 5. Emerald Group Publishing; 2005. p. 249–267. https://doi.org/10.1016/S1479-358X (05)05012-6
Kendall MG, Babington Smith B. The problem of m rankings. The Annals of Mathematical Statistics 1939;10(3):275–287. https://doi.org/10.1214/aoms/1177732186
Krippendorff K. Content Analysis: An Introduction to Its Methodology. 4th ed. SAGE; 2018. https://methods.sagepub.com/book/mono/content-analysis-4e/toc
Kruskal WH, Wallis WA. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association 1952;47(260):583–621. https://doi.org/10.1080/01621459.1952.10483441
Lawshe CH. A quantitative approach to content validity. Personnel Psychology 1975;28(4):563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
López-Ruiz C, Flores-Flores R, Galindo-Quispe A, Huayta-Franco Y. Pensamiento crítico en estudiantes de educación superior: Una revisión sistemática. Revista Innova Educación 2021;3(2):374–385. https://doi.org/10.35622/j.rie.2021.02.006
Muñoz SR, Bangdiwala SI. Interpretation of kappa and B statistics measures of agreement. Journal of Applied Statistics 1997;24(1):105–112. https://doi.org/10.1080/02664769723918
Nevárez Jiménez LF, Velasco Crespo DR, Martínez Ángulo MA. Impacto de la educación inclusiva en el desarrollo de competencias socioemocionales y su influencia en la convivencia universitaria y el rendimiento académico en los estudiantes universitarios, con un enfoque en la salud mental. Reincisol 2025;4(7):3402–3425. https://doi.org/10.59282/reincisol.V4(7)3402-3425
Ordaz Hernández M. La educación de habilidades sociales desde la Extensión Universitaria: Propuesta de acciones. Educar em Revista 2013;29(50):269–283. https://doi.org/10.1590/S0104-40602013000400017
Ortega-Sánchez D, Bernal López-Sanvicente A. Design, content validity, and inter-observer reliability of the “Digitization of Cultural Heritage, Identities, and Education” (DICHIE) instrument. Humanities and Social Sciences Communications 2023;10:53. https://doi.org/10.1057/s41599-023-01550-z
Paz Maldonado E. Competencias del profesorado universitario para la atención a la diversidad en la educación superior. Revista Latinoamericana de Educación Inclusiva 2018;12(2):115–131. https://doi.org/10.4067/S0718-73782018000200115
Peifer JS, Taasoobshirazi G. Heightened cognitive empathy through ethnic identity and intercultural competence for individuals with historically marginalized identities. International Journal on Social and Education Sciences 2022;4(2):290–306. https://doi.org/10.46328/ijonses.324
Penfield RD, Giacobbi PR Jr. Applying a score confidence interval to Aiken’s item content-relevance index. Measurement in Physical Education and Exercise Science 2004;8(4):213–225. https://doi.org/10.1207/s15327841mpee0804_3
Pérez Serrano G, Sarrate Capdevila ML. Diversidad cultural y ciudadanía: Hacia una educación superior inclusiva. Educación XX1 2013;16(1):85–104. https://doi.org/10.5944/EDUCXX1.16.1.718
Simons L, Schoneman S, Hoffman M, Blank N. Equity, inclusion, diversity, and belonging in a multicultural psychology course during the COVID-19 pandemic. In: Mohiuddin M, Aziz MT, Jayashree S, editors. Multiculturalism and Interculturalism: Managing Diversity in Cross-Cultural Environment. IntechOpen; 2023. https://doi.org/10.5772/intechopen.109316
Solbes J, Torres Merchán NY. Análisis de las competencias de pensamiento crítico desde el aborde de las cuestiones sociocientíficas: Un estudio en el ámbito universitario. Didáctica de las Ciencias Experimentales y Sociales 2012;26:247–269. https://doi.org/10.7203/DCES.26.1928
Published
Issue
Section
License
Copyright (c) 2026 Delfín Ortega-Sánchez, Carlos Pérez-González (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
The article is distributed under the Creative Commons Attribution 4.0 License. Unless otherwise stated, associated published material is distributed under the same licence.
