Validating the Development of Instrument for Measuring Nurses’ Performance Scale


Inter Rater Reliability, Content Validity, Nurses’ Performance Scale, Expert Review

How to Cite

Haron, S., Suzana Ariffin, A., & Idrus, D. (2019). Validating the Development of Instrument for Measuring Nurses’ Performance Scale. Journal of Management Info, 6(1), 31-38.


Measuring and evaluating nurses’ performance are vital to identify areas for improvement in maintaining quality of service delivery and ensuring sustainability of current practices. This study attempts to examine the content validity of the nurses’ performance scale. It is also aimed to achieve acceptable criteria for content validity of this instrument. Construct and content domain of nurses’ performance were identified followed by items generation and instrument formation. Subsequently, assessments of content validity were performed based on content validity Index (CVI), Inter-rater agreement percentage (IRA%) and modified Kappa statistic. Two level of judgement were performed using the lay expert panel and research expert panel. Criteria were established based on these indices as basis for item reduction process. Pilot study was conducted on 50 respondents to assess the internal consistency using Cronbach’s alpha value of finalized NPQ instrument. 71 items are yielded during developmental stage of instrument to measure four dimensions of nurses’ performance. Assessment of content validity based on lay and research expert judgement resulting in elimination of 27 items (38%). Computed modified Kappa statistic further supplemented that the remaining 44 items as ‘excellent’. As for conclusion, NPQ instrument has attain acceptable criteria of content validity assessment utilized in this study and therefore proved its potential for further research


Abd Manaf, N. H., Abdullah, A. H. A., Abu Bakar, A., Ali, R., Bidin, N., Ismail, W. I. W., … Wan Ismail, W. I. (2011). "Hospital waiting time: the forgotten premise of healthcare service delivery? International Journal of Health Care Quality Assurance, 24(7), 506–522.
Agarwal, H. S., Saville, B. R., Slayton, J. M., Donahue, B. S., Daves, S., Christian, K. G., … Harris, Z. L. (2012). Standardized postoperative handover process improves outcomes in the intensive care unit: A model for operational sustainability and improved team performance. Critical Care Medicine, 40(7), 2109–2115.
Al-Makhaita, H. M., Sabra, A. A., & Hafez, A. S. (2014). Job performance among nurses working in two different health care levels, Eastern Saudi Arabia: A comparative study. International Journal of Medical Science and Public Health, 3(7), 832–837.
Ali Memon, M., Ting, H., Ramayah, T., Chuah, F., & Cheah, J.-H. (2017). Editorial - A Review of the Methodological Misconceptions and Guidelines Related to the Application of Structural Equation Modelling. Journal of Applied Structural Equation Modeling, 1(1).
Ali, N., Tretiakov, A., & Whiddett, D. (2014). A Content Validity Study for a Knowledge Management Systems Success Model in Healthcare. Jitta, 15(2), 21–36.
Beckstead, J. W. (2009). Content validity is naught. International Journal of Nursing Studies, 46(9), 1274–1283.
Blazun, H., Kokol, P., & Vosner, J. (2015). Survey on specific nursing competences: Students’ perceptions. Nurse Education in Practice, 15(5), 359–365.
Bogaert, P. Van, Peremans, L., Heusden, D. Van, Verspuy, M., Kureckova, V., Van De Cruys, Z., & Franck, E. (2017). Predictors of burnout, work engagement and nurse reported job outcomes and quality of care: a mixed method study. BMC Nursing, 16(5), 1–14.
Bramley, L., & Matiti, M. (2014). How does it really feel to be in my shoes? Patients’ experiences of compassion within nursing care and their perceptions of developing compassionate nurses. Journal of Clinical Nursing, 23(19–20), 2790–2799.
Brooten, D., Youngblut, J. M., & Youngblut, J. M. (2006). Nurse dose as a concept.(patient nurses ). Journal of Nursing Scholarship, 38(1), 94.
Cha, E. S., Kim, K. H., & Erlen, J. A. (2007). Translation of scales in cross-cultural research: Issues and techniques. Journal of Advanced Nursing, 58(4), 386–395.
Chan, E. A., Jones, A., Fung, S., & Wu, S. C. (2012). Nurses’ perception of time availability in patient communication in Hong Kong. Journal of Clinical Nursing, 21(7–8), 1168–1177.
Chan, J. (2014). Modelling The gynecologic Oncology Workforce Using System Dynamics. University of Toronto.
Cho, E., & Kim, S. (2015). Cronbach’s Coefficient Alpha. Organizational Research Methods, 18(2), 207–230.
Ciconelli, R. M., de Soárez, P. C., Kowalski, C. C. G., & Ferraz, M. B. (2006). The Brazilian Portuguese version of the Work Productivity and Activity Impairment - General Health (WPAI-GH) Questionnaire. Sao Paulo Medical Journal, 124(6), 325–332.
Cimiotti, J. P., Aiken, L. H., Sloane, D. M., & Wu, E. S. (2012). Nurse staffing, burnout, and health care-associated infection. American Journal of Infection Control, 40(6), 486–490.
Coatsworth, K., Hurley, J., & Miller-Rosser, K. (2015). A phenomenological study of student nurses volunteering in Nepal: Have their experiences altered their understanding of nursing? Collegian.
Dagne, T., Beyene, W., & Berhanu, N. (2015). Motivation and Factors Affecting it… Motivation and Factors Affecting It among Health Professionals in the Public Hospitals, Central Ethiopia. Ethopian Journal of Health Science, 25(3), 231–242.
de Vet, H. C. W., Terwee, C. B., Knol, D. L., & Bouter, L. M. (2006). When to use agreement versus reliability measures. Journal of Clinical Epidemiology, 59(10), 1033–1039.
DeVon, H. a., Block, M. E., Moyle-Wright, P., Ernst, D. M., Hayden, S. J., Lazzara, D. J., … Kostas-Polston, E. (2007). A psychometric toolbox for testing validity and reliability [Electronic Version]. Journal of Nursing Scholarship, 39(2), 155–164.
Dubois, C.-A., D’Amour, D., Pomey, M.-P., Girard, F., & Brault, I. (2013). Conceptualizing performance of nursing care as a prerequisite for better measurement: a systematic and interpretive review. BMC Nursing, 12(1), 7.
Epstein, J., Osborne, R. H., Elsworth, G. R., Beaton, D. E., & Guillemin, F. (2015). Cross-cultural adaptation of the Health Education Impact Questionnaire: Experimental study showed expert committee, not back-translation, added value. Journal of Clinical Epidemiology, 68(4), 360–369.
Ferrer, M., Alonso, J., Prieto, L., Plaza, V., Monsó, E., Marrades, R., … Antó, J. M. (1996). Validity and reliability of the St George’s respiratory questionnaire after adaptation to a different language and culture: The Spanish example. European Respiratory Journal, 9(6), 1160–1166.
Fleiszer, A. R., Semenic, S. E., Ritchie, J. A., Richer, M. C., & Denis, J. L. (2015). An organizational perspective on the long-term sustainability of a nursing best practice guidelines program: A case study. BMC Health Services Research, 15(1), 1–16.
Flinkman, M., Leino-Kilpi, H., Numminen, O., Jeon, Y., Kuokkanen, L., & Meretoja, R. (2017). Nurse Competence Scale: a systematic and psychometric review. Journal of Advanced Nursing, 73(5), 1035–1050.
Gisev, N., Bell, J. S., & Chen, T. F. (2013). Interrater agreement and interrater reliability: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9(3), 330–338.
Guyatt, G., Walter, S., & Norman, G. (1987). Measuring Change Over Time- Aseessing the Usefulness of Evaluative Instruments. Journal of Chronic Diseases, 40(2), 171–178.
Horgas, A. L., Yoon, S. L., Nichols, A. L., & Marsiske, M. (2008). Is the CVI an Acceptable Indicator of Content Validity? Appraisal and Recommendations. Research in Nursing & Health, 31(4), 341–354.
Huicho, L., Dieleman, M., Campbell, J., Codjia, L., Balabanova, D., Dussault, G., & Dolea, C. (2010). Increasing access to health workers in underserved areas: A conceptual framework for measuring results. Bulletin of the World Health Organization, 88(5), 357–363.
Jaakkimainen, L., Glazier, R., Barnsley, J., Salkeld, E., Lu, H., & Tu, K. (2014). Waiting to see the specialist: patient and provider characteristics of wait times from primary to specialty care. BMC Family Practice, 15(1), 16.
Johns, G. (2010). Presenteeism in the workplace: a review and research agenda. Journal of Organizational Behavior, 31, 519–542.
Kalisch, J. B., Lee, H., & Salas, E. (2012). The development and testing of nursing teamwork survey. In F. D. Polit & C. T. Beck (Eds.), Resource Manual for Nursing Research: Generating and Assessing Evidence for Nursing Practice (9th ed., pp. 338–351). Lippincott Williams & Wilkins.
Kanchanachitra, C., Lindelow, M., Johnston, T., Hanvoravongchai, P., Lorenzo, F. M., Huong, N. L., … Dela Rosa, J. F. (2011). Human resources for health in southeast Asia: Shortages, distributional challenges, and international trade in health services. The Lancet, 377(9767), 769–781.
Kassa, H., Murugan, R., Zedwu, F., Hailu, M., & Woldeyohannes, D. (2014). Assessment of knowledge , attitude and practice and associated factors towards palliative care among nurses working in selected hospitals , Addis Ababa , Ethiopia. BMC Palliative Care, 13(4), 1–11.
Kitchenham, B., & Pfleeger, S. L. (2002). Principles of survey research part 3: Constructing a survey instrument. ACM SIGSOFT Software Engineering Notes, 27(2), 20.
Kneafsey, R., Clifford, C., & Greenfield, S. (2013). What is the nursing team involvement in maintaining and promoting the mobility of older adults in hospital? A grounded theory study. International Journal of Nursing Studies, 50(12), 1617–1629.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575.
Leach, L. S., & Mayo, A. M. (2013). Rapid Response Teams : qualitative analysis of their effectiveness. American Journal of Critical Care, 22(3), 198–210.
Lemieux-Charles, L., & McGuire, W. L. (2006). What Do We Know about Health Care Team Effectiveness? A Review of the Literature. Medical Care Research and Review, 63(3), 263–300.
Lutwama, G. (2011). The performance of health workers in decentralised services in Uganda. Retrieved from
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Reserach, 35(6), 382–385.
Mackenzie, S. B., & Podsakoff, P. M. (2012). Commentary on " Common Method Bias : Nature , Causes , and Procedural Remedies ". Journal of Retailing, 88(January), 542–555.
Mackenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and Behavioral Research : Integrating New and Existing Techniques. MIS Quarterly, 35(2), 293–334.
Makai, P., Cramm, J. M., van Grotel, M., & Nieboer, A. P. (2014). Labor productivity, perceived effectiveness, and sustainability of innovative projects. Journal for Healthcare Quality : Official Publication of the National Association for Healthcare Quality, 36(2), 14–24.
Manojlovich, M., & Sidani, S. (2008). Nurse dose: What’s in a concept? Research in Nursing and Health, 31(4), 310–319.
McElroy, C., & Esterhuizen, P. (2017). Compassionate communication in acute healthcare: establishing the face and content validity of a questionnaire. Journal of Research in Nursing, 22(1–2), 72–88.
Mehmet, T. (2013). Organizational variables on nurses’ job performance in Turkey: Nursing Assesments. Iranian Journal of Public Health, 42(3), 261–271.
Meretoja, R., Isoaho, H., & Leino-Kilpi, H. (2004). Nurse Competence Scale: development and psychometric testing. Journal of Advanced Nursing, 47(2), 124–133.
Meretoja, R., & Koponen, L. (2012). A systematic model to compare nurses’ optimal and actual competencies in the clinical setting. Journal of Advanced Nursing, 68(2), 414–422.
Mert, T., & Ekici, D. (2015). Development of an Assessment Model for Evaluating the Performance of Nursing Services. International Journal of Hospital Research, 4(1), 9–14.
Meyer, M. A., & Booker, J. M. (2001). Eliciting and Analysing Expert Judgement A practical Guide. American Statistical Association and the Society for Industrial and Applied Mathematics.
Needleman, J., Kurtzman, E. T., & Kizer, K. W. (2007). Performance measurement of nursing care : State of the science and the current consensus. Medical Care Research and Review, 64(2), 10S–43S.
Netemeyer, R., Bearden, W., & Sharma, S. (2003). Scaling Procedures Issues and Applications. Thousand Oak, London: SAGE Publications.
North, N., & Hughes, F. (2012). A systems perspective on nursing productivity. Journal of Health Organization and Management, 26(2), 192–214.
Numminen, O., Leino-Kilpi, H., Isoaho, H., & Meretoja, R. (2015). Newly Graduated Nurses’ Competence and Individual and Organizational Factors: A Multivariate Analysis. Journal of Nursing Scholarship, 47(5), 446–457.
Numminen, O., Meretoja, R., Isoaho, H., & Leino-Kilpi, H. (2013). Professional competence of practising nurses. Journal of Clinical Nursing, 22(9–10), 1411–1423.
Nunnally, J. C., & Bernstein, I. (1994). Psychometric Theory (3rd Editio). McGraw-Hill, Inc.
Olsson, C., Forsberg, A., & Bjersa, K. (2016). Safety climate and readiness for implementation of evidence and person centered practice - A national study of registered nurses in general surgical care at Swedish university hospitals. BMC Nursing, 15(1), 54.
Pillay, M. S., Johari, R., Hazilah, M., Asaari, A., Azman, B., Salikin, F., … Ismefariana, I. W. (2011). Hospital waiting time : the forgotten premise of healthcare service delivery ? International Journal of Healthcare, 24(7), 506–522.
Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2016). Recommendations for Creating Better Concept Definitions in the Organizational, Behavioral, and Social Sciences. Organizational Research Methods, 19(2), 159–203.
Polit, D. F., & Beck, C. T. (2006). The content Validity Index: Are You Sure You Know What’s Being Reported? Critique and Recommendations. Research in Nursing & Health, 29, 489–497.
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an Acceptable Indicator of Content Validity? Appraisal and Recommendations. Research in Nursing & Health, 30, 459–467.
Rattray, J., & Jones, M. C. (2007). Essential elements of questionnaire design and development. Journal of Clinical Nursing, 16(2), 234–243.
Raykov, T. (2008). “Alpha if Deleted” and Loss in Criterion Validity Appeared in. British Journal of Mathematical and Statistical Psychology, 61(2), 275–285.
Rowe, A. K., De Savigny, D., Lanata, C. F., & Victora, C. G. (2005). How can we achieve and maintain high-quality performance of health workers in low-resource settings? Lancet, 366(9490), 1026–1035.
Sand-Jecklin, K., & Sherman, J. (2014). A quantitative assessment of patient and nurse outcomes of bedside nursing report implementation. Journal of Clinical Nursing, 23(19–20), 2854–2863.
Scott, T., Mannion, R., Davies, H., & Marshall, M. (2003). Methods The Quantitative Measurement of Organizational Culture in Health Care : A Review of the Available Instruments. Health Services Research, 38(3), 923–945.
Sheaff, R., Pickard, S., & Smith, K. (2002). Public service responsiveness to users’ demands and needs: theory, practice and primary healthcare in England. Public Administration, 80(3), 435–452.
Smith, S. (2012). Nurse Competence : A Concept Analysis. International Journal of Nursing Knowledge, 23(3), 172–182.
Sonnentag, S., Volmer, J., & Spychala, A. (2008). Job Performance. The SAGE Handbook of Organizational Behavior, 1, 427–447.
Stolt, M., Charalambous, A., Radwin, L., Adam, C., Katajisto, J., Lemonidou, C., … Suhonen, R. (2016). Measuring trust in nurses ??? Psychometric properties of the Trust in Nurses Scale in four countries. European Journal of Oncology Nursing, 25, 46–54.
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55.
Topf, M. (1986). Three estimates of interrater reliability for nominal data. Methodology Corner, 35(4), 253–255.
Umann, J. ., Guido, L. A. ., & Grazziano, E. S. . (2012). Presenteeism in hospital nurses [Presenteísmo em enfermeiros hospitalares]. Revista Latino-Americana de Enfermagem, 20(1), 159–166.
Vanessa, A., Rodrigues, D., Vituri, D. W., Louren, C., Terezinha, M., Vannuchi, O., & Tiago, W. (2012). Nursing responsiveness the client ’ s view. Rev Esc Enferm USP, 46(6), 1446–1452.
Viswesvaran, C., & Ones, D. S. (2000). Perspectives on Models of Job Performance. International Journal of Selection and Assessment, 8(4), 216–226.
WHO. (2006). Working together For Health, The WHO Health Report 2006. World Health (Vol. 19).
WHO. (2010). Increasing Access To Health Workers In Remote And Rural Areas Through Improved Retention: Global Policy Recommendations. WHO. WHO Press.
Wynd, C. A., Schmidt, B., & Schaefer, M. A. (2003). Two quantitative approaches for estimating content validity. Western Journal of Nursing Research, 25(5), 508–518.
Yakusheva, O., & Weiss, M. (2017). Rankings matter: nurse graduates from higher-ranked institutions have higher productivity. BMC Health Services Research, 17(1), 134.
Zamanzadeh, T and Nemati, N. (2014). Details of content validity and objectifying it in instrument development. Nursing Practice Today, 1(3), 163–171.
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.