Volume 9, Issue 4 (2021)
Health Educ Health Promot 2021, 9(4): 395-402 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Ghasemi M, Ghofranipour F, Shahbazi H, Aminshokravi F. Skills of Health-Workers in Providing a Self-Care Program for Pre-Diabetic Individuals; Psychometrics of a Questionnaire. Health Educ Health Promot 2021; 9 (4) :395-402
URL: http://hehp.modares.ac.ir/article-5-51849-en.html
URL: http://hehp.modares.ac.ir/article-5-51849-en.html
1- Department of Health Education & Health Promotion, Faculty of Medical Sciences, Tarbiat Modares University, Tehran, Iran
2- Department of Health Education & Health Promotion, Faculty of Medical Sciences, Tarbiat Modares University, Tehran, Iran , h.shahbazi@modares.ac.ir
2- Department of Health Education & Health Promotion, Faculty of Medical Sciences, Tarbiat Modares University, Tehran, Iran , h.shahbazi@modares.ac.ir
Keywords: Psychometrics [MeSH], Health-Workers [MeSH], Empowerment [MeSH], Self-Care [MeSH], Prediabetes [MeSH]
Full-Text [PDF 633 kb]
(921 Downloads)
| Abstract (HTML) (1315 Views)
Full-Text: (285 Views)
Introduction
Concerns about non-communicable diseases are among the severe global problems that will cause many issues in these years and the future [1]. Type2 diabetes is one of the four most significant non-communicable diseases along with cardiovascular diseases, cancers, and chronic respiratory diseases, which account for 63% of deaths related to non-communicable diseases in the world [2]. Due to the latest reports from the International Diabetes Federation, there are 425 million diabetics in the world, the number of which will rise to 629 million by 2045, and due to the report, the number of patients in Iran is 5 million, which will increase by nearly 10 million by 2045 [3].
Some predictors of prediabetes, such as age and family history of diabetes, cannot be inhibited. Still, authorities, policymakers, and planners should consider other efficient and preventable factors in prediabetes. Bodyweight in both sexes, waist circumference in women, fasting blood sugar, and two hours after meals and triglyceride and HDL cholesterol concentrations can be controlled by extensive public education programs for healthy nutrition and increased physical activity. Cessation of smoking and increasing literacy levels are other essential factors to decrease prediabetes incidence [4-7].
The global prevalence of prediabetes due to IGT (Impaired Glucose Tolerance) level in 2017 was 7.3% in the adult population (79-20 years), equivalent to 352.1 million people. In general, the highest prevalence was reported in North America and the Caribbean (15.4%), Central and South America (10%), and the lowest prevalence was reported in Southeast Asia (3%) and Europe (5.5%) [8]. The incidence rate of prediabetes in Tehran society, after nine years of follow-up of about 6,000 men and women aged 20 years and over healthy in Tehran, 46 men and 38 women out of every 1,000 people developed prediabetes. It means that each year, 4% of Tehran's total adult population is converted from healthy blood sugar to prediabetes, and this finding is very shocking and alarming [9].
One of the methods to inhibit and control diabetes is self-care behavior. Self-care is defined as a strategy to adapt to life's problems and issues which promote health. Many researchers have indicated that utilizing lifestyle modification and self-care principles can delay or even prevent diabetes in prediabetes. Behavioral changes alone can decrease the risk of diabetes by 40 to 70% [10-16]. For patients and prediabetes to deal with diabetes issues, having awareness, knowledge, skills, abilities, and motivations for self-care behaviors requires healthcare workers and the health team to play a significant role in educating these issues [17, 18].
Today, education is one of the most important ways to control non-communicable diseases, and studies show that the complications of non-communicable diseases can be reduced [18]. Appropriate educational methods and models empower health workers to provide optimal health programs [19]. The results of many studies show that to improve the quality of health care, the educational skills of health workers should be increased; Also, the growth of the vulnerable population, the change in the view of health from the individual to the community, and the change in the overall health goals from treatment to prevention have led to the provision of primary care and disease prevention by health officials. In this regard, the existence of an appropriate and standard tool that can examine the educational skills of health workers and lead to savings in time and cost of conducting research studies is felt [20-24].
Considering first the increasing prevalence of diabetes and the importance of self-care in controlling diabetes progression in prediabetes, and second the efficient and undeniable role of health workers, this study aimed to develop and assess the psychometric properties of health-worker skills evaluation to implement a self-care program for prediabetes.
Instrument and Methods
The current descriptive-analytical cross-sectional study was conducted from January to September 2020 on all health workers in Mazandaran province who were able to evaluate the validity and reliability of health workers' skills in obtaining self-care programs for prediabetes. Four hundred health workers of Mazandaran province were enrolled in the research Utilizing the below formula and considered p=0.5, α=0.05, an error value of 0.05, a confidence interval of 95%, and a 10% fall.
The subjects were chosen by random cluster sampling proportional to the size and entered the research. The sampling method was that each of the city's health centers was chosen as a cluster, and the subjects were randomly chosen from each class due to the number of health worker populations in each city. After determining each city's share, the subjects were chosen to utilize random numbers table due to the list of names in the county's health center. Health workers were eligible to participate if they completed the written consent form to participate in the study and had at least three years of work experience. The exclusion criteria include the reluctance of participants to continue to participate in the study and those who have over 20 years of work experience.
Firstly, valid and specialized resources were surveyed, including databases such as PubMed, ProQuest, etc., papers, and scientific books (more than 50 papers, two treatises, and two books). Since no specific comprehensive and standard questionnaire was found for this subject, the questionnaire items (75 items) were designed by interviewing experts and specialists in diabetes and health education and re-examining the most relevant and comprehensive texts and papers. The researchers examined the extracted expressions to place each item in relevant behavioral and environmental factors (predisposing, enabling, and reinforcing factors) based on the PRECEDE-PROCEED model. The expressions were overlapping concepts that were merged. Additionally, the highly specialized items such as the diabetes treatments and complications had been removed, and the initial questionnaire expressions were reduced to 54 phrases. The questionnaire had three parts:
1. Demographic information;
2. Surveying the awareness and knowledge of health workers that the questions of the awareness section were designed as correct/incorrect, and the questions of the knowledge section were designed as multiple choice;
3. Behavioral and environmental factors (predisposing, enabling, and reinforcing factors) were relevant to health workers' skills in obtaining self-care educational programs for prediabetes patients. The questions of this section were designed in a 5-points Likert scale from 1 (strongly disagree) to 5 (strongly agree).
Psychometric properties of the instrument
The initial questionnaire was evaluated through face validity, content validity, and construct validity (Exploratory factor analysis and confirmatory factor analysis) were applied to identify the questionnaire's validity. On the other hand, the reliability was evaluated through internal consistency and stability of the instrument via test-retests. The details of the implementation of the psychometric properties are as follows:
Face validity: To identify the face validity in the qualitative section, researchers asked 15 health workers to assess each item for "ambiguity", "relevancy", and "difficulty". For the quantitative approach, the same health workers were asked to evaluate the questionnaire and score each item's importance on a 5-point Likert scale to calculate each item's impact score. It was calculated by multiplying the importance of an item with its frequency (Impact Score=Frequency (%)×Importance). The impact score of 1.5 or above was considered satisfactory as recommended [25, 26].
Content validity: Two qualitative and quantitative methods were applied to identify the content validity of the designed questionnaire; for the qualitative method, an expert panel consisting of 5 health education and health promotion experts, three diabetes experts, and two general practitioners. The questionnaire was evaluated for grammar, wording, item allocation, and scaling indices at this step. The content validity ratio (CVR) and content validity index (CVI) was calculated using the quantitative method. To identify the content validity ratio, experts will consider each item's necessity, and CVR values higher than 0.62 were accepted due to the Lawshe table. To identify the content validity index (CVI), the criteria for relevancy, clarity, and simplicity of each item were surveyed, and values higher than 0.79 were accepted.
Construct validity: Exploratory factor analysis and confirmatory factor analysis were applied to identify construct validity, so in this research, due to 43 items of the questionnaire, 200 health workers for exploratory factor analysis and 200 health workers were chosen for confirmatory factor analysis. To investigate the feasibility of exploratory factor analysis (principal component analysis), KMO (Kaiser-Meyer-Olkin) test was utilized for sample size adequacy, and Bartlett's Test of Sphericity was used to assess the appropriateness of the data. The recommended value of KMO for doing factor analysis on data is between zero and 1. KMO values between 0.5 and 0.7 showed moderate factor analysis, 0.7 to 0.8 balanced factor analysis, 0.8 to 0.9 of optimal factor analysis, and greater than 0.9 excellent factor analysis [27-30]. Exploratory factor analysis was performed utilizing SPSS software using 200 questionnaires applying principal components with Varimax rotation. Evaluating the structure of factors derived from the exploratory factor analysis must fit the confirmatory factor analysis model. All sub-branches ' internal reliability and correlation among sub-things were performed to perform confirmatory factor analysis with data from 200 questionnaires to investigate the suggested factor structure. Several indicators must be considered to identify a model's fitness, and first, the chi-square index is investigated. Since the chi-square value is commonly very high and significant in high volume samples, it is not a good indicator for identifying the model's fitness. It is better to examine the χ2/DF index, a ratio of Chi-square to the degree of model freedom. For this index, values 1 to 5 are appropriate for fitting the model. Some of the indicators applied in this research were the comparative fit index (CFI), the goodness of fit index (GFI), the root mean square error of approximation (RMSEA), chi-square index (χ2), and its corresponding degree of freedom. Moreover, the adjusted goodness of fit index (AGFI) was that the acceptable values for RMSEA were less than 0.1, and for CFI, GFI, and AGFI between 0 and 1, the closer to 1, they have more excellent fitness [25, 31-33].
Reliability: Internal consistency was measured utilizing Cronbach's alpha coefficient on a group of 30 health workers with a work experience of at least ten years. Cronbach's alpha was equal to or greater than 0.7 was considered appropriate [34-37]. Stability evaluation was handled through the test-retest method. The significant point in this method is the time interval between the two tests. The researchers have suggested this interval of two weeks to a month. They completed the instrument within two weeks and then compared the scores obtained in two steps applying the Pearson correlation coefficient and intra-cluster correlation index [25].
This article was supported Elite Researcher Grant Committee under award number IR.NIMAD.REC.1398.244 from the National Institutes for Medical Research Development (NIMAD), Tehran, Iran. All participants gave informed written consent. Before the beginning of the study, the consent of the relevant authorities was obtained. At the beginning of the interviews, the purpose of the research, interview method, data confidentiality, and the freedom to enter or quit the study were explained to the participants, and their informed consent was obtained. Due to the prevalence of coronavirus and travel restrictions, the study questionnaire was designed online and sent to selected community health workers through the WhatsApp application. The questionnaires were completed from March to May 2020.
The statistical analyses and confirmatory factor analyses were performed using the SPSS 21 and the LISREL 8.80 for Windows, respectively. Data analysis was conducted through exploratory factor analysis using the principal component method with varimax rotation. The normal distribution of the data was verified using the Kolmogorov-Smirnov test. Cronbach’s coefficient alpha and ICC were also calculated. Confirmative factor analysis was performed for comparing and assessing the model fitness.
Findings
The mean age of participants was 34.13±8.94 years that 37.5% were male and 62.5% were female (Table 1).
Face validity and Content validity: In surveying the face validity in the effect stage of the item with the criterion of 1.5, 8 questions were deleted from the knowledge section. The items that were deleted at this step were mainly related to laboratory results and complications of diabetes, which due to participants, awareness of these items in obtaining self-care programs for prediabetes people was less important, so their impact factor score was less than 1.5 (e.g., 1. Which of the following is not the complications of diabetes?; 2. Which of the following symptoms are ketoacidosis symptoms?; Or 3. What may be caused by low blood glucose levels?). In the next step, in the content validity ratio (CVR), due to the number of experts and the criterion (0.62) of the Lawshe table, three questions were removed from the reinforcing construct because of the lack of necessity in the questionnaire (1. Prediabetics are interested in the training, which makes me encouraged; 2. Material encouragement increases my motivation to perform my duties; and 3. Immaterial encouragement increases my motivation to perform my duties). In contrast, the question was not deleted in the content validity index (CVI) due to the criterion of 0.79. Furthermore, 43 questions out of 54 questions, with suitable content validity and face validity, entered the questionnaire's reliability applying Cronbach's alpha, Pearson correlation coefficient, and intra-cluster correlation index.
Reliability: Cronbach's alpha coefficients for the questionnaire structures were 0.87-0.88. A test-retest was used to evaluate the consistency of the questionnaire. The Pearson correlation was computed among the questionnaire's scores twice in a two-week interval in this method. If this index was above 0.75, stability was desirable (Table 2).
Table 1) Demographic characteristics of participants (n=400)
Table 2) Reliability results of the questionnaire
Construct validity: Construct validity was assessed utilizing exploratory factor analysis, and the principal components method was handled on 25 items. The KMO and Barlett sphericity test results showed the model adequacy (Table 3).
Table 3) Results of factor analysis
The first factor described 36.909% of the common variance (self-efficacy), 10.047% by the second factor (attitude), 6.392% by the third factor (reinforcing), and 4.168% by the fourth factor (enabling) (Table 4).
Some items with different loads were placed in several factors. Although it was tried to place the items in factors with the highest factor load, items no. 5 and 6 of the enabling structure were transferred to the lower factor load because it was not compatible with the factor with the highest factor load. All items were loaded in the extracted agents, and no items were removed (Table 5).
Table 4) Total variance explained
Table 5) Matrix of rotated factors with varimax rotation and factor load rates of items in each factor
Utilizing the Scree plot, and after identifying the internal consistency of the instrument items, four factors with Eigen Value above 1.04, which explained 57.516% of the total variance, were identified (Figure 1).
The whole questionnaire was loaded in the related domains, and the KMO value for the extractive agent was higher than 0.8. Also, the percentage of variance expressed for the extractive agent was 63.93. Besides, Bartlett's test results confirmed the adequacy of exploratory factor analysis for this extractive factor (Table 6).
Figure 1) Scree plot for determining factors of the designed instrument
Table 6) Exploratory Factor Analysis Results for the Whole Instrument
In the next step, confirmatory factor analysis was handled to approve the structure of exploratory factor analysis. Due to the values of indicators presented for each of these sub-structures, the chi2 index was less than 5 degrees free, confirming models’ appropriateness. Also, GFI, AGFI, NFI, RFI, IFI, and CFI indices more significant than 0.9, RMSEA was less than 0.1, which confirmed this model's validity (Table 7).
As a result, this model had achieved a relatively good level of grace, and due to them, the PRECEDE model factors can confirm (Figure 2).
Table 7) Fitting indicators to confirm the suitability of the model studied
Figure 2) Results from confirmatory factor analysis
Discussion
Due to the lack of access to comprehensive and suitable scientific tools in evaluating Health workers' skills in obtaining a self-care program for prediabetes, this study was conducted to design and psychometrically evaluate a suitable questionnaire.
Evaluation of internal reliability (Cronbach's alpha) between 0.87 and 0.88 for this instrument's structures shows the acceptability of these scales. In other words, these amounts showed that each structure measures the same subject [36]. The prepared questionnaire had good stability due to the results obtained from the Pearson correlation coefficient and the intra-cluster correlation coefficient. This result indicates that this questionnaire can provide reliable results in different temporal and spatial conditions [38]; it should be noted that this study's findings are consistent with the results of some similar research [39-41].
Another significant feature of the questionnaire is validity, which aims to identify a tool's ability to measure what is designed to measure it. The most significant step in identifying the questionnaire's validity is determining the construct's validity, which factor analysis is the best method for this part [42]. Factor analysis identifies clusters of related questions of the instrument or scale used. The value of this index between 0.8-0.9 shows optimal factor analysis and more than 0.9 excellent factor analysis. The results of the KMO (0.88) and Bartlett test (p>0.001) confirm the factor analysis model and indicate its suitability [33]. The results show that the questionnaire is multifactorial (three factors). Since the minimum factor load required to include a phrase in a factor depends on the number of instrument expressions and the intended specific value, the cut-out point of 0.3 was considered the minimum factor load required for each expression to maintain it. Considering that the factor load of item 2 of the reinforcing construct was less than 0.3, this item can be removed from the questionnaire.
Due to the confirmatory factor analysis results, it was found that the designed questionnaire has good fitness in Iranian society. Because the fitness statistics had reached an acceptable threshold, similar to the findings of this study, it can be seen in the Bukhsh study that the designed questionnaire had a good quality [43]. Thus, the differential validity of defined structures is confirmed due to the obtained results.
Many similar research pieces that have designed and standardized the diabetes questionnaire have mainly examined patients' behavior [44-47], and a model-based tool that can examine health workers' skills was not found, so similar researches are handled to compare and better survey this tool.
One positive point of this study was developing a questionnaire about improving pre-diabetes individuals’ self-care program, which seems designed for the first time in Iran. The participants of this study were health workers working at rural health houses. Even though they have a crucial responsibility to peoples' health, researchers could not find sufficient research about this target group in Iran.
Nevertheless, this study also had some limitations. Firstly, it was more appropriate that this research would be held by both qualitative and quantitative methods; due to the limitations caused by the spread of COVID-19, researchers were not able to interview participants, as well as the questionnaire was completed online. It means that the researchers were not aware of the conditions of questionnaire completion, and it could impact the answers provided. Additionally, considering that this study has been done on health workers, it is impossible to
generalize to other care groups such as nurses and physicians. It is suggested that similar studies on other health caregivers be done.
Conclusion
The designed questionnaire has good validity and reliability.
Acknowledgments: The authors consider it necessary to appreciate and thank Tarbiat Modares University and the experts who helped us in various stages with their valuable opinions, community health workers, and Mazandaran University of Medical Sciences and Health Services, who had the necessary cooperation with researchers to carry out this project.
Ethical Permissions: Ethical approval for this study was obtained from the National Institute for Medical Research Development (NIMAD) of the Islamic Republic of Iran (IR.NIMAD.REC.1398.244).
Conflicts of Interests: This article was supported Elite Researcher Grant Committee under award number IR.NIMAD.REC.1398.244 from the National Institutes for Medical Research Development (NIMAD), Tehran, Iran.
Authors’ Contribution: Ghasemi M. (First author), Introduction writer/Original researcher/Statistical analyst/Discussion writer (25%); Ghofranipour F. (Second author), Methodologist/Original researcher/Statistical analyst/Discussion writer (25%); Shahbazi H. (Third author), Methodologist/Original researcher /Statistical analyst/Discussion writer (25%); Aminshokravi F. (Fourth author) Methodologist/Assistant researcher/Discussion writer (25%).
Funding/Support: No external funding sources were provided for this study.
Concerns about non-communicable diseases are among the severe global problems that will cause many issues in these years and the future [1]. Type2 diabetes is one of the four most significant non-communicable diseases along with cardiovascular diseases, cancers, and chronic respiratory diseases, which account for 63% of deaths related to non-communicable diseases in the world [2]. Due to the latest reports from the International Diabetes Federation, there are 425 million diabetics in the world, the number of which will rise to 629 million by 2045, and due to the report, the number of patients in Iran is 5 million, which will increase by nearly 10 million by 2045 [3].
Some predictors of prediabetes, such as age and family history of diabetes, cannot be inhibited. Still, authorities, policymakers, and planners should consider other efficient and preventable factors in prediabetes. Bodyweight in both sexes, waist circumference in women, fasting blood sugar, and two hours after meals and triglyceride and HDL cholesterol concentrations can be controlled by extensive public education programs for healthy nutrition and increased physical activity. Cessation of smoking and increasing literacy levels are other essential factors to decrease prediabetes incidence [4-7].
The global prevalence of prediabetes due to IGT (Impaired Glucose Tolerance) level in 2017 was 7.3% in the adult population (79-20 years), equivalent to 352.1 million people. In general, the highest prevalence was reported in North America and the Caribbean (15.4%), Central and South America (10%), and the lowest prevalence was reported in Southeast Asia (3%) and Europe (5.5%) [8]. The incidence rate of prediabetes in Tehran society, after nine years of follow-up of about 6,000 men and women aged 20 years and over healthy in Tehran, 46 men and 38 women out of every 1,000 people developed prediabetes. It means that each year, 4% of Tehran's total adult population is converted from healthy blood sugar to prediabetes, and this finding is very shocking and alarming [9].
One of the methods to inhibit and control diabetes is self-care behavior. Self-care is defined as a strategy to adapt to life's problems and issues which promote health. Many researchers have indicated that utilizing lifestyle modification and self-care principles can delay or even prevent diabetes in prediabetes. Behavioral changes alone can decrease the risk of diabetes by 40 to 70% [10-16]. For patients and prediabetes to deal with diabetes issues, having awareness, knowledge, skills, abilities, and motivations for self-care behaviors requires healthcare workers and the health team to play a significant role in educating these issues [17, 18].
Today, education is one of the most important ways to control non-communicable diseases, and studies show that the complications of non-communicable diseases can be reduced [18]. Appropriate educational methods and models empower health workers to provide optimal health programs [19]. The results of many studies show that to improve the quality of health care, the educational skills of health workers should be increased; Also, the growth of the vulnerable population, the change in the view of health from the individual to the community, and the change in the overall health goals from treatment to prevention have led to the provision of primary care and disease prevention by health officials. In this regard, the existence of an appropriate and standard tool that can examine the educational skills of health workers and lead to savings in time and cost of conducting research studies is felt [20-24].
Considering first the increasing prevalence of diabetes and the importance of self-care in controlling diabetes progression in prediabetes, and second the efficient and undeniable role of health workers, this study aimed to develop and assess the psychometric properties of health-worker skills evaluation to implement a self-care program for prediabetes.
Instrument and Methods
The current descriptive-analytical cross-sectional study was conducted from January to September 2020 on all health workers in Mazandaran province who were able to evaluate the validity and reliability of health workers' skills in obtaining self-care programs for prediabetes. Four hundred health workers of Mazandaran province were enrolled in the research Utilizing the below formula and considered p=0.5, α=0.05, an error value of 0.05, a confidence interval of 95%, and a 10% fall.
The subjects were chosen by random cluster sampling proportional to the size and entered the research. The sampling method was that each of the city's health centers was chosen as a cluster, and the subjects were randomly chosen from each class due to the number of health worker populations in each city. After determining each city's share, the subjects were chosen to utilize random numbers table due to the list of names in the county's health center. Health workers were eligible to participate if they completed the written consent form to participate in the study and had at least three years of work experience. The exclusion criteria include the reluctance of participants to continue to participate in the study and those who have over 20 years of work experience.
Firstly, valid and specialized resources were surveyed, including databases such as PubMed, ProQuest, etc., papers, and scientific books (more than 50 papers, two treatises, and two books). Since no specific comprehensive and standard questionnaire was found for this subject, the questionnaire items (75 items) were designed by interviewing experts and specialists in diabetes and health education and re-examining the most relevant and comprehensive texts and papers. The researchers examined the extracted expressions to place each item in relevant behavioral and environmental factors (predisposing, enabling, and reinforcing factors) based on the PRECEDE-PROCEED model. The expressions were overlapping concepts that were merged. Additionally, the highly specialized items such as the diabetes treatments and complications had been removed, and the initial questionnaire expressions were reduced to 54 phrases. The questionnaire had three parts:
1. Demographic information;
2. Surveying the awareness and knowledge of health workers that the questions of the awareness section were designed as correct/incorrect, and the questions of the knowledge section were designed as multiple choice;
3. Behavioral and environmental factors (predisposing, enabling, and reinforcing factors) were relevant to health workers' skills in obtaining self-care educational programs for prediabetes patients. The questions of this section were designed in a 5-points Likert scale from 1 (strongly disagree) to 5 (strongly agree).
Psychometric properties of the instrument
The initial questionnaire was evaluated through face validity, content validity, and construct validity (Exploratory factor analysis and confirmatory factor analysis) were applied to identify the questionnaire's validity. On the other hand, the reliability was evaluated through internal consistency and stability of the instrument via test-retests. The details of the implementation of the psychometric properties are as follows:
Face validity: To identify the face validity in the qualitative section, researchers asked 15 health workers to assess each item for "ambiguity", "relevancy", and "difficulty". For the quantitative approach, the same health workers were asked to evaluate the questionnaire and score each item's importance on a 5-point Likert scale to calculate each item's impact score. It was calculated by multiplying the importance of an item with its frequency (Impact Score=Frequency (%)×Importance). The impact score of 1.5 or above was considered satisfactory as recommended [25, 26].
Content validity: Two qualitative and quantitative methods were applied to identify the content validity of the designed questionnaire; for the qualitative method, an expert panel consisting of 5 health education and health promotion experts, three diabetes experts, and two general practitioners. The questionnaire was evaluated for grammar, wording, item allocation, and scaling indices at this step. The content validity ratio (CVR) and content validity index (CVI) was calculated using the quantitative method. To identify the content validity ratio, experts will consider each item's necessity, and CVR values higher than 0.62 were accepted due to the Lawshe table. To identify the content validity index (CVI), the criteria for relevancy, clarity, and simplicity of each item were surveyed, and values higher than 0.79 were accepted.
Construct validity: Exploratory factor analysis and confirmatory factor analysis were applied to identify construct validity, so in this research, due to 43 items of the questionnaire, 200 health workers for exploratory factor analysis and 200 health workers were chosen for confirmatory factor analysis. To investigate the feasibility of exploratory factor analysis (principal component analysis), KMO (Kaiser-Meyer-Olkin) test was utilized for sample size adequacy, and Bartlett's Test of Sphericity was used to assess the appropriateness of the data. The recommended value of KMO for doing factor analysis on data is between zero and 1. KMO values between 0.5 and 0.7 showed moderate factor analysis, 0.7 to 0.8 balanced factor analysis, 0.8 to 0.9 of optimal factor analysis, and greater than 0.9 excellent factor analysis [27-30]. Exploratory factor analysis was performed utilizing SPSS software using 200 questionnaires applying principal components with Varimax rotation. Evaluating the structure of factors derived from the exploratory factor analysis must fit the confirmatory factor analysis model. All sub-branches ' internal reliability and correlation among sub-things were performed to perform confirmatory factor analysis with data from 200 questionnaires to investigate the suggested factor structure. Several indicators must be considered to identify a model's fitness, and first, the chi-square index is investigated. Since the chi-square value is commonly very high and significant in high volume samples, it is not a good indicator for identifying the model's fitness. It is better to examine the χ2/DF index, a ratio of Chi-square to the degree of model freedom. For this index, values 1 to 5 are appropriate for fitting the model. Some of the indicators applied in this research were the comparative fit index (CFI), the goodness of fit index (GFI), the root mean square error of approximation (RMSEA), chi-square index (χ2), and its corresponding degree of freedom. Moreover, the adjusted goodness of fit index (AGFI) was that the acceptable values for RMSEA were less than 0.1, and for CFI, GFI, and AGFI between 0 and 1, the closer to 1, they have more excellent fitness [25, 31-33].
Reliability: Internal consistency was measured utilizing Cronbach's alpha coefficient on a group of 30 health workers with a work experience of at least ten years. Cronbach's alpha was equal to or greater than 0.7 was considered appropriate [34-37]. Stability evaluation was handled through the test-retest method. The significant point in this method is the time interval between the two tests. The researchers have suggested this interval of two weeks to a month. They completed the instrument within two weeks and then compared the scores obtained in two steps applying the Pearson correlation coefficient and intra-cluster correlation index [25].
This article was supported Elite Researcher Grant Committee under award number IR.NIMAD.REC.1398.244 from the National Institutes for Medical Research Development (NIMAD), Tehran, Iran. All participants gave informed written consent. Before the beginning of the study, the consent of the relevant authorities was obtained. At the beginning of the interviews, the purpose of the research, interview method, data confidentiality, and the freedom to enter or quit the study were explained to the participants, and their informed consent was obtained. Due to the prevalence of coronavirus and travel restrictions, the study questionnaire was designed online and sent to selected community health workers through the WhatsApp application. The questionnaires were completed from March to May 2020.
The statistical analyses and confirmatory factor analyses were performed using the SPSS 21 and the LISREL 8.80 for Windows, respectively. Data analysis was conducted through exploratory factor analysis using the principal component method with varimax rotation. The normal distribution of the data was verified using the Kolmogorov-Smirnov test. Cronbach’s coefficient alpha and ICC were also calculated. Confirmative factor analysis was performed for comparing and assessing the model fitness.
Findings
The mean age of participants was 34.13±8.94 years that 37.5% were male and 62.5% were female (Table 1).
Face validity and Content validity: In surveying the face validity in the effect stage of the item with the criterion of 1.5, 8 questions were deleted from the knowledge section. The items that were deleted at this step were mainly related to laboratory results and complications of diabetes, which due to participants, awareness of these items in obtaining self-care programs for prediabetes people was less important, so their impact factor score was less than 1.5 (e.g., 1. Which of the following is not the complications of diabetes?; 2. Which of the following symptoms are ketoacidosis symptoms?; Or 3. What may be caused by low blood glucose levels?). In the next step, in the content validity ratio (CVR), due to the number of experts and the criterion (0.62) of the Lawshe table, three questions were removed from the reinforcing construct because of the lack of necessity in the questionnaire (1. Prediabetics are interested in the training, which makes me encouraged; 2. Material encouragement increases my motivation to perform my duties; and 3. Immaterial encouragement increases my motivation to perform my duties). In contrast, the question was not deleted in the content validity index (CVI) due to the criterion of 0.79. Furthermore, 43 questions out of 54 questions, with suitable content validity and face validity, entered the questionnaire's reliability applying Cronbach's alpha, Pearson correlation coefficient, and intra-cluster correlation index.
Reliability: Cronbach's alpha coefficients for the questionnaire structures were 0.87-0.88. A test-retest was used to evaluate the consistency of the questionnaire. The Pearson correlation was computed among the questionnaire's scores twice in a two-week interval in this method. If this index was above 0.75, stability was desirable (Table 2).
Table 1) Demographic characteristics of participants (n=400)
Table 2) Reliability results of the questionnaire
Construct validity: Construct validity was assessed utilizing exploratory factor analysis, and the principal components method was handled on 25 items. The KMO and Barlett sphericity test results showed the model adequacy (Table 3).
Table 3) Results of factor analysis
The first factor described 36.909% of the common variance (self-efficacy), 10.047% by the second factor (attitude), 6.392% by the third factor (reinforcing), and 4.168% by the fourth factor (enabling) (Table 4).
Some items with different loads were placed in several factors. Although it was tried to place the items in factors with the highest factor load, items no. 5 and 6 of the enabling structure were transferred to the lower factor load because it was not compatible with the factor with the highest factor load. All items were loaded in the extracted agents, and no items were removed (Table 5).
Table 4) Total variance explained
Table 5) Matrix of rotated factors with varimax rotation and factor load rates of items in each factor
Utilizing the Scree plot, and after identifying the internal consistency of the instrument items, four factors with Eigen Value above 1.04, which explained 57.516% of the total variance, were identified (Figure 1).
The whole questionnaire was loaded in the related domains, and the KMO value for the extractive agent was higher than 0.8. Also, the percentage of variance expressed for the extractive agent was 63.93. Besides, Bartlett's test results confirmed the adequacy of exploratory factor analysis for this extractive factor (Table 6).
Figure 1) Scree plot for determining factors of the designed instrument
Table 6) Exploratory Factor Analysis Results for the Whole Instrument
In the next step, confirmatory factor analysis was handled to approve the structure of exploratory factor analysis. Due to the values of indicators presented for each of these sub-structures, the chi2 index was less than 5 degrees free, confirming models’ appropriateness. Also, GFI, AGFI, NFI, RFI, IFI, and CFI indices more significant than 0.9, RMSEA was less than 0.1, which confirmed this model's validity (Table 7).
As a result, this model had achieved a relatively good level of grace, and due to them, the PRECEDE model factors can confirm (Figure 2).
Table 7) Fitting indicators to confirm the suitability of the model studied
Figure 2) Results from confirmatory factor analysis
Discussion
Due to the lack of access to comprehensive and suitable scientific tools in evaluating Health workers' skills in obtaining a self-care program for prediabetes, this study was conducted to design and psychometrically evaluate a suitable questionnaire.
Evaluation of internal reliability (Cronbach's alpha) between 0.87 and 0.88 for this instrument's structures shows the acceptability of these scales. In other words, these amounts showed that each structure measures the same subject [36]. The prepared questionnaire had good stability due to the results obtained from the Pearson correlation coefficient and the intra-cluster correlation coefficient. This result indicates that this questionnaire can provide reliable results in different temporal and spatial conditions [38]; it should be noted that this study's findings are consistent with the results of some similar research [39-41].
Another significant feature of the questionnaire is validity, which aims to identify a tool's ability to measure what is designed to measure it. The most significant step in identifying the questionnaire's validity is determining the construct's validity, which factor analysis is the best method for this part [42]. Factor analysis identifies clusters of related questions of the instrument or scale used. The value of this index between 0.8-0.9 shows optimal factor analysis and more than 0.9 excellent factor analysis. The results of the KMO (0.88) and Bartlett test (p>0.001) confirm the factor analysis model and indicate its suitability [33]. The results show that the questionnaire is multifactorial (three factors). Since the minimum factor load required to include a phrase in a factor depends on the number of instrument expressions and the intended specific value, the cut-out point of 0.3 was considered the minimum factor load required for each expression to maintain it. Considering that the factor load of item 2 of the reinforcing construct was less than 0.3, this item can be removed from the questionnaire.
Due to the confirmatory factor analysis results, it was found that the designed questionnaire has good fitness in Iranian society. Because the fitness statistics had reached an acceptable threshold, similar to the findings of this study, it can be seen in the Bukhsh study that the designed questionnaire had a good quality [43]. Thus, the differential validity of defined structures is confirmed due to the obtained results.
Many similar research pieces that have designed and standardized the diabetes questionnaire have mainly examined patients' behavior [44-47], and a model-based tool that can examine health workers' skills was not found, so similar researches are handled to compare and better survey this tool.
One positive point of this study was developing a questionnaire about improving pre-diabetes individuals’ self-care program, which seems designed for the first time in Iran. The participants of this study were health workers working at rural health houses. Even though they have a crucial responsibility to peoples' health, researchers could not find sufficient research about this target group in Iran.
Nevertheless, this study also had some limitations. Firstly, it was more appropriate that this research would be held by both qualitative and quantitative methods; due to the limitations caused by the spread of COVID-19, researchers were not able to interview participants, as well as the questionnaire was completed online. It means that the researchers were not aware of the conditions of questionnaire completion, and it could impact the answers provided. Additionally, considering that this study has been done on health workers, it is impossible to
generalize to other care groups such as nurses and physicians. It is suggested that similar studies on other health caregivers be done.
Conclusion
The designed questionnaire has good validity and reliability.
Acknowledgments: The authors consider it necessary to appreciate and thank Tarbiat Modares University and the experts who helped us in various stages with their valuable opinions, community health workers, and Mazandaran University of Medical Sciences and Health Services, who had the necessary cooperation with researchers to carry out this project.
Ethical Permissions: Ethical approval for this study was obtained from the National Institute for Medical Research Development (NIMAD) of the Islamic Republic of Iran (IR.NIMAD.REC.1398.244).
Conflicts of Interests: This article was supported Elite Researcher Grant Committee under award number IR.NIMAD.REC.1398.244 from the National Institutes for Medical Research Development (NIMAD), Tehran, Iran.
Authors’ Contribution: Ghasemi M. (First author), Introduction writer/Original researcher/Statistical analyst/Discussion writer (25%); Ghofranipour F. (Second author), Methodologist/Original researcher/Statistical analyst/Discussion writer (25%); Shahbazi H. (Third author), Methodologist/Original researcher /Statistical analyst/Discussion writer (25%); Aminshokravi F. (Fourth author) Methodologist/Assistant researcher/Discussion writer (25%).
Funding/Support: No external funding sources were provided for this study.
Article Type: Descriptive & Survey |
Subject:
Health Education and Health Behavior
Received: 2021/04/21 | Accepted: 2021/05/3 | Published: 2021/11/12
Received: 2021/04/21 | Accepted: 2021/05/3 | Published: 2021/11/12
References
1. Williams J, Allen L, Wickramasinghe K, Mikkelsen B, Roberts N, Townsend N. A systematic review of associations between non-communicable diseases and socioeconomic status within low-and lower-middle-income countries. J Glob Health. 2018;8(2):20409. [Link] [DOI:10.7189/jogh.08.020409] [PMID] [PMCID]
2. Frouzanfar MH, Alexander L, Anderson HR, Bachman VF, Biryukov S, Brauer M, et al. Global, regional, and national comparative risk assessment of 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks in 188 countries, 1990-2013: A systematic analysis for the Global Burden of Disease Study 2013. Lancet. 2015;386(10010):2287-323. [Link] [DOI:10.1016/S0140-6736(15)00128-2]
3. International diabetes federation. IDF diabetes atlas. Brussels: International Diabetes Federation; 2009. [Link]
4. Le Boudec J, Marques-Vidal P, Cornuz J, Clair C. Smoking cessation and the incidence of pre-diabetes and type 2 diabetes: A cohort study. J Diabetes Complications. 2016;30(1):43-8. [Link] [DOI:10.1016/j.jdiacomp.2015.10.005] [PMID]
5. Qin L, Xu H. A cross-sectional study of the effect of health literacy on diabetes prevention and control among elderly individuals with prediabetes in rural China. BMJ Open. 2016;6(5):011077. [Link] [DOI:10.1136/bmjopen-2016-011077] [PMID] [PMCID]
6. Beulens J, Rutters F, Ryden L, Schnell O, Mellbin L, Hart HE, et al. Risk and management of pre-diabetes. Eur J Prev Cardiol. 2019;26(Suppl 2):47-54. [Link] [DOI:10.1177/2047487319880041] [PMID]
7. Swindell N, Mackintosh K, McNarry M, Stephens JW, Sluik D, Fogelholm M, et al. Objectively measured physical activity and sedentary time are associated with cardiometabolic risk factors in adults with prediabetes: The preview study. Diabetes Care. 2018;41(3):265-9. [Link] [DOI:10.2337/dc17-1057] [PMID]
8. Hostalek U. Global epidemiology of prediabetes-present and future perspectives. Clin Diabetes Endocrinol. 2019;5:5. [Link] [DOI:10.1186/s40842-019-0080-0] [PMID] [PMCID]
9. Hadaegh F, Derakhshan A, Zafari N, Khalili D, Mirbolouk M, Saadat N, et al. Pre‐diabetes tsunami: incidence rates and risk factors of pre‐diabetes and its different phenotypes over 9 years of follow‐up. Diabet Med. 2017;34(1):69-78. [Link] [DOI:10.1111/dme.13034] [PMID]
10. Wang J, Zhang RY, Chen RP, Sun J, Yang R, Ke XY, et al. Prevalence and risk factors for diabetic retinopathy in a high-risk Chinese population. BMC Public Health. 2013;13(1):633. [Link] [DOI:10.1186/1471-2458-13-633] [PMID] [PMCID]
11. Ibrahim N, Moy FM, Awalludin IAN, Ali Z, Ismail IS. The health-related quality of life among pre-diabetics and its association with body mass index and physical activity in a semi-urban community in Malaysia-a cross sectional study. BMC Public Health. 2014;14:298. [Link] [DOI:10.1186/1471-2458-14-298] [PMID] [PMCID]
12. Hansen S, Huttunen-Lenz M, Sluik D, Brand-Miller J, Drummen M, Fogelholm M, et al. Demographic and social-cognitive factors associated with weight loss in overweight, pre-diabetic participants of the preview study. Int J Behav Med. 2018;25(6):682-92. [Link] [DOI:10.1007/s12529-018-9744-x] [PMID] [PMCID]
13. Feldman AL, Griffin SJ, Ahern AL, Long GH, Weinehall L, Fharm E, et al. Impact of weight maintenance and loss on diabetes risk and burden: A population-based study in 33,184 participants. BMC Public Health. 2017;17(1):170. [Link] [DOI:10.1186/s12889-017-4081-6] [PMID] [PMCID]
14. Muraki I, Imamura F, Manson JE, Hu FB, Willett WC, Van Dam RM, et al. Fruit consumption and risk of type 2 diabetes: Results from three prospective longitudinal cohort studies. BMJ. 2013;347:5001. [Link] [DOI:10.1136/bmj.f5001] [PMID] [PMCID]
15. Xi B, Li S, Liu Z, Tian H, Yin X, Huai P, et al. Intake of fruit juice and incidence of type 2 diabetes: A systematic review and meta-analysis. Plos One. 2014;9(3):93471. [Link] [DOI:10.1371/journal.pone.0093471] [PMID] [PMCID]
16. Tabak AG, Herder C, Rathmann W, Brunner EJ, Kivimaki M. Prediabetes: A high-risk state for diabetes development. Lancet. 2012;379(9833):2279-90. [Link] [DOI:10.1016/S0140-6736(12)60283-9]
17. American association of diabetes educators. Competencies for diabetes educators and diabetes paraprofessionals [Report]. Chicago: American Association of Diabetes Educators; 2016, Report NO: Unknown. [Link]
18. Somerville M, Ball L, Sierra-Silvestre E, Williams LT. Understanding the knowledge, attitudes and practices of providing and receiving nutrition care for prediabetes: An integrative review. Aust J Prim Health. 2019;25(4):289-302. [Link] [DOI:10.1071/PY19082] [PMID]
19. Mansoorian MR, Khosrovan S, Sajjadi M, Soltani M, Khazaei A. Effect of team member teaching design on knowledge of health personnel about diabetes. J Diabetes Nurs. 2018;6(2):439-51. [Persian] [Link]
20. Okusanya BO, Ogunjimi OH, Osanyin G, Okojie OE, Oye-Adeniran BA. Effect of training on the knowledge and use of the partograph for low risk pregnancies among health workers in a tertiary hospital in Lagos State, Nigeria. Journal of Community Medicine and Primary Health Care. 2018;30(2):47-54. [Link]
21. Jamshidi HR, Cook DA. Some thoughts on medical education in the twenty-first century. Med Teach. 2003;25(3):229-38. [Link] [DOI:10.1080/0142159031000100256] [PMID]
22. Silverman J, Krieger J, Sayre G, Nelson K. The value of community health workers in diabetes management in low-income populations: A qualitative study. J Community Health. 2018;43(5):842-7. [Link] [DOI:10.1007/s10900-018-0491-3] [PMID]
23. Hill J, Peer N, Oldenburg B, Kengne AP. Roles, responsibilities and characteristics of lay community health workers involved in diabetes prevention programmes: A systematic review. Plos One. 2017;12(12):0189069. [Link] [DOI:10.1371/journal.pone.0189069] [PMID] [PMCID]
24. Javanparast S, Windle A, Freeman T, Baum F. Community health worker programs to improve healthcare access and equity: Are they only relevant to low-and middle-income countries. Int J Health Policy Manag. 2018;7(10):943-54. [Link] [DOI:10.15171/ijhpm.2018.53] [PMID] [PMCID]
25. Nikniaz Z, Jafarabadi MA, Ghaffarifar S, Saeedi Z, Namvar ZA, Shirmohammadi M. Psychometric properties of the Persian version of the celiac disease adherence test questionnaire. BMC Gastroenterol. 2020;20(1):1-6. [Link] [DOI:10.1186/s12876-020-01396-8] [PMID] [PMCID]
26. Nahidi F, Tavafian SS, Heidarzadeh M, Hajizadeh E, Montazeri A. The Mother-Newborn Skin-to-Skin Contact Questionnaire (MSSCQ): Development and psychometric evaluation among Iranian midwives. BMC Pregnancy Childbirth. 2014;14(1):85. [Link] [DOI:10.1186/1471-2393-14-85] [PMID] [PMCID]
27. Lawshe CH. A quantitative approach to content validity. Pers Psychol. 1975;28(4):563-75. [Link] [DOI:10.1111/j.1744-6570.1975.tb01393.x]
28. Tavousi M, Haeri-Mehrizi A, Rakhshani F, Rafiefar S, Soleymanian A, Sarbandi F, et al. Development and validation of a short and easy-to-use instrument for measuring health literacy: The Health Literacy Instrument for Adults (HELIA). BMC Public Health. 2020;20(1):656. [Link] [DOI:10.1186/s12889-020-08787-2] [PMID] [PMCID]
29. Kharazi S, Peyman N, Esmaily H. An evaluation of the validity and reliability of the maternal health literacy and pregnancy outcome questionnaire. J Health Syst Res. 2017;12(4):512-9. [Persian] [Link]
30. Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychol Bull. 1955;52(4):281-302. [Link] [DOI:10.1037/h0040957] [PMID]
31. Kline RB. Principles and practice of structural equation modeling. New York: Guilford Press; 2011. [Link]
32. Ainur A, Sayang M, Jannoo Z, Yap B, Deni SM. Sample size and non-normality effects on goodness of fit measures in structural equation models. Pertanika J Sci Technol. 2015;25(2):575-86. [Link]
33. Orcan F. Exploratory and confirmatory factor analysis: Which one to use first. J Meas Eval Educ Psychol. 2018;9(4):414-21. [Link] [DOI:10.21031/epod.394323]
34. Sadat Z, Ghofranipour F, Azin SA, Montazeri A, Goshtasebi A, Bagheri A, et al. Development and psychometric evaluation of the sexual knowledge and attitudes scale for premarital couples (SKAS-PC): An exploratory mixed method study. Int J Reprod Biomed. 2018;16(1):41-50. [Link]
35. Okado I, Cassel K, Pagano I, Holcombe RF. Development and psychometric evaluation of a questionnaire to measure cancer patients' perception of care coordination. BMC Health Serv Res. 2020;20(1):52. [Link] [DOI:10.1186/s12913-020-4905-4] [PMID] [PMCID]
36. Taber KS. The use of Cronbach's alpha when developing and reporting research instruments in science education. Res Sci Educ. 2018;48(6):1273-96. [Link] [DOI:10.1007/s11165-016-9602-2]
37. Xu C, Yin M, Mo W. Correlation and differences in lumbopelvic sagittal alignment parameters between lumbar radiographs and magnetic resonance images. Glob Spine J. 2020 Aug:2192568220947049. [Link] [DOI:10.1177/2192568220947049] [PMID]
38. Bobak CA, Barr PJ, O'Malley AJ. Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales. BMC Med Res Methodol. 2018;18(1):93. [Link] [DOI:10.1186/s12874-018-0550-6] [PMID] [PMCID]
39. Bazvand Z, Tavafian SS, Boozari S, Shahrbanian S. Development and psychometric of a questionnaire about knowledge, attitude and practice of nurses towards neck pain prevention. J Nurs Educ. 2020;9(1):1-10. [Persian] [Link]
40. Kazemi SS, Tavafian SS, Hidarnia A, Montazeri A. Development and validation of the occupational back pain prevention behaviors questionnaire in nursing personnel. Res Sq. 2019:1-16. [Link] [DOI:10.21203/rs.2.12282/v2]
41. Meng R, Li J, Wang Z, Zhang D, Liu B, Luo Y, et al. The Chinese version of the perceived stress questionnaire: Development and validation amongst medical students and workers. Health Qual Life Outcomes. 2020;18(1):70. [Link] [DOI:10.1186/s12955-020-01307-1] [PMID] [PMCID]
42. Ebadi A, Taghizadeh Z, Montazeri A, Shahvari Z, Tavousi M, Bagherzadeh R. Translation, development and psychometric properties of health related measures-Part 2: Construct validity, reliability and responsiveness. PAYESH. 2017;16(4):445-55. [Persian] [Link]
43. Bukhsh A, Lee SWH, Pusparajah P, Schmitt A, Khan TM. Psychometric properties of the diabetes self-management questionnaire (DSMQ) in Urdu. Health Qual Life Outcomes. 2017;15(1):200. [Link] [DOI:10.1186/s12955-017-0776-8] [PMID] [PMCID]
44. Robat Sarpooshi D, Taghipour A, Mahdizadeh M, Azadeh S, AliReza J, Peyman N. Design and psychometric evaluation of self-care behavior assessment tool in diabetic patients. J Diabetes Metab Disord. 2020;19(2):709-16. [Link] [DOI:10.1007/s40200-020-00549-6] [PMID] [PMCID]
45. Stuhmann LM, Paprott R, Heidemann C, Ziese T, Hansen S, Zahn D, et al. Psychometric properties of a nationwide survey for adults with and without diabetes: The disease knowledge and information needs-diabetes mellitus (2017) survey. BMC Public Health. 2020;20(1):192. [Link] [DOI:10.1186/s12889-020-8296-6] [PMID] [PMCID]
46. Lee EH, Lee YW, Lee KW, Nam M, Kim SH. A new comprehensive diabetes health literacy scale: Development and psychometric evaluation. Int J Nurs Stud. 2018;88:1-8. [Link] [DOI:10.1016/j.ijnurstu.2018.08.002] [PMID]
47. Khodaveisi M, Peyghani Asl S, Purabdollah M, Tapak L, Cheragi F, Amini R. An investigation on the effect of health belief model-based education on preventive behaviors of diabetes type 2 in male adolescences. Health Educ Health Promot. 2020;8(4):209-16. [Persian] [Link]
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |