Reliability and Validity of the Turkish Version of the Acceptance and Action Questionnaire-Substance Abuse (AAQ-SA) on a Clinical Sample

 

Hilal Uygur, Kaasim Fatih Yavuz, Ibrahim Eren, Omer Faruk Uygur, Mahmut Selcuk, Nalan Varsak, Seda Yildirim Ozbek, Basak Demirel

Abstract

Background: In this study, it is aimed to examine validity and reliability of the Turkish version of the Acceptance and Action Questionnaire-Substance Abuse (AAQ-SA) which is developed for assessing psychological flexibility levels of individuals with alcohol and/or substance misuse.

Methods: The research sample consisted of a total of 191 participants diagnosed with alcohol and substance use disorder. For reliability analysis, Cronbach alpha coefficient, test-retest correlation and item-total correlation methods were used. The construct validity of the scale was carried out by exploratory factor analysis and confirmatory factor analysis methods. For assessing the criterion related validity were used Self Concealment Scale (SCS), Addiction Profile Index (BAPI), Beck Depression Inventory (BDI), Internalized Stigma of Mental Illness Scale (ISMIS), Multidimensional Scale of Perceived Social Support (MSPSS) and Rosenberg Self-Esteem Scale (RSES).

Results: Principle component analysis with varimax rotation and confirmatory factor analysis were applied to examine the factor structure of the Turkish AAQ-SA and two-factor structure was obtained similar to the original scale. In the construct validity analysis conducted by confirmatory factor analysis method, it was determined that the regression load of one item was not at the level of significance and the item was excluded from the scale. The Cronbach alpha coefficient of the 17-item final version was 0.736 and the Cronbach alpha coefficients of the sub-scales were 0.700-0.766. The item-total score correlation coefficients ranged from 0.100 to 0.523 (p <0.01). Test–retest reliability analysis at three weeks also showed good temporal stability (r=0.83). In terms of criterion related validity, the total score of the scale was significantly correlated with BAPI, SCS, BDI, MSPSS, ISMIS, RSES scores in the expected direction. In addition, the AAQ-SA scores were compared according to the severity of addiction which is obtained from BAPI scores and results indicated significant difference.

Conclusions: Our results of the study indicated that the Turkish version of the AAQ-SA can be used as a satisfactory reliable and valid scale.

Keywords: acceptance and action questionnaire, substance abuse, psychological flexibility, reliability, validity

Full Text XML

Reliability and Validity of the Turkish Version of the Acceptance and Action Questionnaire-Substance Abuse (AAQ-SA) on a Clinical Sample

Hilal Uygura, Kaasım Fatih Yavuzb, Ibrahim Erenc, Omer Faruk Uygura, Mahmut Selcukd, Nalan Varsake, Seda Yildirim Ozbekf, Basak Demirelc

a Department of Psychiatry, Necip Fazil City Hospital, Kahramanmaras, Turkey; b Department of Psychology, Medipol University, Istanbul, Turkey, c Department of Psychiatry, Training and Research Hospital, Konya, Turkey; d Department of Psychiatry, State Hospital, Balikesir, Turkey; e Psychiatry Specialist, Istanbul, Turkey; f Department of Psychiatry, Eregli State Hospital, Konya, Turkey.

 

 

ABSTRACT

Background: In this study, it is aimed to examine validity and reliability of the Turkish version of the Acceptance and Action Questionnaire-Substance Abuse (AAQ-SA) which is developed for assessing psychological flexibility levels of individuals with alcohol and/or substance misuse.

Methods: The research sample consisted of a total of 191 participants diagnosed with alcohol and substance use disorder. For reliability analysis, Cronbach alpha coefficient, test-retest correlation and item-total correlation methods were used. The construct validity of the scale was carried out by exploratory factor analysis and confirmatory factor analysis methods. For assessing the criterion related validity were used Self Concealment Scale (SCS), Addiction Profile Index (BAPI), Beck Depression Inventory (BDI), Internalized Stigma of Mental Illness Scale (ISMIS), Multidimensional Scale of Perceived Social Support (MSPSS) and Rosenberg Self-Esteem Scale (RSES). 

Results: Principle component analysis with varimax rotation and confirmatory factor analysis were applied to examine the factor structure of the Turkish AAQ-SA and two-factor structure was obtained similar to the original scale. In the construct validity analysis conducted by confirmatory factor analysis method, it was determined that the regression load of one item was not at the level of significance and the item was excluded from the scale. The Cronbach alpha coefficient of the 17-item final version was 0.736 and the Cronbach alpha coefficients of the sub-scales were 0.700-0.766. The item-total score correlation coefficients ranged from 0.100 to 0.523 (p <0.01). Test–retest reliability analysis at three weeks also showed good temporal stability (r=0.83). In terms of criterion related validity, the total score of the scale was significantly correlated with BAPI, SCS, BDI, MSPSS, ISMIS, RSES scores in the expected direction. In addition, the AAQ-SA scores were compared according to the severity of addiction which is obtained from BAPI scores and results indicated significant difference

Conclusions: Our results of the study indicated that the Turkish version of the AAQ-SA can be used as a satisfactory reliable and valid scale.

Keywords: acceptance and action questionnaire, substance abuse, psychological flexibility, reliability, validity

INTRODUCTION

Psychological flexibility is a relatively new construct which is defined as the ability to contact the present moment, the open acceptance of unpleasant sensations, thoughts, and feelings, and moving in a pattern of behavior in the service of chosen values (1). In contrast to psychological flexibility, psychological inflexibility refers to a model of behavior in which actions are rigidly guided by internal experiences (i.e., thoughts, feelings, and urges) rather than personal values (2). This model consists of six interrelated processes:  experiential avoidance, cognitive fusion, dominance of conceptualized past or/and future, attachment to conceptualized self, disruption of values, and inaction or impulsivity(3) Experiential avoidance is the most focused one and defined as trying to avoid or get rid of unwanted private experiences even when these avoidance cause behavioral harm (4). It is associated with a broad range of psychological and behavioral health problems including substance use disorders (5). Consistent with an experiential avoidance perspective, some individuals engage in substance use in an attempt to avoid a wide range of unpleasant internal states including unwanted thoughts, emotions, sensations (e.g., cravings or urges) (6). Problematic substance use may initially function to reduce psychological pain in the moment, but over time, that rigid patterns of avoidance can lead to paradoxical increases in unpleasant experiences and substance use itself becomes a trigger in dealing with craving and withdrawal symptoms (7). Furthermore a lifestyle persisting in substance use restricts engagement in personally important life activities, which results in loneliness, self-concealment, depression, self-stigmatization, decreased self-esteem and lack of social support, that all associated with psychological inflexibility (8,9).

Acceptance and Commitment Therapy (ACT) is a transdiagnostic cognitive-behavioral intervention that targets to improve psychological flexibility (10). There is good empirical evidence that targeting psychological flexibility with an ACT protocol is beneficial for a range of clinical disorders (11). ACT helps patients with substance use disorders to cope with challenges without using substances, to develope strategies to tolerate emotionally difficult or painful experiences (cravings, urges, bodily sensations, ets. ) (7). 

Psychological flexibility/inflexibility levels is typically measured by The Acceptance and Action Questionnaire-II (AAQ-II) (2). While AAQ-II has demonstrated inadequate psychometric properties in specific clinical samples, it has been developed specific variants of the AAQ in such areas as weight control, workplace stress, social phobia, body image, trichotillomania, stigma, and auditory hallucinations (12-18). Due to content-specific variants of the AAQ have been effective in other treatment areas , Luoma et al. (19) have developed  a substance abuse focused version of the AAQ: Acceptance and Action Questionnaire-Substance Abuse  (AAQ-SA). In this study, we aimed to examine the psychometric properties and factor structure of the Turkish AAQ-SA in a clinical sample, suggesting that it may be useful in mediating treatment outcomes in addiction.

METHODS

Participants 

The study includes participants, who were recieving treatment in outpatient or inpatient clinics at Konya Training and Research Hospital, Alcohol and Drug Reseach, Treatment and Training Center.  Exclusion criteria included being diagnosed with any acute psychotic or mood disorder, mental retardation or cognitive impairment, at that moment being under the influence of substance or alcohol. After psychiatric interview, all participants were informed about the study and their written consent was obtained. The study was conducted on a total of 191 patients. In order to examine the test-retest reliability, Turkish AAQ-SA was re-administered to randomly selected 26 patients three weeks after the initial survey.

Procedure

Ethics committee approval was obtained from Selçuk University Non-invasive Clinical Researches Ethics Committee (Date: 29.06.2016, Approval Number: 2016/200). The study was conducted between July 2016/January 2017.  Jason B. Luoma, who developed the original form of the scale, was contacted by e-mail and the required permission was obtained to be adapted to Turkish. The scale items were translated from English into Turkish independently by five psychiatrists who were talented in English grammar. The obtained translations are reviewed  and the statements which are thought to represent the best of each item are adopted by translation team.  The translated English version was  back translated into Turkish by three experts in linguistics who was blinded to the research. Then the final version of the scale was compared and checked for discrepancies between the Turkish and English translations by a associate professor who specializes in ACT.

Measurement Tools

Sociodemographic Data Form

It is a semi-structured form prepared by researchers in order to determine age, gender, education, place of residence, medical/psychiatric history and substance use characteristics of the sample. 

The Acceptance and Action Questionnaire-Substance Abuse  (AAQ-SA)

AAQ-SA is a self-report scale to evaluate psychological flexibility/inflexibility levels in alcohol and substance use disorder samples (19). The scale consists of 18 items rated on a 7-point Likert-type scale ranging from 1 (never true) to 7 (always true). High scores represent high psychological flexibility, which consists of two subscales under the heading of  “values commitment” and “defused acceptance”. Defused acceptance subscale is scored inversely because it represents psychological inflexibility. In the original study, the Cronbach alpha values for values commitment and defused acceptance were 0.82 and 0.84. The internal consistency of the overall scale was 0.85.

Self-Concealment Scale (SCS)

The original form was developed by Larson and Chastain (20) and consist of 10 items rated on a 5-point Likert scale. SCS refers to the tendency of a person to hide his/her personal information from others, which he perceives as distress or negative. Turkish version of the scale was conducted by Terzi et al. (21).

Rosenberg Self-Esteem Scale (RSES)

The Rosenberg Self-Esteem Scale is a common tool for assessing global self-esteem consisting of ten items which are rated on a 4-point Likert scale (22). Higher scores reflect higher levels of self-esteem. The validity and reliability of the Turkish version of scale was carried out by Fusun Cuhadaroglu (23). 

The Internalized Stigma of Mental Illness Scale (ISMIS)

ISMIS, which is developed by Ritsher et al. (24), evaluates self-stigma that reflects the inner experiences related to stigmatization among individuals with psychiatric disorders. The Turkish validity and reliability study of the scale was conducted by Ersoy and Varan (25). 

Multidimensional Scale of Perceived Social Support (MSPSS)

MSPSS is a 12-item, five-point Likert-type scale assessing one’s perception of social support from family, friends, and significant others (26). The reliability and validity of the Turkish version of the scale was performed by Eker and Arkar (27).

Beck Depression Inventory (BDI) 

The BDI is a 21-item self-report questionnaire rated on a 4-point Likert scale. It was developed to assess severity of depressive symptoms. Higher scores indicate an increase in depressive mood (28). Turkish validity and reliability study of the BDI was conducted by Hisli (29). 

Addiction Profile Index (BAPI)

Addiction Profile Index (BAPI) was developed by Ögel et al. (30).  The scale is a self report questionnaire that consists of 37 items and five subscales. The subscales measures the characteristics of substance use, dependency diagnosis, the effect of substance use on the person’s life, craving and the motivation for quitting using substances. Scores less than 12 indicates low levels of dependence, 12-14 medium and bigger than 12 points showes high severity of dependence levels. Cronbach alfa coefficient for the whole questionnaire is 0.89.

Statistical Analysis

SPSS (Statistical Package for the Social Sciences) 20.0 for Windows was used for all data analysis. Kolmogorov-Smirnov and Shapiro Wilk normality tests were performed to analyse homogeneity of variables. For the reliability of the Turkish version of AAQ-SA were applied test-retest method, Cronbach alpha correlation coefficient and item-total correlation by using Spearman's correlation coefficient. The construct validity of the scale was carried out by exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) methods. The Kaiser-Meyer-Olkin coefficient (KMO) and Bartlett’s sphericity test were used to verify the suitability of the data for factor analysis (31,32). The EFA was carried out by using varimax rotation based on the main compounds method. For assessing the criterion related validity was examined the relationship between AAQ-SA, SCS, ISMIS, BDI, MSPSS, RSES and BAPI by using Spearman’s correlation analysis. The independent Kruskal-Wallis test and Bonferroni correction were used to compare AAQ-SA scores with the severity of addiction which was obtained from BAPI. 

CFA was performed using SPSS AMOS 23 version for testing our factor structure which obtained from EFA. The quality of models can also be evaluated by its goodness of fit to data (33). Chi-square (χ2) is very sensitive to sample size, therefore it is used relative chi-square, which is the chi-square fit index divided by the degree of fredom (χ2/df), so makes χ2 less dependent on sample size (34). The other fit indices we used in our study are the Comparative Fit Index (CFI) (35), the General Fit Index (GFI), the Incremental Fit Index (IFI),  and the root mean square error of approximation (RMSEA) (36).  Values of CFI, GFI, IFI> 0.900, χ2 / df<5 and RMSEA <0.0854 are used as criteria for indicating a good fit (37,38).

RESULTS

Descriptive Statistics

The mean age of the participants was 25.6 (± 6.4), and 174 were male (91.1%). The marital status of the patients was 24.1% married and 75.9% was single / divorced / widow. In terms of education, 74.9% were primary school graduates, 19.3% were high school graduates and 5.8% were university graduates.

Construct validity

First, to control the suitability of the sample adequacy for the EFA, the Kaiser-Meier-Olkin (KMO) and Bartlett Tests were used. KMO coefficient of 0.60 and Barlett sphericity test calculated in the chi-square value should be statistically significant (39). In our study, KMO sample adequacy r = 0.737 (p <0.001) and Bartlett Chi-square test value sphericity 788.65 (p <0.001) showed that data were suitable for factor analysis. So 18 items were analysed through exploratory factor analysis (EFA) using Principal Component Analysis (PCA) and Varimax rotation. As result of PCA was obtained six factors (eigenvalues >1 as a criterion) (40) which accounted for  60.92% (n = 191) of the total variance  in contrast with the original scale. Since the original scale consists of two-factor structure, it was decided to test the two-factor solution by varimax rotation method. As result of Varimax rotation of AAQ-SA were obtained two factors (F1 = 3.60 and F2 = 2.75) which eigenvalues were extracted over 1, and explained 35.27% of the total variance. Also the scree plot supported two factor solution (Figure 1). Different from the original scale, item 12 was included in the second factor, and 18 was not loaded on any factors (Table 1). 

In addition to EFA, we performed confirmatory factor analysis (CFA) in order to test the new two-factor model. According to the fit indices, it was determined that the 18-item version of Turkish AAQ-SA did not show adequate compliance. Also, it was decided to remove item 11 from the model due to the low regression weights (p> 0.05). Goodness-of-fit indices revealed a high covariance-related measurement error between items 9-10 and 17-18 (figure 2). According to the final corrected goodness-of-fit indices of revised model with 17 items was found to be better to the previous model (RMSEA = 0.084, CFI = 0.942, IFI = 0.880, GFI = 0.906 and χ2 /df= 1.740) (Table 2).  Estimated standardized factor loadings for Turkish AAQ-SA (ranged between 0.39 and 0.88, p<0.001) were displayed in Figure 2. Then, further analyses were carried out on 17 items.

Reliability Analysis

Internal consistency, temporal stability and item analyses were performed to evaluate the reliability of Turkish AAQ-SA. Firstly, Cronbach's alpha correlation analysis was used to calculate the internal consistency of Turkish AAQ-SA. Cronbach's alpha for ‘values commitment’ was 0.700 and the alpha for ‘defused acceptance’ was 0.766, the internal consistency of the overall scale was 0.736. 

For the temporal stability of Turkish AAQ-SA was performed test-retest reliability analysis three weeks after the initial survey with 26 patient randomly selected. The test–retest correlation coefficient was calculated as r=0.830 (n=26; p<0.001). Besides, a t-test was conducted to compare the mean score between three weeks interval application. There was no significant difference for values commitment (t=1.061, p=0.299) and defused acceptance (t=-0.830, p=0.414) at three weeks. Results indicated that Turkish AAQ-SA has showed a good temporal stability between the two assessments. 

For item analysis was used the corrected item-total correlation method. Item-total correlation scores ranged between 0.100 (item 13) and 0.523 (item 16) as shown in Table 3.

Criterion Validity

For assessing the criterion related validity was used concurrent validity (41). The relationships between Self Concealment Scale (SCS), Addiction Profile Index (BAPI), Beck Depression Inventory (BDI), Internalized Stigma of Mental Illness Scale (ISMIS), Multidimensional Scale of Perceived Social Support (MSPSS), Rosenberg Self-Esteem Scale (RSES) with AAQ-SA (Table 4).  In addition, the AAQ-SA scores were compared according to the severity of addiction and results indicated significant difference (Table 5).

DISCUSSION

In this study, it was aimed to examine the Turkish adaptation, validity and reliability analysis of the AAQ-SA, which was developed by Jason B. Luoma to measure the level of psychological flexibility in relation to specifically substance use related thoughts, feelings, and urges (19). Initially, Exploratory Factor Analysis (EFA) was performed. As a result of EFA with varimax rotation obtained two factors but  different from the original scale item 12 was loaded on the second factor, and 18 was not loaded on any factors.  In order to test the new two-factor model was performed an initial confirmatory factor analysis (CFA). When item 18 was placed on defused acceptance subscale as in the original, was compatible with the model. Also, it was determined that the 18-item version of Turkish AAQ-SA did not show adequate compliance according to the fit indices. Therefore, it was decided to remove item 11 (If I promised to do something, I’ll do it, even if I later don’t feel like it) from the model due to the low regression weights (p> 0.05). Following the removal of the item 11, it was determined that the final corrected goodness-of-fit indices of revised model with 17 items was found to be better. 

Besides, it is considerable that, as a result of both CFA and EFA, item 12 was loaded on ‘defused acceptance’ subscale different from the original scale. Considering the content of item 12 (“Having some worries about substance use will not prevent me from living a fulfilling life”), it is seemed that emphasis both on value-oriented behaviors and on the attitudes that provides resistance towards a desired life, in relation to internal experiences, such as anxiety. When unwanted inner experiences are decisive rather than impressive on one's behavior and presence of attitudes towards avoiding them indicate psychological inflexibility. Defused acceptance subscale assesses the weakness of these skills of reducing psychological inflexibility. Considering the fact that which are mentioned above, it is seen to be understandable that item 12 was located on defused acceptance subscale because emphasizing the concerns about substance use, rather than value-oriented behaviours. 

In contrast to the exploratory factor analysis, CFA allows the detection of measurement errors caused by item similarities, content overlaps, demand characteristics, intricacy and methodological effects (42). In our study, we found two correlated measurement errors mentioned specified by CFA. Similarly worded items as ‘recovery’ and overlapping of content which emphasize the existence of negative situations in the recovery process, being located on successive may explain the high correlation level and first method effect between item 9 and 10. Further, both items 17 and 18 focus on worrying about craving and the desire to use substance, have similar contents and this similarity may explain the high level of correlation. After correcting the measurement errors, the final corrected goodness-of-fit indices of revised model with 17 items was found to be acceptable. With these findings, we can say that Turkish AAQ-SA shows an acceptable factorization structure in our study. Further analyses were carried out on 17 items. 

Moreover the concurrent validty was tested  with BAPI, SCS, BDI, MSPSS, ISMIS and RSES.  The relationships between Turkish AAQ-SA and other scales were significantly correlated in the expected direction. The highest correlation with full scale was found with the internalized stigma of mental illness scale (r = -. 453**). It is known that the people with alcohol and substance addiction are labeled as negative judgments such as low morality, personality weakness and crime tendency by the society resulting in self-stigmatization (43). Similar to other studies in the literature there was a significant negative correlation between the level of stigmatization and psychological flexibility in our study (8,44). Likewise, it is expected that the use of substance which is an approved or unaccepted behaviour in the community may affect self-esteem and therefore self-esteem may be low in substance addicts (9). Examining the correlations with addiction properties, the AAQ-SA scores were significantly different compared according to the severity of addiction . The results confirm the hypothesis that those with more severe and persistent addiction would score lower on the AAQ-SA, as substance use is considered to reflect a form of experiential avoidance (45). Relationships with depression, social support, and self concealment were also at a low label correlated. 

The internal reliability coefficients of the Turkish AAQ-SA were satisfactory, the Cronbach alpha coefficient of the 17-item final version was 0.736 and for the subscales were 0.700-0.766. Test-retest reliability analysis at three weeks showed good temporal stability (46).

As result of item-total correlation analysis were found that item-total correlations of items 10, 13 and 15 to be less than 0.20. In the literature, the cut-off values vary for the correlation of item-total analysis, some authors have stated that this value should be at least 0.20 (47). In several studies, total item correlation serves as a criterion for initial assessment and purification, but for very small values, before removing from the scale; it is recommended to decide the importance of the item, checking how the scale will be modified if the item is deleted (46). When these items reviewed, it was seen that their factors were loaded of a medium size (r=0.693 for the item 10; r=0.502 for the item 13 and r=0.369 for the item 15) (40), they showed high regression weights in the CFA model, strengthened the meaning integrity of the factor on which they are, and provided a conceptually vital dimension to the scale. Therefore, it was considered they should not be excluded from the scale although the correlations of these items with the item-sum were not adequate. 

Limitations of the present study is that the sample was consisted with a limited age and mostly of men. There is a need for future studies of homogeneous gender distribution and wider age range among substance use disorders patients.

In conclusion, our findings suggest that the Turkish version of the 17-item (item 11 removed) of AAQ-SA could be used as a satisfactory reliable and valid scale. In our study, we tried to make a scale that would contribute the treatment goals in psychosocial treatment of addiction. We think that Turkish AAQ-SA may provide useful information about processes of substance abuse treatment and guide modifications to ongoing treatment strategies.

REFERENCES

  1. Hayes SC, Luoma JB, Bond FW, Masuda A, Lillis J. Acceptance and commitment therapy: model, processes and outcomes. Behav Res Ther 2006;44(1):1-25

  2. Bond FW, Hayes SC, Baer RA, Carpenter KM, Guenole N, Orcutt HK, et al. Preliminary psychometric properties of the acceptance and action questionnaire–II: a revised measure of psychological inflexibility and experiential avoidance. Behav Ther 2011;42(4):676-88.

  3. Hayes SC, Pistorello J, Levin ME. Acceptance and commitment therapy as a unified model of behavior change. The Counseling Psychologist 2012;40(7):976-1002.

  4. Hayes SC, Strosahl K, Wilson KG, Bissett RT, Pistorello J, Toarmino D, Niccolls R. Measuring experiential avoidance: A preliminary test of a working model. The Psychological Record 2004;54:553-78.

  5. Hayes SC, Wilson KG, Gifford EV, Follette VM, & Strosahl K. Experiential avoidance and behavioral disorders: A functional dimensional approach to diagnosis and treatment. J Consult Clin Psychol 1996;64(6):1152-68.

  6. Zgierska A, Rabago D, Chawla N, Kushner K, Koehler R & Marlatt A. Mindfulness meditation for substance use disorders: A systematic review. Substance Abuse 2009;30(4), 266-94.

  7. Serowik KL and Orsillo SM. The relationship between substance use, experiential avoidance and personally meaningful experiences. Substance Use & Misuse 2019; 54(11):1834-44.

  8. Luoma JB, Twohig MP, Waltz T, Hayes SC, Roget N, Padilla M, Fisher G. An investigation of stigma in individuals receiving treatment for substance abuse. Addictive Behaviors 2007;32:1331-46.

  9. Uba I, Yaacob SN, Talib MA, Abdullah R, Mofrad S. The Role of Self-Esteem in the diminution of substance abuse among adolescents. International Review of Social Sciences and Humanities 2013;5(2):140-49.

  10. Hayes SC, Strosahl KD, Wilson KG. Acceptance and Commitment Therapy: The Process and Practice of Mindful Change. 2nd ed. New York: Guilford; 2011.

  11. Ruiz FJ. A review of Acceptance and Commitment Therapy (ACT) empirical evidence: Correlational, experimental psychopathology, component and outcome studies. International Journal of Psychology and Psychological Therapy 2010;10,125-62.

  12. Lillis J, Hayes SC. Measuring avoidance and inflexibility in weight related problems. International Journal of Behavioral Consultation and Therapy 2008;4(4):348-54.

  13. Bond FW, Lloyd J, Guenole N. The work-related acceptance and action questionnaire (WAAQ): Initial psychometric findings and their implications for measuring psychological flexibility in specific contexts. Journal of Occupational and Organizational Psychology 2012;1-25.

  14. MacKenzie MB, Kocovski NL. Self-reported acceptance of social anxiety symptoms: development and validation of the Social Anxiety-Acceptance and Action Questionnaire. International Journal of Behavioral Consultation and Therapy 2010;6:214-32.

  15. Sandoz K, Wilson KG, Merwin RM, Kellum KK. Assessment of body image flexibility: The Body Image-Acceptance and Action Questionnaire. Journal of Contextual Behavioral Science 2013;2:39-48. 

  16. Houghton DA, Compton SN, Twohig MP, Saunders SM, Franklin ME et al. Measuring psychological inflexibility in trichotillomania. Psychiatry Res 2014;220(1-2):356-61.

  17. Levin ME, Luoma JB, Lillis J, Hayes SC, Vilardaga R. The Acceptance and Action Questionnaire-Stigma (AAQ-S): Developing a measure of psychological flexibility with stigmatizing thoughts. Journal of Contextual Behavioral Science 2014;3(1):21-26. 

  18. Shawyer F, Ratcliff K, Mackinnon A, Farhal J, Hayes SC, Copolov D. The voices acceptance and action scale (VAAS): Pilot data. Journal of clinical psychology 2007;63:593-606.

  19. Luoma JB, Drake CE, Kohlenberg BS, Hayes SC. Substance abuse and psychological flexibility: The development of a new measure. Addiction Research and Theory 2011;19(1):3-13. 

  20. Larson DG, Chastain RL. Self-Concealment: Conceptualization, measurement and health implications. Journal of Social and Clinical Psychology 1990;9(4):439-55. 

  21. Terzi S, Gungor HC, Erdayi GS. Adaptation of the Self- Concealment Scale: A validity and reliability study. The Journal of Turkish Educational Sciences 2010;8(3):645-660.

  22. Rosenberg M. Society and the adolescent self-image. Princeton, NJ, Princeton University Press, 1965.

  23. Cuhadaroglu F. Self-Esteem in Adolescents. Unpublished Dissertation, Ankara, Hacettepe University, Faculty of Medicine, Department of Psychiatry, 1986.

  24. Ritsher JB, Otilingam PG, Grajales M. Internalized stigma of mental illness: psychometric properties of a new measure. Psychiatry Research 2003;121:31-49. 

  25. Ersoy MA and Varan A. Reliability and Validity of Turkish Version of Internalized Stigmatization of Mental Illness Scale. Turkish Journal of Psychiatry 2007; 18 (2): 163-71.

  26. Zimet GD, Dahlem NW, Ziment SG, Farley GK. The Multidimensional Scale of Perceived Social Support. Journal of Personality Assessment 1988;52:30-41.

  27. Eker D and Arkar H. The factor structure, validity and reliability of the Multidimensional Scale of Perceived Social Support. Turkish Journal of Psychology 1995;10(34):17-25.

  28. Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Arch Gen Psychiatry 1961;4(6):561-71.

  29. Hisli N. Reliability and validity of Beck Depression inventory among university students. Turkish Journal of Psychology 1989;7(23):3-13

  30. Ogel K, Evren C, Karadag F, Tamar GD. Development of the addiction profile index, its validity and reliability, Turkish Journal of Psychiatry 2012;23(4):264-73

  31. Kaiser HF. A second generation little jiffy. Psychometrika 1970;35(4):401-15.

  32. Bartlett MS. A note on the multiplying factors for various chi-square approximations. Journal of the Royal Statistical Society 1954;16(Series B):296-98.

  33. Byrne BM. Structural Equation Modeling With AMOS: Basic concepts, applications, and programming. New York, NY: Routledge Academic; 2010.

  34. Jon WH. The analysis of covariance structures: Goodness-of-fit indices. Sociological Methods and Research 1983;11(3):325-44. 

  35. Bentler PM. Comparative fit indexes in structural models. Psychological Bulletin 1990;107(2):238-46. 

  36. Hu LT, Bentler P. Evaluating model fit. In: Hoyle RH, ed. Structural equation Modeling. concepts, issues, and applications. London: Sage 1995; p. 76-99.

  37. Munro BH. Statistical methods for health care research (Vol.1): Lippincott Williams & Wilkins, 2005. 

  38. Simsek OF. Introduction to structural equation modeling: Basic principles and lisrel applications. Ankara, Ekinoks Publishing, 2007.

  39. Buyukozturk S. Factor Analysis: Basic concepts and their use in scale development. Educational Administration in Theory and Practice 2002; 32: 470-83.

  40. Buyukozturk S. Guidebook of data analysis for social sciences. The 13th edition, Ankara: Pegem Academy, 2011.

  41. Taherdoost H. Validity and reliability of the research instrument: How to test the validation of a questionnaire/survey in a research. International Journal of Academic Research in Management 2016;5(3):28-36.

  42. Brown TA, Moore MT. Confirmatory factor analysis. In: Hoyle RH, ed. Handbook of structural equation modeling. New York: Guilford Press; 2012, p.361-79.

  43. Vardar E. Stigmatization of alcohol and substance abuse. Anatolian Journal of Psychiatry 2009;10:62-63.

  44. Krafft J, Ferrell J, Levin ME, Twohig MP. Psychological inflexibility and stigma: A meta-analytic review.  Journal of Contextual Behavioral Science 2018;7:15-28.

  45. Wilson KG, Hayes SC, Byrd MR. Exploring compatibilities between acceptance and commitment therapy and 12-step treatment for substance abuse. Journal of Rational-Emotive and Cognitive-Behavior Therapy 2000;18:209-34.

  46. Alpar R. Applied statistics and validity-reliability with examples in sports, health and educational sciences. 5th ed. Ankara, Detay Publishing, 2010; 361-3.

  47. Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Psychological Assessment 1995;7:309-19.