Psychometric validation of the Turkish Gaming Disorder Test: a measure that evaluates disordered gaming according to the World Health Organization Framework

 

Cuneyt Evrena, Halley M. Pontesbc, Ercan Dalbudakd, Bilge Evrene, Merve Topcuf, Nilay Kutlue 

 

aResearch, Treatment, and Training Center for Alcohol and Substance Dependence (AMATEM), Bakirkoy Training and Research Hospital for Psychiatry Neurology and Neurosurgery, Istanbul, Turkey

bSchool of Psychological Sciences, College of Health and Medicine, University of Tasmania, Australia

cThe International Cyberpsychology and Addictions Research Laboratory (iCARL), University of Tasmania, Australia

dDepartment of Psychiatry, Yuksek Ihtisas University Faculty of Medicine, Ankara, Turkey

eDepartment of Psychiatry, Baltalimani State Hospital for Muskuloskeletal Disorders, Istanbul, Turkey 

fCankaya University Department of Psychology, Ankara, Turkey

   

ABSTRACT

Background: Previous research on gaming disorder (GD) used psychometric tools, which evaluates according to the American Psychiatric Association (APA) diagnostic framework. The Gaming Disorder Test (GDT), a standardized measure to assess symptoms and prevalence of GD according to the World Health Organization (WHO) diagnostic framework. The main aim of the current study was to adapt GDT to Turkish. 

METHOD: In the present study participants were assessed with the GDT, the Internet Gaming Disorder Scale–Short-Form (IGDS9-SF), and the CAGE-Problematic Internet Use Questionnaire (CAGE-PIUQ). The factor structure of the scale was tested with Confirmatory Factor Analysis (CFA), and reliability and validity analyses were conducted.

RESULTS: A sample of 932 Turkish gamers (58.3% male, mean age 23.64 years, SD=5.42) was recruited online. Confirmatory factor analyses demonstrated that the unidimensional factor structure of the GDT was satisfactory. The scale was also reliable (i.e., internally consistent with a Cronbach’s alpha of 0.879) and showed adequate convergent and criterion-related validity, as indicated by statistically significant positive correlations between average time daily spent playing games (ATDSPG) during last year, IGDS9-SF and CAGE-PIUQ scores. By applying the International Classification of Diseases 11th edition (ICD-11) threshold for diagnosing GD (e.g., meeting all four criteria by answering them either with ‘often’ [4] or ‘very often’ [5]), it was found that the prevalence of GD is 1.9% (n = 18).

CONCLUSION: Online gaming preference, ATDSPG and probable ADHD predicted the severity of disordered gaming. These findings support the Turkish version of the GDT as a valid and reliable tool for determining the extent of GD related problems among young adults and for the purposes of early GD diagnosis in clinical settings and similar research.

Keywords: e-sports; gaming disorder; GDT; Internet; scale; university students; young adults 

Introduction

The interest of researchers and the society in the problems arising from excessive use of technology, especially the use of the Internet and playing video games, is gradually increasing [1]. Although the positive effects of healthy gaming have been widely demonstrated by previous research [2], gaming can become pathological for a minority of players when the activity becomes dysfunctional, harming an individual’s social, occupational, family, school, and psychological functioning due emerging functional impairment caused by excessive engagement with the activity [3]. 

The adoption of inconsistent terminologies and non-standardized assessment tools in previous research investigating Gaming Disorder (GD) [4] has led to several methodological issues and subsequent debates among scholars as to whether the phenomenon represents a unique clinical entity worth being officially recognized as a behavioral addiction [5-7]. More recently, several milestones have been achieved in the field. According to Pontes and Griffiths [8], the first major milestone for GD research took place in May 2013, when the fifth edition of Diagnostic and Statistical Manual of Mental Disorders (DSM-5) [9] included nine criteria (i.e., preoccupation, tolerance, withdrawal, persistence, displacement conflict, deception, escape, and problems) defining Internet Gaming Disorder (IGD), in Section III ‘Emerging Measures and Models’ as a tentative disorder requiring additional research before possible formal recognition could be achieved in future revisions of the DSM [5,10-12]. More specifically, the nine IGD criteria refer to preoccupation with Internet games, withdrawal symptoms, tolerance, unsuccessful attempts to control participation in Internet games, loss of interest in previous hobbies, continued excessive use of Internet games, deceiving family members, use Internet games to escape, and losing a significant relationship, job or education, or career opportunity [9]. According to the DSM-5, IGD is clinically characterized by a “persistent and recurrent use of the Internet to engage in games, often with other players, leading to clinically significant impairment or distress” [9]. At more severe levels, IGD may lead to academic failure, job loss, or marriage failure as the problematic behavior tends to displace usual and expected social, occupational and/or educational, relationship, and family activities [13]. To be diagnosed as a disordered gamer, the American Psychiatric Association [9] suggested that at least five out of these nine criteria need to be endorsed by the gamer over a period of 12 months.

Seven clinical psychometric tools covering the nine IGD criteria have been developed following the initial inclusion of this condition by the APA in the DSM-5 [9,14]. Among these measures, particularly the Internet Gaming Disorder Scale (IGDS) [15], the nine-item Internet Gaming Disorder Scale–Short-Form (IGDS9-SF) [7]  and the 10-item Internet Gaming Disorder Test (IGDT-10) [16] have received clinical attention and have been widely examined cross-culturally. Turkish versions of the IGDS [15,17], the IGDS9-SF [7,18], and the IGDT-10 [16,19] were previously validated in Turkish populations. Moreover, the prevalence rates of IGD, according to the APA framework has been found to range from 0.03% in Finland [20] to 9.3% in Lithuania [21] in robust studies collecting large and representative samples around the world. A study conducted in Turkey among both university students and gamers found that the prevalence of disordered gamers according to the APA framework ranged from 0.96% (in the whole sample including university students) to 2.57% (in e-sports players) [18]. Also, a large-scale epidemiological study (Turkey’s Addiction and Mental Health Risk Profile Map Project [TURBAHAR]) carried out in Turkey in 2018 with 24,494 participants aged 18–81 years reported the rate of disordered gaming as 1.6% [22]. 

According to Pontes and Griffiths [8], the second major milestone for GD research took place in 2016, when GD was included in the beta draft of the 11th edition of the International Classification of Diseases (ICD-11) as a “behavioral addiction” [23]. Research on the clinical significance of playing video games, the health burden it brings, and its neurobiological similarities with other substance use disorders have made this inclusion necessary [24]. More specifically, GD is defined in the beta draft of the ICD-11, as a pattern of persistent or recurrent online and/or offline gaming behavior manifested by three core diagnostic criteria: (1) impaired control over gaming (e.g., onset, frequency, intensity, duration, termination, context); (2) increasing priority given to gaming to the extent that gaming takes precedence over other life interests and daily activities; and (3) continuation or escalation of gaming despite the occurrence of negative consequences [23]. Although GD involves recurrent gaming behavior which may be online or offline, its clinical symptoms must be assessed within a 12-month timeframe (the required diagnostic duration may be shortened if all diagnostic requirements are met, and symptoms are severe) and be of sufficient severity to result in significant impairment across different life domains including personal, family, social, educational, occupational and/or other broad areas of functioning [23]. The WHO further specifies that differential diagnosis of GD can be achieved by screening for the following exclusion criteria: hazardous gaming, bipolar type I, and type II disorders [23]. Finally, according to Pontes and Griffiths [8], the third key milestone took place in May 25th 2019 at the 72nd World Health Assembly which culminated in the historic and long-awaited decision by the WHO [25] officially recognizing GD as a mental health disorder after an extensive and iterative review process

In order to advance research and further the scientific understanding of GD under the new diagnostic framework established by the WHO, Pontes et al. [12] developed the Gaming Disorder Test (GDT), a brief four-item measure with a single-factor structure to assess GD. In their study Pontes et al. [12] included populations from China and United Kingdom (UK), because addictive gaming has become a major public health issue in Asian countries such as China (with prevalence rates ranging from 3.5 to 17%) [26] and an emerging issue in developed Western countries such as the UK (with prevalence rates reported around 14.6%) [27]. The study suggested that the four items of the GDT for both Chinese and English versions are valid, reliable, and proved to be highly suitable for measuring GD within a cross-cultural context. A follow-up study conducted by Montag et al. [13] provided additional support for the WHO diagnostic framework for GD and its measurement with the German version of the GDT.

In Turkey, the Ministry of Health adopts ICD codes for official diagnoses. Therefore, the diagnosis of GD can now be made in the Turkish health system, which may be beneficial to support disordered gamers seeking help from the public health system and clinicians. However, there is still the need to be able to distinguish between potentially disordered and non-disordered gamers using updated psychometric tools based on ICD criteria. At present, a brief Turkish assessment tool for GD, according to the WHO framework, is lacking. Thus, the aim of the present study is the adaptation of the GDT to Turkish (see “Appendix” section) and to evaluate its psychometric properties (i.e., internal consistency, unidimensionality, construct validity, convergent and criteria-related validity) among different gamers (e.g., professional and non-professional). The present study also aims to provide reliable prevalence rates of GD among Turkish young adults. By conducting the present study and achieving the proposed aims, this study will hopefully make a unique contribution to advancing the knowledge base on the cross-cultural implications in the assessment of GD using the new framework developed by the WHO within the ICD-11.

Method

Participants and procedure

An online survey using a cross-sectional design was conducted in order to adapt and test the psychometric properties of the Turkish GDT. Initially, data were collected from people who were in the e-mail database of a company located in Istanbul that organizes e-sports tournaments (ESL Turkey Amateur e-sport players), who were in the e-mail database of a game development company located in Ankara (Taleworlds Entertainment), and Turkish speaking gamers from gaming forums. Moreover, additional data were collected from Turkish university students from Cankaya University in Ankara, with people who had reported that they did not engage in gaming activities being excluded from the study.

The study protocol was approved by the Ethical Committee of Cankaya University (Turkey). The institutional review board approval date was March 9th, 2020, and the number was 90705970-050.99E.00000057153. The study was anonymous and confidential. After reading the Plain Language Information Statement, online informed consent was obtained from all participants. The online survey was created using Qualtrics. Furthermore, participants were informed that they would not be penalized for not wanting to participate and/or abandoning the study once they have started. Participants sampled from Cankaya University who agreed to participate in the study were rewarded with a bonus credit, which added to their overall grade for specific courses they were enrolled. 

Overall, the data collection stage spanned from March 10th 2020 to April 4th 2020. A total of 1,260 potential participants initiated the online survey. Among these, data from 328 participants were systematically missing. Although these participants provided informed consent, they did not fully complete the survey, which resulted in their exclusion from the study. Thus, a total of 932 participants were included in the study. The mean age of the sample was 23.64 years (SD = 5.42). Among these, 543 were male (58.3%), and 389 were female (41.7%). More than half of participants reported being university students that occasionally engaged in gaming activities on the Internet (n = 577, 61.9%).  

Measures

Sociodemographic and Gaming-Related Variables

Sociodemographic data included participants’ gender, age, and relationship status. Gaming related behaviors were controlled by self-reported average time spent playing video games during the last year. Additional self-reported questions were included asking if participants had purchased any video games, preferred mode of play (online/offline gaming), gaming more than usual in weekends, watching games at platforms like Twitch and Youtube, and whether they had experienced any significant problems in their lives due to gaming (yes/no).

Gaming time

Categories for weekly gaming time were the following: (1) “less than seven hours weekly (less than one hour a day),”(2) “7-14 hours weekly (1-2 hours per day)”, (3) “15- 28 hours weekly (2-4 hours per day)”, (4) “29-42 hours weekly (4-6 hours per day)”, and (5) “more than 42 hours weekly (more than 6 hours per day)”.

Gaming Disorder Test (GDT)

To measure disordered gaming through the WHO diagnostic framework [23], the Gaming Disorder Test (GDT) [12] was used. The GDT is a brief standardized assessment tool, which consists of four items (reflecting the key defining diagnostic features of GD in the ICD-11) answered on a five-point Likert scale ranging from 1=‘never’ to 5= ‘very often’ developed to assess symptoms of GD in the past 12 months (the WHO criteria for GD are based on persistent and recurrent gaming). This most often involves specific online and/or offline games, regardless of the device used to play (e.g., consoles, computers, smartphones). Total scores are obtained by summing up all four items (range = 4 to 20 points), with higher scores being indicative of higher degrees of disordered gaming. Although the main purpose of this instrument is not to diagnose GD but to assess its severity and to accompany detrimental effects to the gamer’s life, for research purposes, a participant may be classed as a disordered gamer by answering all four items either with ‘often’ (4) or ‘very often’ (5). By adopting this (≥ 4) diagnostic approach, researchers are able to distinguish between potentially disordered and non-disordered gamers [12]Previous research reported excellent psychometric properties for the GDT [12,13].

In this study, two Turkish psychiatrists fluent in English translated the GDT from English to Turkish. The translated version was agreed upon by these specialists. In order to establish their comparability, the Turkish version of the GDT was then translated from Turkish to English by a separate translator. The final translation was presented to 40 students (20 were male, and 20 were female; meanage = 21.9 years, SDage = 3.3 years) from Cankaya University to determine whether the language was clear and to ensure the scale’s face validity. The respondents did not report any significant problems when completing the questionnaire.

Internet Gaming Disorder Scale–Short-Form (IGDS9-SF)

The IGDS9-SF was used to further assess the validity of the GDT [7]. According to the APA diagnostic framework, the IGDS9-SF assesses the symptoms and severity of IGD and its detrimental effects by examining both online and/or offline gaming activities occurring over a 12-month period [7]. The scale comprises nine items corresponding to the nine core criteria defined by the DSM-5. They are answered on a five-point Likert scale ranging from 1=‘never’ to 5=’ very often,’ and high scores on the scale translate onto a higher level of disordered gaming. In the present study, the Turkish version of the IGDS9-SF was used [18]. The IGDS9-SF was also found to have an excellent level of internal consistency in the present study (Cronbach’s alpha = 0.895).

CAGE- Problematic Internet Use Questionnaire (CAGE-PIUQ)

Given that a high association has been found between Internet addiction and GD in previous studies, some authors have argued that GD represents “a part of the postulated construct of Internet addiction” [28]. Thus, in the present study, the CAGE- Problematic Internet Use Questionnaire (CAGE-PIUQ) [29] was also used to further investigate the validity of the GDT. The CAGE- PIUQ was adapted from the CAGE [30,31], which is a 4- item, relatively non-confrontational questionnaire for detection of alcoholism, to screen individuals for problematic Internet use, via changing words “alcohol” to “Internet use” among Turkish university students. Confirmatory factor analysis revealed that the CAGE-PIUQ resulted in an acceptable model fit (χ2/df=3.34/2=1.67; root mean square error of approximation RMSEA=0.088, goodness of fit index GFI=0.982, adjusted GFI=0.910, parsimony GFI=0.196, normed fit index NFI=0.962, comparative fit index CFI=0.984, incremental fit index IFI=0.984). The internal consistency reliability coefficient of the scale was 0.68. The Pearson product-moment correlation between the CAGE-PIUQ and the Addiction Profile Index Internet Addiction Form (APIINT) [32] scores was moderate (r=0.62, p<0.001). The CAGE-PIUQ has been shown to be reliable and valid for university students [29]. In the present study, the Cronbach’s alpha of the CAGE-PIUQ was 0.713.

Data Analysis 

    To carry out the analyses, IBM SPSS Amos was used for Confirmatory Factor Analysis (CFA), and IBM SPSS Statistics Version 20 for Windows (SPSS Inc., Chicago, IL, USA) was used for the remaining statistical analyses. Prior to the analyses, data cleaning was conducted by the inspection of cases with severe missing values across the main instruments. The distribution of all items across all psychometric tests utilized in the present study was examined to assess univariate normality. As a result, no item of the GDT and the other psychometric tests had absolute values of skewness > 3.0 and kurtosis > 8.0 [33]. Frequencies and percentages were given for sociodemographic variables, whereas means and standard deviations were given for age and scale scores.

    The following strategies were used to investigate the psychometric properties of the Turkish GDT: (a) its factorial structure was first examined using CFA, in which goodness of fit indexes was used; (b) convergent and criterion-related validity were determined by estimating Pearson product-moment correlation coefficients between the total scores of the GDT, IGDS9-SF, CAGE-PIUQ and the self-reported average daily time spent playing games during last year; (c) internal consistency was assessed using Cronbach’s alpha.

 

Results

Factor structure

    In order to investigate the factor structure and dimensionality of the GDT, CFA was conducted using the data collected. Prior to any further analysis, the adequacy of sample size was verified using Bartlett’s Test of Sphericity and the Keiser-Meyer-Olkin (KMO) measurement of sampling adequacy. The Bartlett’s Test of Sphericity was significant (χ= 1911.317, df=6, p < 0.001) for the GDT, and the KMO measure of sampling adequacy was acceptable at 0.82. 

    The unidimensionality of the Turkish GDT was then subsequently assessed via CFA with maximum likelihood. In order to evaluate the quality of the model estimated in the CFA, several fit indices were used and the following thresholds adopted: χ2/df ≤ 5, Tucker-Lewis Fit Index (TLI) and Comparative Fit Index (CFI) > 0.90, and Root Mean Square Error of Approximation (RMSEA) < 0.05 [34-37]. The estimation of a unidimensional model produced a good fit (χ2/df = 9.46/2 = 4.73; CFI = 0.98, TLI = 0.91 and RMSEA = 0.048). As seen in Table 2, all item-component loadings were statistically significant (ranged from 0.758 to 0.847) and within the conventionally acceptable threshold of > 0.50 [38]. Thus, results from the CFA suggest that the GDT assesses a unidimensional construct.

Convergent and criterion-related validity

    The literature defines convergent validity as the extent to which items of a psychometric test appear to be indicators of a single underlying construct [39]. Convergent validity is deemed adequate when the Average Variance Extracted (AVE) of the latent variable is ≥ 0.50, and composite reliability (CR) is ≥ 0.70 [38,40]. As shown in Table 2, the AVE value for the GDT was adequate (0.73), and the composite reliability coefficient was well beyond the desired threshold (0.92).

    Convergent validity was also assessed by correlating the GDT scores with the scores of two related scales (i.e., the IGDS9-SF and CAGE-PIUQ), and criterion-related validity was evaluated through examination of the correlation between the GDT scores and self-reported average daily time spent gaming during the last year. The correlation between the GDT and the IGDS9-SF (r = 0.78, p < 0.001) and the CAGE-PIUQ (r = 0.45, p < 0.001) was statistically significant. Moreover, this result was also consistent with the association between the GDT scores and self-reported average daily time spent on gaming during the last year (r = 0.40, p < 0.001). Overall, these results demonstrate positive correlations among the variables of interest in the expected direction according to the underlying theory, thus supporting the validity of the Turkish GDT.

Internal Consistency Reliability

    In terms of the reliability of the Turkish GDT, the Cronbach’s alpha coefficient was high (α = 0.88) (Table 2). Moreover, item-total correlations for the GDT were equally robust, ranging between 0.70 (item 1) and 0.77 (item 3) (Table 2). Furthermore, inter-item correlations for the GDT ranged between 0.56 (between items 1 and 4) and 0.69 (between items 2 and 3) (Table 2). 

Prevalence rates of GD

    As per the diagnosis recommendation made by the WHO, participants answering all four items either with ‘often’ (4) or ‘very often’ (5) in this study were operationally classed as disordered gamer [25]. Thus, the prevalence of potential GD was 1.9% (n = 18) in the present study.

Discussion

    The main aim of the current study was to adapt and examine the psychometric properties of the Turkish GDT. To achieve this aim, the newly developed psychometric tool was tested in a cross-sectional study using an online survey to recruit Turkish university students and gamers (both amateur and professional). The findings of the present study supported the validity of the GDT across several levels. A single-factor solution for the GDT was found in CFA, further supporting the unidimensional factor structure of the GDT, as found in previous studies [12,13]. The results of the CFA yielded statistically significant and relatively high factor loadings, further demonstrating that all items were adequate indicators of the GD construct and that the scale has adequate psychometric properties, alongside a solid factor structure. In the original study, the GDT showed adequate reliability (Cronbach’s alpha of 0.84; 0.87 for Chinese participants and 0.84 for British participants) [12]. Cronbach’s alpha was even higher (0.96) for the German version [13]. Consistent with these Cronbach’s alphas, the Turkish version obtained similar results regarding the reliability of the GDT (Cronbach’s alpha of 0.88). In addition to this result, criterion-related and convergent validity was supported by the expected positive pattern of correlations that have emerged between the GDT and all the related measures. The convergent validity of the scale was indicated by the significant correlations of the GDT with the IGDS9-SF and the CAGE-PIUQ, whereas the criterion-related validity of the scale was indicated by the significant correlation with the average daily time spent gaming during the last year. In general, higher mean scores on the scale indicated greater average daily time spent on games and higher severities of IGD and Internet addiction. The GDT provided a valid and reliable measure of GD with adequate diagnostic accuracy that can be used for research and diagnostic purposes among young adult male and female gamers. Overall, the results obtained in the present study corroborated prior validity studies on the GDT [12,13].

    Although there are obvious discrepancies in the two GD diagnostic frameworks put forth by the APA and the WHO (i.e., different focus on core diagnostic criteria and a different number of criteria needed to be endorsed for diagnosis), a recent study suggested the existence of minor discrepancies in the estimation of prevalence rates of GD according to between the two frameworks [13]. Consistent with this, the prevalence of those potentially meeting a positive GD diagnosis according to the WHO framework among the present sample was 1.9% (n = 18), which was consistent with a large-scale epidemiological study carried out recently in Turkey and found the rate of disordered gaming as 1.6% according to the APA framework [22]

    The present study is not without its potential limitations. Several potential shortcomings within the present study should be considered. First, the use of an online survey excludes people who do not have access to the Internet. Therefore, these findings may not be extended to participants who mostly play offline games due to a lack of Internet access. Second, because all participants were self-selected, a generalization of the present findings to the general population cannot be directly made. Third, students from Cankaya University were awarded extra credits for participation, which may have resulted in a higher number of participants in this group. This can be considered as a limitation that could have affected the results. Fourth, the study may also be limited by the fact that all the data were collected using self-report questionnaires, which is a method subject to well-known associated biases, such as social desirability biases, short-term recall biases, etc. Finally, there was no diagnosis of GD for each participant using a gold standard. Therefore, this study was unable to estimate the sensitivity and specificity of the GDT in detecting GD. Future studies could benefit from replicating these findings among individuals clinically diagnosed with GD as data on clinical samples are currently sparse. 

    Despite these potential limitations, the results of the validity and reliability testing of the Turkish GDT were found to be similar to the findings reported by previous studies. The present findings support the Turkish version of the GDT, which measures a unidimensional construct, as being a valid and reliable GD screening tool in assessing the symptoms and prevalence of GD among young adults. These findings support the use of the GDT for the purposes of early diagnosis and in other relevant research examining excessive and disordered gaming. The present study will enable Turkish clinicians to diagnose GD in the Turkish public health system as the Turkish Ministry of Health uses ICD codes for official diagnoses. It is envisaged that improved GD assessment may also encourage disordered gamers to seek help from the public health system.

   

References

[1] Zajac K, Ginley MK, Chang R, Petry NM. Treatments for Internet gaming disorder and Internet addiction: A systematic review. Psychol Addict Behav. 2017; 31:979-94.

[2] Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM. A systematic literature review of empirical evidence on computer games and serious games. Comput Educ. 2012; 59:661–86.

[3] Gentile DA, Choo H, Liau A, et al. Pathological video game use among youth: A two-year longitudinal study. Pediatrics. 2011:127; 319-29.

[4] King DL, Haagsma MC, Delfabbro PH, Gradisar M, Griffiths MD. Toward a consensus definition of pathological videogaming: A systematic review of psychometric assessment tools. Clin Psychol Rev. 2013; 33:331-42. 

[5] Griffiths MD, Pontes HM. Internet addiction disorder and internet gaming disorder are not the same. J Addict Res Ther. 2014;5: e124.

[6] Pontes HM, Griffiths MD. Assessment of Internet Gaming Disorder in clinical research: Past and present perspectives. Clin Res Regul Aff. 2014; 31:35-48.

[7] Pontes HM, Griffiths MD. Measuring DSM-5 Internet gaming disorder: Development and validation of a short psychometric scale. Comput Human Behav. 2015; 45: 137–43.

[8] Pontes HM, Griffiths MD. A new era for gaming disorder research: Time to shift from consensus to consistency. Addict Behav. 2020; 103:106059.

[9] American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (5th edition). Arlington, VA. American Psychiatric Press, 2013.

[10] Petry NM, O’Brien CP. Internet gaming disorder and DSM-5. Addiction. 2013;108: 1186–7.

[11] Petry NM, Rehbein F, Gentile DA, et al. An international consensus for assessing Internet gaming disorder using the new DSM-5 approach. Addiction. 2014; 109:1399–1406. 

[12] Pontes HM, Schivinski B, Sindermann C, et al. Measurement and conceptualization of Gaming Disorder according to the World Health Organization framework: The development of the Gaming Disorder Test. Int J Ment Health Addict. 2019; 1-21.

[13] Montag C, Schivinski B, Sariyska R, Kannen C, Demetrovics Z, Pontes HM. Psychopathological Symptoms and Gaming Motives in Disordered Gaming-A Psychometric Comparison between the WHO and APA Diagnostic Frameworks. J Clin Med. 2019; 8:1691.

[14] Kuss DJ, Pontes HM. Internet addiction (Vol. 41). Boston: Hogrefe Publishing Corporation 2019.

[15] Lemmens JS, Valkenburg PM, Gentile DA. The Internet Gaming Disorder Scale. Psychol Assess. 2015; 27:567–82.

[16] Király O, Sleczka P, Pontes HM, Urbán R, Griffiths MD, Demetrovics Z. Validation of the Ten-Item Internet Gaming Disorder Test (IGDT-10) and evaluation of the nine DSM-5 Internet Gaming Disorder criteria. Addict Behav. 2017; 64:253-60. 

[17] Evren C, Dalbudak E, Topcu M, Kutlu N, Evren B. The psychometric properties of the Turkish version of the Internet Gaming Disorder Scale. Dusunen Adam: The Journal of Psychiatry Neurological Sciences 2017; 30:316–24.

[18] Evren C, Dalbudak E, Topcu M, Kutlu N, Evren B, Pontes HM. Psychometric validation of the Turkish nine-item Internet Gaming Disorder Scale–Short Form (IGDS9-SF). Psychiatry Res. 2018; 265:349-54.

[19] Evren C, Evren B, Dalbudak E,  Topcu M, Kutlu NPsychometric validation of the Turkish Ten-Item Internet Gaming Disorder Test (IGDT-10). Dusunen Adam: The Journal of Psychiatry Neurological Sciences. 2020; 33:19-28.

[20] Karhulahti VM, Koskimaa R. On the prevalence of addicted or problematic gaming in Finland Addictive Behaviors Reports. Addict Behav Rep.  2019; 10: 100225. 

[21] Ustinavičienė R, Škėmienė L, Lukšienė D, Radišauskas R, Kalinienė G, Vasilavičius P. Problematic computer game use as expression of Internet addiction and its association with self-rated health in the Lithuanian adolescent population. Medicina. 2016; 52:199-204. 

[22] Ünübol H, Koç AŞ, Hızlı Sayar G, Stavropoulos V, Kircaburun K et al. Measurement, profiles, prevalence, and psychological risk factors of problematic gaming among the Turkish community: a large-scale national study. Int J Ment Health Addiction. 2020; 1-21.

[23] World Health Organization 2018. International classification of diseases, 11th Revision (ICD-11). ICD-11 Beta Draft: Gaming Disorder. Available from: www.who.int/classifications/icd/.

[24] Saunders JB, Hao W, Long J, King DL, Mann K, Fauth-Bühler M et al. Gaming disorder: Its delineation as an important condition for diagnosis, management, and prevention. J Behav Addict. 2017; 6:271-9. 

[25] World Health Organization. 6C51 Gaming disorder 2019. Available from:  https://www.who.int/features/qa/gaming-disorder/en/

[26] Long J, Liu T, Liu Y, Hao W, Maurage P, Billieux J. Prevalence and correlates of problematic online gaming: a systematic review of the evidence published in Chinese. Curr Addic Rep. 2018; 5: 359–71. 

[27] Lopez-Fernandez O, Honrubia-Serrano ML, Baguley T, Griffiths MD.  Pathological video game playing in Spanish and British adolescents: towards the exploration of internet gaming disorder symptomatology. Comput Human Behav. 2014; 41: 304–12.

[28] Monacis L, Palo V, Griffiths MD, Sinatra M. Validation of the Internet Gaming Disorder Scale – Short-Form (IGDS9-SF) in an Italian-speaking sample. J Behav Addict. 2016; 5:683-90.

[29] Dalbudak E, Evren C, Aldemir S, Evren B. Psychometric properties of the CAGE questionnaire assessment for problematic internet use among university students. Bulletin of Clinical Psychopharmacology 2015; 25(Suppl. 1): S72-3.

[30] Ewing JA. Detecting alcoholism. The CAGE questionnaire. JAMA 1984; 252:1905–7.

[31] Mayfield D, McLeod G. Hall P. The CAGE questionnaire: validation of a new alcoholism screening instrument. Am J Psychiatry. 1974; 131:1121–3.

[32] Ogel K, Karadag F, Satgan D, Koc CDevelopment of the Addiction Profile Index Internet Addiction Form (APIINT): validity and reliability Dusunen Adam: The Journal of Psychiatry and Neurological Sciences 2015;28:337-43

[33] Kline RB. Principles and practice of structural equation modeling (third edition). New York: The Guildford Press 2011.

[34] Hair JF, Anderson RE, Tatham RL, Black WC. Multivariate Data Analysis. Seventh edition, Prentice-Hall, Upper Saddle River, New Jersey 2010.

[35] Ferguson E, Cox T. Exploratory factor analysis. A user’s guide. International Journal of Selection and Assessment. 1993; 1:84-94.

[36] Kaiser HF. The application of electronic computers to factor analysis. Educ Psychol Meas. 1960; 20: 141-51.

[37] Lin CY, Luh WM, Cheng CP, Yang AL, Su CT, Ma HI. Measurement equivalence across child self-reports and parent-proxy reports in the Chinese version of the Pediatric Quality of Life Inventory Version 4.0. Child Psychiatry Hum Dev. 2013; 44: 583-90.

[38] Wu TH, Chang CC, Chen CY, Wang JD, Lin CY. Further psychometric evaluation of the Self-Stigma Scale-Short: measurement invariance across mental illness and gender. PLoS One. 2015; 10: e0117592.

[39] Lee ZWY, Cheung CMK, Chan TKH. Massively multiplayer online game addiction: instrument development and validation. Inf Manag. 2015; 52:413–30.

[40] Fornell C, Larcker DF. Evaluating structural equation models with unobservable variables and measurement error. J Mark Res. 1981;18: 39–50.

[41] Dong G, Wang L, Du X, Potenza MN. Gender-related differences in neural responses to gaming cues before and after gaming: implications for gender-specific vulnerabilities to Internet gaming disorder. Soc Cogn Affect Neurosci. 2018; 13:1203–14.

[42] Fuster H, Carbonell X, Pontes HM, Griffiths MD. Spanish validation of the Internet Gaming Disorder-20 (IGD-20) Test. Comput Hum Behav. 2016; 56: 215–24.

[43] Lemmens JS, Valkenburg P, Peter J. Development and validation of a game addiction scale for adolescents. Media Psychol. 2009;12: 77-95.

[44] Schivinski B, Brzozowska-Woś M, Buchanan EM, Griffiths MD, Pontes HM. Psychometric assessment of the internet gaming disorder diagnostic criteria: an item response theory study. Addict Behav Rep. 2018; 8:176–84.