Y researchers and therapists, which might aid the investigation of negative effects in a variety of different psychological treatments and to explore their relationship with treatment outcome. Providing an instrument that can identify adverse and unwanted events during the treatment period may also help therapists identify patients at risk of faring worse, and to offer other treatment interventions as a way of reversing a negative treatment trend.Methods Item designItems were carefully generated using a consensus statement regarding the monitoring and reporting of negative effects [32], findings from a treatment outcome study of patients with social anxiety disorder that probed for adverse and unwanted events [42], the results of a qualitative content analysis of the responses from four different clinical trials [44], and a literature review of books and published articles on negative effects. This is in line with thePLOS ONE | DOI:10.1371/journal.pone.order FPS-ZM1 0157503 June 22,4 /The Negative Effects Questionnairerecommendations by Cronbach and Meehl [45], advising researchers to articulate the theoretical concept of an instrument before developing and testing it empirically in order to increase content validity. Also, instead of restricting the number of items to be included in a final version, the concept of overinclusiveness was adapted, that is, embracing more items than necessary to aid the statistical analyses necessary for detecting those that are related to the underlying construct(s) [46]. Subsequently, 60 items were generated, characterized by interpersonal issues, problems with therapeutic relationship, deterioration, novel symptoms, stigma, dependency, hopelessness, difficulties understanding the treatment content, as well as problems implementing the treatment interventions. An additional open-ended question was also included for the investigation of negative effects that might have been experienced but were not listed, i.e., “Describe in your own words whether there were any other negative incidents or effects, and what characterized them”. Further, in order to assess the readability and understanding of the items, cognitive interviews were conducted on five individuals unrelated to the current study and without any prior knowledge of negative effects or psychological treatments, i.e., encouraging them to read the items out load and speak freely of whatever comes to mind [47]. Cognitive interviews are often suggested as a way of pretesting an instrument so that irrelevant or difficult items can be revised and to increase its P144 site validity [48]. In relation to the proposed items, several minor changes were made, e.g., rephrasing or clarifying certain expressions. In addition, the instrument included general information about the possibility of experiencing negative effects, and was comprised of three separate parts; 1) “Did you experience this?” (yes/no) 2) “If yes ere is how negatively it affected me” (not at all, slightly, moderately, very, and extremely), and 3) “Probably caused by” (the treatment I received/other circumstances). The instrument is scored 0? and contains no reversed items as this may introduce errors or artifacts in the responses [49].Data collectionThe instrument was distributed via the Internet using an interface for administering surveys and self-report measures, Limesurvey (www.limesurvey.org). Participants were recruited via two different means in order to include a diverse and heterogeneous sample: patients under.Y researchers and therapists, which might aid the investigation of negative effects in a variety of different psychological treatments and to explore their relationship with treatment outcome. Providing an instrument that can identify adverse and unwanted events during the treatment period may also help therapists identify patients at risk of faring worse, and to offer other treatment interventions as a way of reversing a negative treatment trend.Methods Item designItems were carefully generated using a consensus statement regarding the monitoring and reporting of negative effects [32], findings from a treatment outcome study of patients with social anxiety disorder that probed for adverse and unwanted events [42], the results of a qualitative content analysis of the responses from four different clinical trials [44], and a literature review of books and published articles on negative effects. This is in line with thePLOS ONE | DOI:10.1371/journal.pone.0157503 June 22,4 /The Negative Effects Questionnairerecommendations by Cronbach and Meehl [45], advising researchers to articulate the theoretical concept of an instrument before developing and testing it empirically in order to increase content validity. Also, instead of restricting the number of items to be included in a final version, the concept of overinclusiveness was adapted, that is, embracing more items than necessary to aid the statistical analyses necessary for detecting those that are related to the underlying construct(s) [46]. Subsequently, 60 items were generated, characterized by interpersonal issues, problems with therapeutic relationship, deterioration, novel symptoms, stigma, dependency, hopelessness, difficulties understanding the treatment content, as well as problems implementing the treatment interventions. An additional open-ended question was also included for the investigation of negative effects that might have been experienced but were not listed, i.e., “Describe in your own words whether there were any other negative incidents or effects, and what characterized them”. Further, in order to assess the readability and understanding of the items, cognitive interviews were conducted on five individuals unrelated to the current study and without any prior knowledge of negative effects or psychological treatments, i.e., encouraging them to read the items out load and speak freely of whatever comes to mind [47]. Cognitive interviews are often suggested as a way of pretesting an instrument so that irrelevant or difficult items can be revised and to increase its validity [48]. In relation to the proposed items, several minor changes were made, e.g., rephrasing or clarifying certain expressions. In addition, the instrument included general information about the possibility of experiencing negative effects, and was comprised of three separate parts; 1) “Did you experience this?” (yes/no) 2) “If yes ere is how negatively it affected me” (not at all, slightly, moderately, very, and extremely), and 3) “Probably caused by” (the treatment I received/other circumstances). The instrument is scored 0? and contains no reversed items as this may introduce errors or artifacts in the responses [49].Data collectionThe instrument was distributed via the Internet using an interface for administering surveys and self-report measures, Limesurvey (www.limesurvey.org). Participants were recruited via two different means in order to include a diverse and heterogeneous sample: patients under.