Introduction

Evidence-based practice (EBP) aims to integrate “the best research evidence with clinical expertise and patient values”1 when providing health and medical services. EBP implementation in routine care has shown to improve outcomes for patients2 as well as cost-effectiveness of interventions for health care systems3,4,5. Still, there is evidence for a persistent research-practice gap6,7. This has given rise to a global focus on research efforts in order to investigate barriers and facilitators of the dissemination and implementation of EBP in health care8. Adequate measurement of implementation mechanisms is required to advance this research area. It should have a conceptual, theoretical, and empirical basis, show good psychometric qualities, and be pragmatic9.

Implementation theories and frameworks have been developed to provide consistent definitions of relevant constructs and a basis for empirical investigations of the relationships between these constructs10. These frameworks are valuable for guiding the planning, design and process of hands-on efforts to implement EBP11. One widely used implementation framework is the Exploration, Preparation, Implementation, Sustainment (EPIS) framework11,12: The framework describes these four phases of the implementation process and identifies especially relevant factors during these phases: the outer system, inner context, bridging factors and innovation factors. Within the inner context, both organizational characteristics (e.g., the organizational implementation climate) and individual adopter characteristics (e.g., providers’ attitudes towards EBP implementation) are highlighted as central to the implementation success during the exploration and active implementation phase12.

Implementation climate is defined as “employees’ shared perceptions of the importance of innovation implementation within the organization”13, or whether “the adoption, implementation, and use of an innovation such as EBP is expected, rewarded, and supported by the organization”14. This organizational factor consistently appears to improve the implementation success15,16,17,18. A 5-year panel analysis indicated that improvements of the implementation climate resulted in increased EBP use by clinicians18. Although implementation research is often criticized for relying on self-report measures when assessing implementation outcome, two recent studies that use expert and observer ratings support the assumption that implementation climate predicts the adherence to EBP19,20. The implementation climate, in turn, is associated and thought to be influenced by other organizational factors, for example leadership16,21,22,23. Moreover, a stronger implementation climate is associated with more positive attitudes towards the adoption of EBP among the providers working within the organization 22,24.

One pragmatic instrument designed to assess the implementation climate of organizations or units within organizations such as a clinic or team is the Implementation Climate Scale (ICS) 14. It was developed based on review of existing literature and through a participatory process in collaboration with subject matter experts (e.g., experts in leadership and climate, clinical program managers, and clinicians). Building upon this past research, it is assumed that organizations that are willing to create an optimal organizational climate for EBP implementation (1) emphasize the priority of EBP implementation, (2) provide educational support for the EBPs, (3) recognize and (4) provide rewards for employees that use EBPs, and select employees that are experienced with the use of EBPs (5) or are willing to adopt new practices (6)14. The 18 items of the ICS comprise six subscales that are then aggregated to represent the overall implementation climate (see Table 1). Accordingly, the factor structure could best be regarded as a Second-order model. The items are designed to be group-referenced, that is referring to the unit level of interest. For example, questions may be phrased to refer to the whole organization, to a specific clinic, or some other organizational unit (e.g., hospital, health center, etc.). The ICS was designed originally in the mental health care context and has subsequently shown strong psychometric properties in a wide range of contexts from substance use disorder treatment organizations25, child welfare26, nursing27 and education settings28. A Norwegian study indicated cross-cultural validity in a translated version29.

Table 1 ICS Subscales with item examples.
Full size table

So far, no German translation of the ICS exists. This is unfortunate in view of the rising importance and efforts of implementation research in German-speaking countries and the associated need for reliable and valid German instruments assessing implementation constructs30. Regarding evidence-based mental health care for example, implementation research is growing in relevance and attention in German-speaking countries31,32. In Germany, a research opportunity for the implementation of EBP arises in the context of the regulation of psychotherapy practice by law33. The relevant legislation (‘Psychotherapeutengesetz’, PTG)33 was enacted on 1 January 1999. It provides legal protection of the title ‘psychotherapist’ and regulates the training requirements. In addition, it stipulates that psychotherapy must be based on scientific evidence (decided about by a Scientific Advisory Board of the Federal Medical Chamber and the Federal Psychotherapeutic Chamber)33. The training requirements to become a licensed psychotherapist include a medical, psychological, or educational degree followed by a period of practical postgraduate training33,34. Once psychotherapists are licensed, they can provide outpatient services working in private practices with reimbursement by the statutory (or private) health insurance. Besides outpatient services, a substantial part of the mental health care in Germany is offered as inpatient services by specific hospitals33,35. An amendment of the PTG was approved in September 2019, aiming to further align the training for psychotherapy to the existing structure of medical education34. Psychotherapy training always had to take place at state approved training sites. In the future, pre-graduate training will be restricted to universities. Whether this will be helpful to close the existing research-practice gap and will lead to an accelerated uptake of EBP in mental health care will be an important field of research during the next years. The assessment of implementation climate may help to inform understanding of determinants and mechanisms of more or less successful strategies and arrangements in this context36.

The goal of the present study was to translate the ICS into German and examine its psychometric properties. Based on survey data from a sample of German psychotherapists, the translated ICS was evaluated on the individual practitioner level in terms of item analysis, scale and subscale reliability, comparison of different factor models, assessment of measurement invariance across gender groups and psychotherapy license status. Convergent validity was assessed based on correlations of the ICS with the individual behavioral intentions for EBP use, with the expectation that behavioral intentions, and particularly ratings of subjective norms, would be higher when climate perceptions were higher. Further analyses included examining the relationships between the ICS and attitudes towards EBP, as well as several demographic characteristics (age, gender, license status, and in-patient vs. outpatient setting).

Methods

Procedure

Translation

The ICS was translated in accordance with the WHO recommendations (www.who.int/substance_abuse/research_tools/translation/en/). The ICS was translated into German by the first author (KS) and back-translated by the bilingual English-speaking senior author (AB). The original authors (MGE, GAA) of the scale reviewed back translated items to ensure that they reflected the original constructs. Item 11 was adjusted to fit the German mental healthcare system, which does not offer any wage increases when EBP is implemented. The original and back-translated versions were then reviewed in a consensus meeting of the translating authors. A group of German clinical psychotherapists (in training) and researchers (n = 26) as well as a graduate linguist reviewed this consented version for comprehensibility and wording. Their suggestions were discussed and considered by the translating authors in a second consensus meeting, resulting in a final German version of the scale (see Supplemental material 1).

Recruitment and data collection

All data were collected online via an openly accessible online survey, using the scientific survey platform SoSci Survey (www.soscisurvey.de). The survey was online and accessible from 14th November 2019 to 27th April 2020. The link to the online survey was widely distributed via e-mail lists of professional psychotherapy organizations that all licensed psychotherapists are members of, universities, training institutes, and psychiatric in- and outpatient institutions as well as Facebook groups of psychotherapists. On the last page of the survey, participants could opt to receive information about the study results or take part in a raffle. If they chose one or both of these options, they were linked to a separate page where they could register their e-mail addresses independently of their survey answers.

Ethics

The Internal Review Board of the University of Marburg approved the online survey study (approval number: 2019-58 k). All methods were performed in accordance with the institutional guidelines. Participants provided informed consent after receiving study information and before they were able to access the survey. All raw data were stored securely at the Department of Psychology at Philipps University in Marburg, Germany, and were collected anonymously.

Participants

Eligible participants were licensed psychotherapists for adults, children, and adolescents as well as psychotherapists enrolled in postgraduate training to obtain such a license. Therefore, all participants had medical, psychological, or educational degrees. No exclusion criteria except other profession were applied. The link to the online survey was visited 2,417 times. Overall, 913 participants continued after providing informed consent. Of these, 863 met the inclusion criteria (i.e., profession). A total of 251 participants were excluded because they discontinued the survey before completing the ICS, five due to conspicuous response patterns in the ICS (e.g., straight-lining despite reverse coded items), two due to implausible answers (e.g., being 99 years old). A total of 180 licensed psychotherapists (29.8% of the total sample) stated working in private practices. These participants (80.6% female, M = 48.1 years, SD = 11.9) were excluded from further analyses since the ICS captures the implementation climate of organizations and work group.

The final analysis sample consisted of 425 participants that stated working in organizations and work groups and thus completed the ICS, with an age of 23 to 61 years (M = 31.9, SD = 6.6) and consisting of 362 (85.2%) women and 62 (14.6%) men. A large portion of the sample reported working in science (44.2%) or stated being in postgraduate training to become psychotherapists (79.4%); 97.9% reported a German nationality. For further information regarding the samples’ professional characteristics, see Table 2.

Table 2 Demographics and information on profession for the sample.
Full size table

Measures

Demographics and information on training and profession

Participants provided information on age, gender and nationality as well as their education and occupation. In detail, we asked participants about their university degree, license status, therapy orientation, and current occupation and work context.

Implementation Climate Scale (ICS)

The ICS is an 18-item instrument measuring the implementation climate in organizations and work groups14. Respondents were asked to rate their agreement with statements describing the climate with regard to the implementation of EBP in the organization or work group that they currently work for. A 5-point Likert scale ranging from 0 (‘not at all’) to 4 (‘to a very great extent’) is used. Six subscales can be calculated: Focus on EBP (α = 0.91 in our sample); Educational Support for EBP (α = 0.85); Recognition for EBP (α = 0.80); Rewards for EBP (α = 0.78); Selection for EBP (α = 0.84); Selection for Openness (α = 0.78). The total scale (α = 0.90) is created by computing means of the subscales. Higher scores (range 0 to 72 for the total scale) indicate a more favorable rating of the organizations’ implementation climate. The German instrument including scoring instructions can be found in the Supplemental material 1.

Evidence-based practice attitudes scale (EBPAS-36D)

The EBPAS-36 was developed to assess mental health and social service providers’ attitudes towards adopting EBP37 and a validated German translation is available38. The 36 items form 12 subscales of three items each: Requirements (α = 0.89 in our sample), Appeal (α = 0.69), Openness (α = 0.75), Divergence (α = 0.65), Limitations (α = 0.82), Fit (α = 0.68), Monitoring (α = 0.77), Balance (α = 0.65), Burden (α = 0.81), Job security (α = 0.89), Organizational support (α = 0.85), and Feedback (α = 0.76). Respondents are asked to rate their agreement with statements on a 5-point Likert scale ranging from 0 (‘not at all’) to 4 (‘to a very great extent’). Most items are worded in a way that higher scores indicate more positive attitudes towards the adoption of EBP; 15 items are scored reversely. A mean of the subscales can be computed to create a total scale (α = 0.89).

Intention scale for providers-direct items (ISP-D)

The ISP is a 70-item instrument assessing individual behavioral intentions for EBP use39 based on the theory of planned behavior40. The Direct Items measure of the ISP was previously investigated regarding its psychometric properties41. It consists of 16 items that assess attitudes (ISP-D-A, 5 items, α = 0.74 in our sample, higher scores indicating more negative attitudes), subjective norms (ISP-D-SN, 3 items, α = 0.84, higher scores indicating greater perceived social pressure to perform EBP), perceived behavioral control (ISP-D-PBC, 4 items, α = 0.71 in our sample, higher scores indicating a higher perceived control to perform EBP) and behavioral intention to use EBP (ISP-D-BI, 4 items, α = 0.88 in our sample, higher scores indicating a higher readiness to implement EBP). Responses are given on 7-point rating scales. The original English version was translated into German by the first author (KS) and back-translated by the bilingual English-speaking senior author (AB).

Statistical analysis

The statistical analyses were performed using IBM SPSS 28 for Windows (Chicago, IL, USA). For the Confirmatory factor analysis (CFA) and Exploratory Structural Equation Model (ESEM), R version 4.1.2 was used with the packages lavaan42 and GPArotation43. In all analyses, p values < .05 were set as thresholds for statistical significance. For the ICS, means were computed if there was a maximum of one missing item per subscale. Otherwise, respondents were excluded from the analyses of factorial validity, which was the case for five participants (0.01%). In all analyses, individual practitioner data were used. Mardia's multivariate skewness and kurtosis statistics44 was used to test multinormality, besides the Kaiser–Meyer–Olkin (KMO) test of sampling adequacy45 and Bartlett’s test of sphericity46. While both the latter indicated suitability of the data for factor analysis (KMO = 0.879, χ2 = 4443.55, df = 153, p < .001), the significant Mardia’s test statistics for skewness (p < .001) and kurtosis (p < .001) indicated multivariate normality deviation, leading to the use of robust estimation methods.

Standard item analyses were calculated, including item difficulties, corrected item-whole correlations, and Cronbach’s alpha if an item is deleted. The item difficulties—which in the context of measurement of attitudes can be interpreted as the mean endorsement of the respective item—was calculated as the actual endorsement of the item (sum of all participants’ scores) divided by the maximum possible endorsement (participant number * maximum score for any individual participant)47. This calculation results in an item difficulty pi between 0 and 1, with 0 signifying no endorsement and 1 maximum endorsement. Values of pi < 0.20 are regarded as low, values of 0.20 < pi < .80 as medium and diagnostically ideal because they differentiate best between high and low endorsers and items of pi > 0.80 as high, i.e. an item that generally was endorsed by most respondents48.

To obtain internal reliability coefficients of the scales and subscales, McDonald’s omega ω49 was calculated with 95% bias-corrected and accelerated bootstrap confidence intervals50 and 10 000 bootstrap replications using the R package MBESS51.

To assess construct validity, the factorial validity of the German version of ICS was examined by comparing the model fits of an independent cluster model of CFA (ICM-CFA), a Bifactor CFA, a Second-order CFA, an ESEM and a Bifactor ESEM. The maximum likelihood estimation method with robust standard error estimator was used with full-information maximum likelihood to handle missingness. The root mean square error of approximation (RMSEA), standardized root mean squared residual (SRMR), comparative fit index (CFI), and the ratio of χ2 to df were assessed as fit indices. RMSEA < 0.06 to 0.08 with confidence interval, SRMR ≤ 0.08, CFI ≥ 0.95 and χ2/df ≤ 2 or 3, indicate acceptable fit52. The Akaike Information Criterion (AIC) was computed to compare the models.

Thereafter, a multiple-group CFA was conducted with the Second-order model and a multiple-group SEM with the ESEM model to test the measurement invariance between men and women and between licensed psychotherapists and psychotherapists in training. The Second-order CFA model was chosen since it demonstrated comparable model fit indices while at the same time it can be regarded as the theoretically assumed factor structure. Different levels of measurement invariance were tested by defining a baseline model to assess configural invariance with similar loading patterns across groups, a metric invariance model with equated factor loadings, a scalar invariance model, constraining factor loadings and item intercepts to be equal across groups, and a latent means model, constraining item and error residuals to be equal across groups53. Configural invariance can be assumed in case of a good fit of the baseline model and groups sharing significant loadings, metric invariance if the metric invariance model is not substantially worse than the baseline model, and scalar invariance if the third model is not substantially worse than the baseline model. As model fit index, the chi-square test statistic was conducted besides the RMSEA, SRMR and CFI. Differences in fit indices between models are assumed to be substantially if the chi-square test statistic is significant and ∆CFI ≥ 0.0154. Scalar invariance is required to be able to infer group differences on actual differences in the latent variable rather than on the measurement.

To assess the convergent validity of the ICS, the following hypothesis was tested by calculating Pearson correlation coefficients assuming content overlap of the TPB construct subjective norms and the organizational implementation climate: The ICS total scale shows a high positive correlation to the ISP-D-SN. Smaller correlations are assumed to exist with the ISP-D-A, ISP-D-PBC and ISP-D-BI. According to Cohen’s classification55, r = 0.50 indicates high, r = 0.30 medium and r = 0.10 low correlation effect sizes. Group differences between men and women and between licensed psychotherapists and psychotherapists in training on the ICS were assessed with independent t-tests. In light of the unbalanced sample sizes56,57, Welch’s58 unequal variances t-test was used.

Pearson coefficients were calculated to assess correlations between ICS and EBPAS-36D as well as between age and the ICS. The latter result was compared to partial correlations between age and ICS total scale when controlling for license status, current occupation (in- or outpatient services) and EBPAS-36D total score. The findings are reported following the STrengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline59 and informed by the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) taxonomy60.

Ethical approval and consent to participate

The study was approved by the Internal Review Board of the University of Marburg (approval number: 2019-58 k). Participants received study information and provided informed consent. Data were collected anonymously.

Results

Item analysis

The item difficulties of the ICS ranged between pi = 0.09 (item 12) and pi = 0.69 (item 17) with a mean difficulty of pi = 0.47. The mean inter-item correlation was r = 0.34. The items 10, 11 and 12 show markedly lower difficulties (pi = 0.09 to pi = 0.15) than all other items. The item-total correlations of the individual items with the total scale ranged from ritc = 0.28 (item 12) to ritc = 0.77 (item 15) with a mean item-total correlation of ritc = 0.55. Only item 12 showed a ritc < 0.30 (see Table 3). The correlations of the individual items with their subscales ranged from ritc = 0.46 (item 18) to ritc = 0.85 (item 3).

Table 3 Item analyses of ICS.
Full size table

Reliability

The internal consistency of the total scale was ω = 0.91 [0.89; 0.92]. Internal consistencies of the original model ICS subscales were: Focus on EBP ω = 0.92 [0.90; 0.93]; Educational Support for EBP ω = 0.86 [0.84; 0.89]; Recognition for EBP ω = 0.82 [0.78; 0.84]; Rewards for EBP ω = 0.79 [0.72; 0.84]; Selection for EBP ω = 0.84 [0.81; 0.87]; Selection for Openness ω = 0.80 [0.75; 0.84].

Subscale correlations

The correlation coefficients between the ICS total scale and the six subscales are presented in Table 4. All subscale correlations were significant. The highest correlation was between the total scale and the Selection for EBP subscale (r = 0.83).

Table 4 Pearson correlations of ICS total scale, ICS subscales and age.
Full size table

Validity

Model comparison

The path diagrams of the models (ICM-CFA, Bifactor CFA, Second-order CFA, ESEM and Bifactor ESEM) including their standardized regression coefficients are shown in the Supplemental material 2. All model fit indices are found in Table 5. It should be noted that for the ESEM and the Bifactor ESEM, the variance–covariance matrices of the estimated parameters were not positive definite probably indicating non-identified models.

Table 5 Comparison of model fit indices.
Full size table

In the ICM-CFA and the Second-order CFA, all regression weights were significant. In the Bifactor CFA, the regression weights of item 9 on the Recognition for EBP subscale was non-significant. In the ESEM, significant regression weights were found for the items 1, 2 and 3 on the Focus on EBP subscale, for the items 4, 5 and 6 on the Educational Support for EBP subscale, for the items 7 and 8 on the Recognition for EBP subscale, for item 11 on the Rewards for EBP subscale and for items 16, 17 and 18 on the Selection for Openness subscale. In the Bifactor ESEM, significant regression weights were found for the items 1, 2 and 3 on the Focus on EBP subscale, for the items 4, 5 and 6 on the Educational Support for EBP subscale, for item 10 on the Rewards for EBP subscale, for items 13 and 14 on the Selection for EBP subscale and for the item 17 on the Selection for Openness subscale. All items except the items 9, 10, 11, 12, 16, 17 and 18 show significant regression weights on the Implementation Climate scale.

The model fit indices (see Table 5) indicated adequate fit for the ICM-CFA (RMSEA [90% CI] = 0.070 [0.062; 0.079], SRMR = 0.067, CFI = 0.94), the Bifactor CFA (RMSEA [90% CI] = 0.062 [0.054; 0.071], SRMR = 0.057, CFI = 0.96), the Second-order CFA (RMSEA [90% CI] = 0.077 [0.069; 0.085], SRMR = 0.078, CFI = 0.93) and slightly better model fits for the ESEM (RMSEA [90% CI] = 0.061 [0.046; 0.077], SRMR = 0.012, CFI = 0.99) and the Bifactor ESEM (RMSEA [90% CI] = 0.039 [0.020; 0.057], SRMR = 0.010, CFI = 0.99). However, the latter models demonstrated less parsimony with significantly fewer degrees of freedom (df = 34–36) compared to the other models (df = 117–129). The AICs of the models were not substantially different, ranging from 18,668.69 (Bifactor ESEM) to 18,872.17 (Second-order CFA).

Measurement invariance

A multiple-group CFA based on the Second-order CFA model and a multiple-group SEM based on the ESEM model were used to test for measurement invariance in the ICS across males and females as well as across licensed psychotherapists and psychotherapists in training. The results of the CFA and ESEM approach do not differ substantially. Fit indices and chi-square test statistics for all measurement models are found in Table 6. Adequate model fits for all measurement models were found. Non-significant χ2 difference test statistics and ∆CFI < 0.01 indicate no substantial differences between the configural, metric and scalar models for both gender and psychotherapy license comparisons, allowing to assume measurement invariance. Differences emerge across licensed psychotherapists and psychotherapists in training comparing the scalar model and the residual invariance model, indicating latent means invariance is not given.

Table 6 Multiple-group Second-order CFA and ESEM model fit indices.
Full size table

Correlation analyses

Table 7 shows the results of the correlation analyses between the ICS total scale and the ISP-D scales. As expected, a high positive correlation between the ICS total scale and the ISP-D-SN was found (r = 0.590**, p < 0.001, n = 403). Correlations to all other ISP-D scales were significant, yet smaller (see Table 7).

Table 7 Pearson correlations of ICS total scale and the ISP-D scales.
Full size table

Group differences and correlations

No significant group differences were found between men and women on the ICS total scale or any subscale. Age showed a slight (but significant) negative correlation with the ICS total scale (r = -0.11) and with the subscales Focus on EBP (r = -0.14) and Educational support for EBP (r = -0.12) (see Table 4). This indicates more favorable ICS ratings on these subscales for younger participants.

Participants in postgraduate training more frequently reported working in outpatient services (n = 189) than in inpatient services (n = 141), while licensed psychotherapists’ current occupation was more balanced (n = 26 resp. n = 34; χ2 = 3.99, df = 1, p = 0.046). Since psychotherapists in training are typically younger than licensed psychotherapists (t = -8.12, df = 106.93, p < 0.001), participants working in outpatient services were younger than those working in inpatient services (t = -2.12, df = 358.25, p = 0.035). Therefore, license status and/or current occupation might explain the association between ICS ratings and age. However, the correlation between age and ICS total scale was still significant when controlling for license status (r = -0.13, p = 0.009) and current occupation in in- or outpatient services (r = -0.15, p = 0.002). However, group differences were found for the ICS subscale Educational Support for EBP between psychotherapists in training (M = 3.19, SD = 1.16) and licensed psychotherapists (M = 2.89, SD = 1.19, t = 2.05, df = 131.02, p = 0.032, d = 0.25). Moreover, licensed psychotherapists and psychotherapists in training differed in their ratings for some ICS subscales on inpatient and outpatient services (see Table 8 and 9).

Table 8 Group differences on ICS between in- or outpatient services rated by licensed psychotherapists.
Full size table
Table 9 Group differences on ICS between in- or outpatient services rated by psychotherapists in training.
Full size table

The ICS total scale correlated positively with the EBPAS-36D total scale (r = 0.41, p < .001, n = 404). When controlling for participants’ scores on the EBPAS-36D total scale, the correlation between age and ICS total scale was no longer significant (r = -− .02, p = .639).

Discussion

Based on survey data from a sample of German psychotherapists, our results demonstrated good item properties and internal consistencies as well as factorial and convergent validity for the German version of the ICS.

Item analyses indicate that most item difficulties were in the medium range. This is desirable as it allows optimal differentiating between respondents. Still, three items of the subscale Rewards for EBP on the ICS (items 10, 11, 12) received less endorsement than the other items. The items 10 and 11 assess the provision of financial incentives (bonuses or pay rises) for the use of EBP, item 12 assesses the ability to accumulate compensated time for the use of EBP. In the German mental health care system, financial incentives, bonuses and pay rises or the possibility to accumulate compensated time are rarely employed implementation strategies. Among samples in the USA14,25,26 and Norway30, scores on the Rewards subscale have been very low as well. Engell et al.29 also attributed this to the fact that there are no systematic practices for financial rewards for EBP use in Norwegian child welfare services. Removing item 12 (Ability to accumulate compensated time) and 17 (Selects staff who are flexible) would even result in a slight improvement (0.01) of the internal consistency of the total scale. However, removing these items might affect the subscales’ content validity and the comparability with the original scale so the items were retained in this version. Moreover, these items allow a comparison with other health care systems where such use of incentives is already practiced.

Despite these minor item issues, the internal consistency of the ICS total scale is excellent (ω = 0.91) and comparable to those in previous studies (α = 0.87 to α = 0.89)28,29. The subscale’s internal consistencies ranged from acceptable to excellent, with the Rewards for EBP and the Selection for Openness subscales demonstrating the lowest internal consistencies. Future studies should investigate whether respondents interpret the items of the Rewards for EBP subscale in the same ways in order to rule out misunderstandings. This could be accomplished through cognitive interviewing in order to assess how clinicians perceive and interpret specific items and constructs61,62.

The results indicate factorial validity with adequate model fits for an ICM-CFA, Bifactor CFA and Second-order CFA and even better model fits for an ESEM and a Bifactor ESEM. However, both ESEM models demonstrated less sparsity, non-positive definite variance–covariance matrices and non-significant regression weights. Accordingly, we prefer the Second-order model with six factors that can be regarded as the theoretically assumed model. It demonstrated significant regression weights and an adequate model fit (RMSEA [90% CI] = 0.077 [0.069; 0.085], SRMR = 0.078, CFI = 0.93). Convergent validity of the ICS is supported by confirming the expected high positive correlations with two scales assessing the TPB construct of subjective norms to use EBP. Convergent validity as one type of construct validity supports the assumption of cross-cultural validity of the ICS, insofar that the ICS can be used to assess the Implementation Climate Scale in the German mental health care system despite differences in the health systems and frameworks for the provision of mental health care.

The results of a multiple-group CFA and a multiple-group ESEM allow the assumption of measurement invariance of the ICS across gender groups and across psychotherapists in training and licensed psychotherapists. Only residual invariance (latent means) is not given across licensed psychotherapists and psychotherapists in training. However, this is inconsequential to calculating and interpreting group mean differences when scalar invariance is found63. Therefore, we were able to assess potential group differences. No gender differences were found for the ICS, but respondents’ age showed small negative correlations with the ICS total scale, Focus on EBP and Educational support for EBP subscale—suggesting that the younger a psychotherapist was, the more favorable he or she judged the implementation climate. One potential explanation for the identified age differences might be the type of organization younger psychotherapists predominantly work in. We found that the younger group of psychotherapists in training more frequently work in outpatient services than older and licensed psychotherapists. Moreover, their ratings on inpatient and outpatient clinics’ implementation climate diverges from those of licensed psychotherapists. Besides the type of organization making a difference, psychotherapists in training might differ in their perceptions of the organizations’ implementation climate—or might be treated differently from their licensed colleagues. Future studies should examine these differences in more detail. Still, the correlation between participants’ age and the ICS total scale was significant even after controlling for their license and current occupation, but was no longer significant when controlling for their attitudes towards EBP. In line with previous studies64,65, the organizations’ implementation climate was associated with psychotherapists’ attitudes towards EBP in our study. The higher they rated the implementation climate of their organization or work group, the more positive was their attitude towards EBP. Younger psychotherapists report more positive attitudes towards EBP39 and consider EBP implementation as more important than older therapists. Therefore, the younger psychotherapists are, the more they may be aware of efforts undertaken on the side of the organization or work group to foster EBP implementation.

Some limitations of our study should be considered when interpreting the results. One major limitation is that due to the open nature of our online study and in order prevent concerns of insufficient anonymity, we did not ask participants to disclose the name of their organization or work group. Accordingly, we could not allocate ratings to one or another organization and assess interrater agreement within organizations. Given sufficient within-group interrater agreement, the aggregation of individual-level data to organizational level data would enable to assess the implementation climate construct as the shared characteristic within organizations. Accordingly, future studies should build on our results demonstrating good psychometric properties of the ICS at the individual level to examine its suitability for investigation at the organizational level. Yet, individual perceptions of the implementation climate might well predict EBP use, especially if implementation climate on the organizational level is weak, e.g. there is high within-group variability in the perceptions of the climate66.

Another limitation of our study is that even through advertised widely and via obligatory professional associations, the online sample constitutes a convenience sample. A large proportion of the sample reported being in training to become psychotherapists. Participants predominantly subscribed to a CBT approach and are not representative of the population of mental health providers in Germany: Nübling et al.67 report that among licensed psychotherapists offering mental health care in private practices with reimbursement by the statutory health insurance, therapy approaches are 50.6% PDT, 49.0% CBT, and 25.6% psychoanalysis. In light of the dropout rate in the present study, although in the average range for online surveys68, participants may have been more likely to complete the survey if they were particularly interested in its topic69. Although this does not affect the psychometric investigation of the ICS, future studies should aim to recruit more representative samples.

Conclusions

Organizations’ implementation climate is an important construct in implementation research and implementation efforts benefit from its consideration. The ICS is a pragmatic and psychometrically strong measure assessing employees’ perceived importance of EBP implementation within the organization or work group. Although further validating research is required, the present study confirms good psychometric properties and validity of a German version of the ICS in an online recruited convenience sample of psychotherapists.