Commentary

For the primary research question of the systematic review by Yi et al.1 that was appraised in this commentary, the authors assessed the differences in success rates between self-drilling and self-tapping (pre-drilling) orthodontic mini-implants. For the secondary question they assessed differences in the rates of implant-root contact between these insertion methods in the studies that were found eligible for the first research question. In appraising this review we (RMR, GC) independently scored the methodological validity and risk of bias for the primary research question with respectively the AMSTAR2,3 and ROBIS tools.4,5

Complete agreement on the AMSTAR and ROBIS scores was reached through discussions. We summarised these outcomes in Tables 1 and 2 and presented the rationale for these scores in the section ‘Limitations of this systematic review’.

Table 1 AMSTAR scores for the systematic review by Yi et al.1
Full size table
Table 2 Tabular presentation for ROBIS results for the systematic review by Yi et al1
Full size table

Limitations of the systematic review

When defining primary outcomes, the authors of this systematic review focused on the success rates of two different interventions for inserting orthodontic mini-implants, but they should also have included at least one adverse effect of these interventions as a primary outcome.6 This is mandatory for research questions of Cochrane systematic reviews7 and is an important issue, because implants could for example damage, ie penetrate, dental roots during the insertion process and this damage might differ between the two insertion techniques.8

The authors did not register or publish their protocol in PROSPERO9 or in open access institutional repositories10-12 or at The Clinical Trial Register at the International Clinical Trials Registry Platform of the World Health Organization.13 Not publishing or registering review protocols is a serious problem, because it could introduce bias in the selection of the reported results.14 The absence of a protocol also made it impossible to verify various items that were under-reported or missing.15

The eligibility criteria were not described in sufficient detail. It was also unclear whether these criteria were specified in a protocol or adopted post-hoc. Specified time-points for measuring outcomes were not defined. This is an important eligibility criterion because short-term implant success, eg, as less than 120 days of force application is not clinically relevant, because most orthodontic objectives cannot be completed in such short time -frames. Eligible definitions of success/failure of orthodontic mini-implants were also not presented, and references of excluded studies were not given.

The authors scored all six eligible studies as ‘high quality’ using the Newcastle Ottawa scale,16 but did not consider the shortcomings of this scale. These limitations refer to: (1) a summary quality score of ‘high quality’ was assigned to a study when five or more of nine quality items were scored as ‘high quality’. However, scoring overall value judgments should be discouraged, because certain limitations could be more critical than others and they could vary for different settings.17 For example, two studies scored a five and one a six for the nine quality items of the NOS. Patient selection, confounding, blinding, selective reporting and lost to follow-up was either not controlled or reported in some or all of the selected studies and could be critical measures for the summary quality score; (2) the NOS has important limitations such as an insufficient guidance on how to use this instrument,18 as was evidenced in a variety of research studies that found low reliability between reviewers that used this scale.19,20,21 The Risk of Bias In Non-randomised Studies of Interventions ROBINS-I tool22 developed by the Cochrane Collaboration and formerly known as the ACROBAT-NRSI tool23 would have been our preferred instrument for assessing the validity of the non-randomised eligible studies in this systematic review.

The authors summarised the primary effect sizes from the six eligible studies in a meta-analysis. We judged that conducting a meta-analysis was not appropriate in this systematic review because of the various bias issues, but above all because different outcome measures were synthesised. For example, the time points of measuring implant success varied significantly between eligible studies, were too short in several studies (1, 2, or 6 weeks after implant insertion) or were not specified at all. The authors should have aimed a priori for one of 2 options: (1) apply broad-spectrum eligibility criteria, present the different effect sizes in a forest plot to display the dispersion of outcomes, and not undertake a meta-analysis (Fig. 1);24 or (2) apply narrow-spectrum eligibility criteria for a well-defined time period for measuring outcomes, eg, six months after orthodontic force application, and conduct a meta-analysis when additional criteria (such as low risk of bias) for undertaking such an analysis are met.25,26

Figure 1
figure 1

Success rates of self-drilling versus self-tapping orthodontic mini-implants**In this forest plot the outcomes of the individual studies are presented, but they are not summarised in a meta-analysis.

Full size image

Some important shortcomings in using and interpreting the statistics were also identified: (1) the authors planned to undertake a meta-analysis with a random-effects model when the ratio of true heterogeneity to total heterogeneity (I2) was moderate (25%<I2<50%) or high (I2>50%). However, they did not adhere to this plan, because they calculated an I2 of 41% and conducted a fixed effect model meta-analysis instead; (2) this choice is further problematic, because the choice between using a fixed-effect or random-effects model should not be based on the statistical test for heterogeneity. Actually the random-effects model should have been the logical starting point, because it is not realistic to assume that the true effect size would be the same for all eligible studies in this review;2; and (3) The authors of this systematic review self-defined specific thresholds for classifying moderate and high heterogeneity. However, the statisticians who first published the I2 warned against using thresholds for this statistic and presented a rough guide for the interpretation of I2, which is not congruent with the values proposed by the authors of this systematic review.26,28

Conclusions

The validity of the primary outcomes of this systematic review should be considered in the context of its numerous limitations. Replicating or updating of this review will be difficult, because of these shortcomings. Adverse effects of interventions were also not assessed. However, not only the authors of this review are responsible for its limitations, but also its peer-reviewers and editors as well as the methodologists who have launched the NOS without sufficient validation and guidance.

The secondary outcomes on the rates of implant-root contact of self-drilling and self-tapping insertion techniques for orthodontic mini-implants also have low validity, because only articles that were eligible for the primary research question, ie, just two studies, were examined for these outcomes, and definitions of implant-root contact, ie touching or penetration of the root, were not given.8

Practice point

  • Because of the numerous limitations in this systematic review, we have no confidence that the success rates of self-drilling and self-tapping insertion techniques of orthodontic mini-implants are similar. We also have no confidence that the rate of implant-root contact is similar for both techniques.

  • Because of the shortcomings and because adverse effects of interventions were not assessed, we believe that practitioners should not consider the findings of this systematic review until better evidence-based knowledge on this research topic will be created.