Ultrasonography of the late-stage knee osteoarthritis prior to total knee arthroplasty: comparison of the ultrasonographic, radiographic and intra-operative findings

Article metrics

  • 515 Accesses

  • 1 Citations

Abstract

The purpose of this study was to assess the effectiveness of the ultrasonography (US) on detecting osteoarthritis of the knee, and compare US and radiographic findings to intraoperative total knee arthroplasty (TKA) findings. Fifty-seven late-stage osteoarthritic knees undergoing TKA were evaluated with US and radiography. Standard knee US assessing femoral cartilage damage, osteophytes, effusion, synovitis, and meniscal extrusion was performed. On radiographs, osteophytes, joint space narrowing, and Kellgren-Lawrence grade were evaluated. Corresponding intra-operative findings were assessed during TKA as the gold standard. On the damage of the medial femoral condyle cartilage, the sensitivity of US was high (92%), whereas on the lateral condyle and sulcus area, sensitivities were 58% and 46%, respectively. On osteophytes, the detection rate of the US was remarkable especially on the medial side yielding sensitivities of 90–95%. The sensitivities for detecting effusion and synovitis were also excellent (97%). US detection rate of femoral cartilage damage was in concordance with the radiographic joint space narrowing. For the detection of osteophytes, US provided superior results to radiography particularly on the medial side. In conclusion, US can reliably assess the late-stage OA changes of the knee especially on the medial side of the knee joint.

Introduction

The prevalence of osteoarthritis (OA) is soaring worldwide with increased age and obesity; currently, the global prevalence is estimated to be 5%1. As the knee OA is a common entity with tremendous sosioeconomical impact with 10–13% patients over 60 years or older suffering from it in the USA1,2, it is of utmost importance to find precise and quick imaging techniques to detect osteoarthritic changes. Ultrasonography (US) of the knee joint is an emerging imaging technique to evaluate the knee OA. To date, clinical examination followed by the knee radiography have been deemed the gold standard for the diagnosis of the OA, but during the last decade the utility of the US on evaluating the knee joint has been studied vigorously3,4. As the radiography offers information preferably on the bone structures, US can be applied to assess the effusion, synovitis, osteophytes, menisci and the femoral cartilage of the knee joint. Moreover, it has been shown that US detects osteophytes more readily than radiographs and that the cartilage defects of the medial femoral condyle correlate well with joint-space narrowing seen on radiographs5,6,7,8. Significant correlation between US-detected and radiographically-detected osteophytes has been also reported8,9,10. Furthermore, excellent inter-observer agreement has been observed for evaluating the osteoarthritic changes of the knee11,12,13. Also, significant association with pain intensity and clinical symptoms has been found8,11,13,14,15. The diagnostic performance of US on detecting osteoarthritic changes has been compared with magnetic resonance imaging (MRI) and arthroscopy16,17: Podlipska et al. reported that osteophytes, cartilage changes in the medial femoral condyle and medial meniscal extrusion can be reliably assessed by US as compared to MRI findings16. Additionally, Saarakkala et al. found that positive findings on US are a strong indicator of arthroscopic degenerative changes, but negative findings do not rule out osteoarthritic changes17. Despite the previous studies, it is still unknown whether the actual weight-bearing joint spaces are visualized on US and how plausible the US findings are. The purpose of this study was to assess US findings on patients with late-stage knee OA undergoing total knee arthroplasty (TKA) and compare US findings with radiographic and intra-operative findings.

Methods

Patients

Fifty-seven patients scheduled for TKA for late-stage OA of the knee were enrolled consecutively in this study during October 2016 and February 2017. Late-stage OA was defined as eligibility for TKA i.e. a combination of typical clinical history and findings for knee OA supplemented with knee radiographs. Written informed consent was obtained from every patient. The mean patient age was 70 years (range 47 to 84) and 28% were males. The study was carried out in accordance with the Declaration of Helsinki and approved by the Ethical Committee of Central Finland Health Care District, Central Finland Hospital (number 6U/2016).

Imaging technique and analysis

Ultrasonography

US imaging was conducted using the GE LOGIC E9 ultrasound device (GE Healthcare, Milwaukee, WI, USA) with 15 MHz linear transducer (type ML6–15). B-mode imaging settings were kept constant for each subject and the focus was set at the level of region of interest. US of the knee was performed by a single radiologist (M.N.) with expertise on musculoskeletal US. The radiologist conducted the acquisition and analysis for the US evaluation, and was blinded to the clinical and radiographic findings. Conventional US technique was applied to assess the knee joint as described previously16,17. First, the knee was scanned with patient in supine position with knee fully extended to evaluate the osteophytes on medial and lateral joint space and the extrusion of the medial and lateral meniscus. Subsequently, the knee was flexed 30° to assess the effusion and synovitis. Doppler imaging was not utilized in this study. Eventually, the knee was flexed as much as possible – typically 90° to 120° – to evaluate the cartilage to femoral sulcus, medial and lateral femoral condyles. The presence and size of osteophytes were evaluated in medial-femoral, medial-tibial, lateral-femoral and lateral-tibial bone margin as follows: Grade 0 = no osteophyte, Grade 1 = marginal/minimal osteophyte, Grade 2 = medium osteophyte and Grade 3 = large osteophyte5. Meniscal extrusion was measured as a perpendicular distance (mm) between the most distant meniscus border and line connecting the femoral and tibial bone ends (measuring below osteophytes if present) and over 4 mm was defined as a sign of extrusion7. Effusion was defined as at least 4 mm thickness of fluid in suprapatellar pouch4. Moreover, synovitis was defined as heterogenous synovial proliferation of at least of over 4 mm of thickness in the suprapatellar pouch and parapatellar recesses4. The femoral cartilage was graded as follows: Grade 0 = normal (a monotonous anechoic band having a sharp hyperechoic anterior and posterior interfaces), Grade 1 = loss of the normal sharpness of cartilage interfaces and/or increased echogenicity of the cartilage, Grade 2 A = in addition to above changes, clear local thinning (less than 50%) of the cartilage, Grade 2B = local thinning (more than 50% but less than 100%) and Grade 3 = total loss of the cartilage17.

Radiography

All patients underwent bilateral weight-bearing postero-anterior radiography on the same day as the US examination. The X-ray beam was 10° caudally angulated and the knee was supported by a frame in 20° flexion and foot in 5° external rotation. The knees were assessed by the same radiologist for osteophytes, joint space narrowing and Kellgren-Lawrence grades18. Osteophytes were graded in medial-femoral, medial-tibial, lateral-femoral and lateral-tibial bone margin as follows: Grade 0 = no osteophyte, Grade 1 = marginal/minimal osteophyte, Grade 2 = a definite osteophyte. Joint spaces (medial and lateral separately) were defined either normal or narrowed. Ultimately, the total Kellgren-Lawrence grade was given for both the medial and the lateral compartment of the knee joint. The reader (same radiologist performing US assessment, M.N.) was blinded to clinical and US findings.

Total knee arthroplasty findings

The TKA operation was performed on average 67 days (range 2 to 181 days) after the US evaluations by 3–5 orthopedic surgeons with at least 10 years of TKA experience. The surgeons were blinded to the US findings, but not to clinical history and radiography findings. The routine TKA protocol was performed using medial parapatellar approach, and the surgical findings were collected as follows: wearing of the cartilage on the femoral sulcus, medial and lateral condyle (normal, marked softening, distinct wearing), osteophytes at medial-femoral, medial-tibial, lateral-femoral and lateral-tibial (yes, no), meniscal extrusion or maseration (yes, no), clinically seen effusion (yes, no) and marked synovial proliferation (yes, no). The grading was kept simple due to several different surgeons performing the TKAs.

Statistical analysis

For statistical analyses, cut-offs were applied to create dichotomous score on certain variables: US-detected cartilage damage was categorized as non-significant (Grades 0 and 1) or significant (Grades 2 A, 2B and 3); US-detected osteophytes as non-significant (Grades 0 and 1) or significant (Grades 2 and 3); radiographically detected osteophytes as non-significant (Grades 0 and 1) or significant (Grade 2); and intraoperative cartilage damage as non-significant (normal) or significant (marked softening, distinct wearing). Data of US and radiography findings are given as numbers of true positive and negative findings according to intraoperative findings. Sensitivity, specificity, accuracy, positive predictive value, and positive and negative likelihood ratios with their 95% confidence intervals were calculated for each finding. The confidence intervals for the first four were calculated using Wilson score method without continuity correction19 and log method20 for the last two. The sensitivities between US and radiography were compared within positive intraoperative findings using Mc-Nemar’s test. P-value < 0.05 was considered statistically significant. SPSS 24.0 was used in analyzing the data.

Results

US versus TKA findings

When comparing the US findings with the intraoperative findings on the 57 knees that underwent TKA, the US examination performed well. For the cartilage degeneration at the femoral medial condyle the sensitivity of the US was 92% and specificity was 50%; moreover, the accuracy was 88% and the positive predictive value was 94%. For the lateral condyle and for the sulcus area, the sensitivities were 58% and 46%; the specificities were 76% and 84%; accuracies were 70% and 67%; and the positive predictive values were 55% and 71%, respectively. Figure 1 depicts an example of the cartilage view on US, radiography and TKA. Concerning the evaluation of the osteophytes, the detection rate of the US was outstanding especially on the medial side: For the femoral medial condyle the sensitivity, specificity, accuracy and positive predictive value were 95%, 50%, 93% and 98%, respectively. For the femoral lateral condyle they were 93%, 27%, 75% and 78%, respectively. For the tibial medial condyle the sensitivity, specificity, accuracy and positive predictive value were 90%, 75%, 88% and 96%, respectively. For the tibial lateral condyle the corresponding values were 65%, 76%, 72% and 59%, respectively. The sensitivities for detecting effusion and synovitis were also excellent yielding a sensitivity of 97% and 97%, respectively. For the damage of the medial and lateral meniscus the sensitivities were 93% and 58%, respectively. The positive likelihood ratios for US findings varied between 1.02 and 3.59, and the negative likelihood ratios between 0.11 and 0.64. Table 1 summarizes the comparison of US and TKA findings.

Figure 1
figure1

The cartilage view on ultrasonography (US), on radiography and during total knee arthroplasty (TKA). On femoral sulcus, axial US-view demonstrates a normal cartilage (white asterisk) without damage (A). On medial femoral condyle, axial parapatellar US-view (patella on the right side, marked as Pat) shows a distinct damage (arrow) on the cartilage (B). The corresponding radiography depicts a clear narrowing (arrow) of the medial joint space (C). Ultimately, the photography of femoral cartilage surfaces taken during the TKA reveals intact femoral sulcus (asterisk) and full-thickness cartilage damage of the medial femoral condyle (arrow) (D).

Table 1 Performance of ultrasonography on detecting osteoarthritic changes of the knee when using the intra-operative findings of total knee arthroplasty as a gold standard.

US versus radiography

When comparing US with radiography – using the TKA findings as the gold standard – the detection rate of cartilage damage was in line with the radiographic joint space narrowing findings: For the medial joint space, the sensitivities of the US versus radiography were 92% and 92%, the specificities 50% and 67%, the accuracies 88% and 89%, and the positive predictive values were 94% and 96%, respectively. For the lateral joint space, the US versus radiography sensitivities were 58% and 42%, specificities 76% and 82%, accuracies 70% and 68%, and the positive predictive values were 55% and 53%. For the detection of osteophytes, the US yielded superior results than the radiography especially on the medial side (Fig. 2): At the femoral medial condyle, the US versus radiography sensitivities were 95% and 44%; at the femoral lateral condyle 93% and 24%; at the tibial medial condyle 90% and 76%; and at the tibial lateral condyle 65% and 70%. Table 2 summarizes the radiography versus intraoperative statistics. Finally, Table 3 demonstrates diagnostic efficiencies of the US and radiography when using the TKA findings as the gold standard.

Figure 2
figure2

The ultrasonography (US) and radiography of the medial compartment of the knee. (A) US image shows a significant osteophyte on the medial-femoral (arrow) and medial-tibial (arrowhead) sites. (B) On the corresponding radiography, no corresponding osteophyte is detected on the medial-femoral site (arrow), whereas only medium osteophyte is seen on the medial-tibial site (arrowhead). The asterisks describe the dislocation of the medial meniscus to the joint space.

Table 2 Performance of radiography on detecting osteoarthritic changes of the knee when using the intra-operative findings of total knee arthroplasty as a gold standard.
Table 3 Ultrasonography (US) findings versus radiography findings when intra-operative total knee arthroplasty findings were used as a gold standard.

Discussion

In this study, we compared the US findings of the late-stage knee OA with the radiographic and intra-operative findings in the TKA procedure. To the best of our knowledge, this is the first study to deploy direct visualization of the knee joint by using TKA as the gold standard for US and radiographic findings. Previously, studies by Saarakkala et al.17 and Koski et al.5 have applied arthroscopy as the gold standard; Saarakkala’s team found that correlation of severity of cartilage damage between US and arthroscopy varied from insignificant to significant depending on the site: at the sulcus area the correlation was highest and at the medial condyle also significant, but at the lateral condyle insignificant. They concluded that a positive finding on US is a potent indicator of arthroscopic degenerative changes of cartilage, but a negative finding does not rule out degenerative changes17. Subsequently, Koski et al. showed a significant correlation between US detected osteophytes and the degenerative cartilage changes at arthroscopy at the medial compartment5. However, neither of these studies comprehensively compared all the US findings (effusion, synovitis, osteophytes, cartilage damage and meniscal pathology) to the macroscopic intra-operative findings; for this purpose, the direct visualization of the joint during TKA indubitably offers the best gold standard. In concordance with previous studies5,17, our study confirms that US performs better on the medial compartment of the knee. Using the MRI as a gold standard, Podlipska et al. reported that the ability of US to detect medial cartilage damage was good, whereas for the lateral femoral condyle it was only fair16. The superior performance at the medial aspect of the knee may be due to the better acoustic window than on the lateral side. Moreover, the bony contour is usually more explicit on the medial edge.

With regard to detection of osteophytes, this is the first study to compare US and radiographic findings to the actual macroscopic findings. Here, US outranked radiography in every compartment (medial-femoral, lateral-femoral, medial-tibial and lateral-tibial) when TKA findings were used as a gold standard, but statistically significant difference was seen only on femoral osteophytes. Furthermore, likelihood ratios were interpreted as described by McGee (2002): Positive and negative likelihood ratios show how the probability of the OA changes when the finding is present or absent, respectively. For instance, positive likelihood ratios of 2.0 and 3.0 increase the probability of OA by 15% and 20%, respectively, whereas negative likelihood ratios of 0.2 and 0.5 decrease the probability of OA by 30% and 15%, respectively. Likelihood ratio of 1 reflects lack of diagnostic value. However, it should be reminded that likelihood ratios are calculated using logarithms (i.e. they are not linear) and therefore meticulous interpretation is recommended21. Results similar to ours have been also shown by Podlipska et al., who found the diagnostic performance of US to detect any osteophytes in the medial and lateral femur and tibia was excellent to good when using MRI as a gold standard16. Moreover, Koski et al. stated that US detected more osteophytes than radiography at both the medial (65% vs. 48%) and lateral compartments (70% vs. 60%); significant statistical correlation was found between US and radiography at the medial side, but only low correlation at the lateral side5. Taken together, our results confirm the recent studies suggesting that US is more sensitive in the detection of osteophytes than radiography5,6,8,16. Previously, good correlation between US findings and radiographic severity (Kellgren Lawrence grade) has been demonstrated8,9,10 with preference on the medial aspect of the knee6. Our results here again confirm this as US-detected cartilage damage correlated well with radiographically-detected joint-space narrowing and Kellgren Lawrence grade.

There are some limitations in this study. First, the high number of osteoarthritic findings – the patients representing late-stage knee OA scheduled for TKA – creates bias to this study. This reflects mostly as the low specificity obtained by the US examination as almost every patient had a positive finding. However, this could not have been avoided, since we wanted to use the direct visualization of the knee joint during TKA as the ultimate gold standard here. Second, the time from the US examination to the TKA operation varied and therefore the inflammatory synovial changes and especially the effusion could have changed. Third, the flexion angle of the knee was not standardized leading to better visualization of the femoral cartilage on some patients; however sufficient acoustic window with at least 90° flexion was obtained with every patient. Fourth, the high BMI of few patients weakened the diagnostic US window. Finally, the relatively large number of operating orthopedic surgeons induced variation to the classification of the TKA findings; accordingly, the intra-operative grading was kept as simple and explicit as possible. Moreover, the surgeons were not blinded to the radiography findings, which could have created bias on the intra-operative classification.

In conclusion, US can be used reliably to evaluate the late-stage OA changes of the knee particularly on the medial side. Our study shows outstanding sensitivities for effusion, synovitis, osteophytes, cartilage damage and meniscal pathology. Moreover, US yields superior detection of osteoarthritic changes as compared to radiographs.

Data Availability

The datasets generated during and analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. 1.

    Osteoarthritis Research Society International. Osteoarthritis: A Serious Disease. Osteoarthritis Research Society International p. 1–103 (2016).

  2. 2.

    Zhang, Y. & Jordan, J. M. Epidemiology of Osteoarthritis. Clinics in geriatric medicine. 26(3), 355–369, https://doi.org/10.1016/j.cger.2010.03.001 (2010).

  3. 3.

    Keen, H. I., Wakefield, R. J. & Conaghan, P. G. A systematic review of ultrasonography in osteoarthritis. Ann Rheum Dis. 68, 611–619 (2009).

  4. 4.

    Oo, W. M. & Bo, M. T. Role of Ultrasonography in Knee Osteoarthritis. J Clin Rheumatol. 22(6), 324–9, https://doi.org/10.1097/RHU.0000000000000436 Review. PubMed PMID: 27556241 (2016 Sep).

  5. 5.

    Koski, J. M. et al. Atlas-based knee osteophyte assessment with ultrasonography and radiography: relationship to arthroscopic degeneration of articular cartilage. Scand J Rheumatol. 45(2), 158–64, https://doi.org/10.3109/03009742.2015.1055797. Epub 2015 Aug 31. PubMed PMID: 26324797 (2016).

  6. 6.

    Okano, T. et al. Ultrasonographic evaluation of joint damage in knee osteoarthritis: feature-specific comparisons with conventional radiography. Rheumatology (Oxford). Nov; 55(11), 2040–2049. Epub 2016 Aug 24. PubMed PMID: 27558583 (2016).

  7. 7.

    Podlipská, J. et al. Comparison of Diagnostic Performance of Semi-Quantitative Knee Ultrasound and Knee Radiography with MRI: Oulu Knee Osteoarthritis Study. Sci Rep. Mar 1, 6, 22365, https://doi.org/10.1038/srep22365. PubMed PMID: 26926836; PubMed Central PMCID: PMC4772126 (2016).

  8. 8.

    Serban, O. et al. Pain in bilateral knee osteoarthritis - correlations between clinical examination, radiological, and ultrasonographical findings. Med Ultrason. Sep; 18(3), 318–25, https://doi.org/10.11152/mu.2013.2066.183.pin. PubMed PMID: 27622408 (2016).

  9. 9.

    Hall, M. et al. Synovial pathology detected on ultrasound correlates with the severity of radiographic knee osteoarthritis more than with symptoms. Osteoarthritis Cartilage. Oct; 22(10), 1627–33, https://doi.org/10.1016/j.joca.2014.05.025. PubMed PMID: 25278071; PubMed Central PMCID: PMC4192137 (2014).

  10. 10.

    Riecke, B. F. et al. An ultrasound score for knee osteoarthritis: a cross-sectional validation study. Osteoarthritis Cartilage. 22(10), 1675–91, https://doi.org/10.1016/j.joca.2014.06.020. PubMed PMID: 25278077 (2014 Oct).

  11. 11.

    Iagnocco, A. et al. Ultrasound imaging for the rheumatologist XXIX. Sonographic assessment of the knee in patients with osteoarthritis. Clin Exp Rheumatol. Sep-Oct; 28(5), 643–6. Epub 2010 Oct 28. PubMed PMID: 21029564 (2010).

  12. 12.

    Abraham, A. M., Goff, I., Pearce, M. S., Francis, R. M. & Birrell, F. Reliability and validity of ultrasound imaging of features of knee osteoarthritis in the community. BMC Musculoskelet Disord. Apr 6; 12, 70, https://doi.org/10.1186/1471-2474-12-70. PubMed PMID: 21470410; PubMed Central PMCID: PMC3079707 (2011).

  13. 13.

    Razek, A. A. & El-Basyouni, S. R. Ultrasound of knee osteoarthritis: interobserver agreement and correlation with Western Ontario and McMaster Universities Osteoarthritis. Clin Rheumatol. 35(4), 997–1001, https://doi.org/10.1007/s10067-015-2990-2. Epub 2015 Jun 20. PubMed PMID: 26089198 (2016 Apr).

  14. 14.

    Chen, Y. J. et al. Association between the severity of femoral condylar cartilage erosion related to knee osteoarthritis by ultrasonographic evaluation and the clinical symptoms and functions. Arch Phys Med Rehabil. 96(5), 837–44, https://doi.org/10.1016/j.apmr.2015.01.004. Epub 2015 (2015 May).

  15. 15.

    Bruyn, G. A. et al. Ultrasound Task Force. An OMERACT reliability exercise of inflammatory and structural abnormalities in patients with knee osteoarthritis using ultrasound assessment. Ann Rheum Dis. 75(5), 842-6, https://doi.org/10.1136/annrheumdis-2014-206774. Epub 2015 Apr 22. PubMed PMID: 25902788. Jan 13. PubMed PMID: 25596002 (2016 May).

  16. 16.

    Podlipská, J. et al. Structure-symptom relationship with wide-area ultrasound scanning of knee osteoarthritis. Sci Rep. 7, 44470, https://doi.org/10.1038/srep44470. PubMed PMID: 28295049; PubMed Central PMCID: PMC5353612 (2017 Mar 15).

  17. 17.

    Saarakkala, S. et al. Diagnostic performance of knee ultrasonography for detecting degenerative changes of articular cartilage. Osteoarthritis Cartilage. 20(5), 376-81, https://doi.org/10.1016/j.joca.2012.01.016. Epub 2012 Feb 1. PubMed PMID: 22343004 (2012 May).

  18. 18.

    Kellgren, J. H. & Lawrence, J. S. Radiological assessment of osteo-arthrosis. Ann Rheum Dis. 16(4), 494–502. PubMed PMID: 13498604; PubMed Central PMCID: PMC1006995 (1957 Dec).

  19. 19.

    Newcombe, R. G. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med. 17(8), 857–72. PubMed PMID:9595616 (1998 Apr 30).

  20. 20.

    Altman, D. G. Diagnostic tests. In Statistics with Confidence, 2nd edition (eds Altman, D. G., Machin, D., Bryant, T. & Gardner, M.) 105–119 (BMJ Books, Bristol, 2000).

  21. 21.

    McGee, S. Simplifying likelihood ratios. J Gen Intern Med. 17(8), 646–9. PubMed PMID: 12213147 (2002 Aug).

Download references

Acknowledgements

The authors wish to thank sonographers Ismo Salmistola and Marjukka Liipo for their assistance for the recruitment of the patients. Additional acknowledgements are given to the chief of radiology at Central Finland Central Hospital, Dr. Ville Mäki for providing support and facilities for this research.

Author information

M.N. Conception and design, collection and assembly of data, analysis and interpretation of the data, statistical expertise, drafting the article, critical revision of the article for important intellectual content, final approval of the article. K.K. Conception and design, analysis and interpretation of the data, drafting the article, critical revision of the article for important intellectual content, final approval of the article. J.P. Conception and design, collection and assembly of data, critical revision of the article for important intellectual content, final approval of the article. K.P. Conception and design, collection and assembly of data, critical revision of the article for important intellectual content, final approval of the article. M.P. Conception and design, collection and assembly of data, critical revision of the article for important intellectual content, final approval of the article. M.H. Analysis and interpretation of the data, statistical expertise, critical revision of the article for important intellectual content, final approval of the article. J.K. Conception and design, critical revision of the article for important intellectual content, final approval of the article. S.S. Conception and design, analysis and interpretation of the data, critical revision of the article for important intellectual content, final approval of the article.

Correspondence to Mika T. Nevalainen.

Ethics declarations

Competing Interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.