Article | Open | Published:

Metformin transiently inhibits colorectal cancer cell proliferation as a result of either AMPK activation or increased ROS production

Scientific Reportsvolume 7, Article number: 15992 (2017) | Download Citation

Abstract

Metformin is a widely used and well-tolerated anti-diabetic drug that can reduce cancer risk and improve the prognosis of certain malignancies. However, the mechanism underlying its anti-cancer effect is still unclear. We studied the anti-cancer activity of metformin on colorectal cancer (CRC) by using the drug to treat HT29, HCT116 and HCT116 p53−/− CRC cells. Metformin reduced cell proliferation and migration by inducing cell cycle arrest in the G0/G1 phase. This was accompanied by a sharp decrease in the expression of c-Myc and down-regulation of IGF1R. The anti-proliferative action of metformin was mediated by two different mechanisms: AMPK activation and increase in the production of reactive oxygen species, which suppressed the mTOR pathway and its downstream targets S6 and 4EBP1. A reduction in CD44 and LGR5 expression suggested that the drug had an effect on tumour cells with stem characteristics. However, a colony formation assay showed that metformin slowed the cells’ ability to form colonies without arresting cell growth, as confirmed by absence of apoptosis, autophagy or senescence. Our finding that metformin only transiently arrests CRC cell growth suggests that efforts should be made to identify compounds that combined with the biguanide can act synergistically to induce cell death.

Introduction

The methods used for the early diagnosis of colorectal cancer (CRC) are insufficiently sensitive and specific and, despite major advances in surgical techniques and adjuvant treatment, there is still no effective therapy for advanced disease. About 50% of patients respond to the currently available systemic treatments, but almost all develop drug resistance; furthermore, targeted treatments are only effective in patients with a specific molecular profile, and these are still at very high risk of developing resistant mutations. There is therefore growing interest in finding alternative treatments.

Metformin (1,1-dimethylbiguanide hydrochloride) is frequently prescribed to reduce hepatic gluconeogenesis and increase skeletal muscle glucose uptake in patients with type 2 diabetes. It also directly inhibits the growth of various tumour types in vitro and in vivo 1, and seems to reduce the growth of tumour initiating cells or cancer stem cells (CSCs)2. The inhibitory effect of metformin may be mediated by the inactivation of the mammalian target of rapamycin (mTOR), mainly as a result of the activation of adenosine monophosphate-activated protein kinase (AMPK), changes in intracellular reactive oxygen species (ROS) levels3, the inhibition of mitochondrial functions4, or by various other mechanisms that have been associated with the drug1.

Earlier in vitro studies demonstrated that metformin can inhibit the proliferation of CRC cells5, and in vivo studies have shown that metformin delays tumour onset in a mouse model of TP53 mutant CRC6 and inhibits the growth of colon carcinomas stimulated by a high-energy diet7. Consequently, a number of clinical trials are investigating the effect of metformin on CRC in humans. The results of some of these suggest that it has anti-tumour activity and improves overall survival8,9,10, but others have come to opposite conclusions. Tsilidis et al.11 compared metformin with sulfonylurea, and did not find that the biguanide was associated with a reduced cancer risk in a large cohort of patients with newly diagnosed diabetes, and a three-year study of patients with stage II CRC found no significant relationship between metformin treatment and the risk of cancer recurrence or death12.

We analysed the activity of metformin in three CRC cell lines, one with BRAF as driver mutation, the others with KRAS mutation, one of which is an isogenic p53 null form and found for the first time that the drug transiently inhibited their growth and mTOR pathway activation by means of AMPK-dependent and -independent mechanisms.

Results

Metformin reduces cell proliferation and motility

In order to investigate the effects of metformin on CRC, the HT29, HCT116 and HCT116 p53−/− cell lines were treated with 5 mM metformin as reported by Zakikhani et al.5. As shown in Fig. 1a and in Supplementary Fig. S1, MTT (3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyltetrazolium bromide) and BrdU incorporation assays confirmed that this drug concentration inhibited cell growth. The decrease in proliferation (BrdU) after continuous exposure to metformin for 24, 48 and 72 hours was already detectable in all of the cell lines after 24 hours, and became more significant after 72 hours (from 54% to 23% in HT29, from 78% to 44% in HCT116, and from 50% to 26% in HCT116 p53−/− cells).

Figure 1
Figure 1

Metformin (Met) decreased the proliferation, migration, and invasion of HT29, HCT116 and HCT116 p53−/− cells. (a) Cell proliferation was evaluated in vitro by means of BrdU incorporation in the absence (Ctrl) or presence of 5 mM Met after 24, 48 and 72 hours’ treatment. The results are shown as mean values ± SD compared with the control group (**P < 0.01, ****P < 0.0001). (b) The wound healing assay was conducted after Met treatment (0.6 mM for HT29 and HCT116 p53−/−; 1.25 mM for HCT116) for 90 hours (HT29), 38 hours (HCT116) or 40 hours (HCT116 p53−/−). (c) The chamber invasion assay was performed after treatment with 0.6 mM or 1.25 mM Met for 96 hours (HT29) or 72 hours (HCT116 and HCT116 p53−/−).

A modified wound scratch assay was used to assess the effects of metformin on the ability of CRC cells to migrate. Metformin was added at scalar concentrations ranging from 5 to 0.3 mM, derived from the MTT assays and including non-cytostatic doses of the drug (Supplementary Fig. S2). In untreated HT29 cells, wound closure was complete within 90 hours (Fig. 1b); in the presence of 0.6 mM metformin, migration was less and wound closure occurred more than 96 hours after treatment. Untreated HCT116 and HCT116 p53−/− cells migrated more quickly, and the wound was closed in respectively 38 and 40 hours (Fig. 1b and Supplementary Fig. S2); in the presence of 1.25 mM (HCT116 cells) and of 0.6 mM (HCT116 p53−/−) metformin, it took respectively 43 and 45 hours.

Finally, a matrigel chamber invasion assay showed that metformin inhibited tumour invasion in the three cell lines at all the concentrations tested, but it was slower in the HT29 cells (Supplementary Fig. S3). Figure 1c displays results obtained at the same drug concentrations where a delay in migration was observed with the wound healing assay. This finding was supported by the reduction in matrix metalloproteinase 9 (MMP9) mRNA expression13 in the HCT116 and HCT116 p53−/− cells (Supplementary Fig. S1), while HT29 cells do not express MMP914.

Metformin increases the percentage of cells in the G0/G1 phase, reduces the expression of cyclin D1 and c-Myc and the phosphorylation of Rb

In order to investigate the cell mechanisms reducing proliferation, we cytometrically evaluated the changes in cell cycle progression induced by metformin. After 72 hours of treatment, there was a slight accumulation of cells in the G0/G1 phase (from 50% to 63% of HT29 cells, from 49% to 64% of HCT116 cells, and from 36% to 46% of HCT116 p53−/− cells), and a corresponding decrease in the percentage of cells in the G2 phase (from 7.17% to 5.52% of HT29 cells, from 16.02% to 12.69% of HCT116 cells, and from 29.11% to 21.99% of HCT116 p53−/− cells) in comparison with the untreated cells (Fig. 2a).

Figure 2
Figure 2

Metformin (Met) increases the percentage of cells in the G0/G1 phase, and affects the expression of various cell cycle regulatory proteins in HT29, HCT116 and HCT116 p53−/− cells. (a) Flow cytometric analysis of proliferating cells 72 hours after the addition of 5 mM Met. The results are representative of three independent experiments. (b) Immunoblots of p-Rb, cycD1, cycE and c-Myc in cells treated for 72 hours with 5 mM Met or left untreated. β-actin was used as the loading control. Full-size blots are shown in Supplementary Fig. S10. (c) Immunohistochemistry of c-Myc.

Immunoblotting analysis of cyclins D1 and E (which are involved in the G1-S transition) confirmed the increase in the percentage of cells in the G0/G1 phase. Cyclin D1 (cycD1) was significantly down-regulated in all three cell lines, and there was no change in cyclin E (cycE). In line with the inactivation of cycD1, metformin decreased retinoblastoma protein (pRb) phosphorylation (a cyclin D1 downstream target) and c-Myc expression (Figs. 2b and c). Finally, IHC revealed a considerable decrease in histone H3 phosphorylation (P-HH3) staining (Supplementary Fig. S1).

Metformin does not induce apoptosis, autophagy or senescence

In order to investigate the biological mechanisms underlying the metformin-induced inhibition of cell proliferation, we evaluated the drug’s ability to induce CRC cell death by means of apoptosis, autophagy or senescence.

Annexin V assay showed no induction of apoptosis after 72 hours of treatment (Fig. 3a), and this was confirmed by the absence of poly (ADP-ribose) polymerase protein (PARP) and of Caspase-3 cleavage (Fig. 3a) as well as from the Caspase-3 FACS analyses (Supplementary Fig. S4).

Figure 3
Figure 3

Metformin (Met) does not induce apoptosis, autophagy and senescence in HT29, HCT116 and HCT116 p53−/− cells. (a) Graphic representation of the results obtained from Annexin V assay (upper panel) and western blot analysis of Caspase-3 and PARP activation after treatment with 5 mM Met for 72 h (lower panel). Vinculin was used as the loading control. (b) Immunoblotting determination of LC3B and BECN1 protein after treatment with 5 mM Met for 72 hours; Vinculin was used as the loading control. Full-size blots are shown in Supplementary Fig. S11. (c) β-galactosidase staining of untreated cells (−) and cells treated with Met for 72 hours (+). The images were acquired at 20x magnification. At the bottom, human primary thyrocytes carrying the ER:RAS vector were left untreated or treated with 200 nM 4-hydroxytamoxifen (4OHT) for seven days, and used as positive controls (magnification 40x). The data are representative of at least three independent experiments.

Immunoblotting analysis of the light chain 3 isoforms I and II (LC3BI and LC3BII) showed that treatment with metformin did not induce a conversion from the LC3-I to the LC3-II form (Fig. 3b), typical of autophagy. Immunofluorescence for LC3B (Supplementary Fig. S5) and also BECN1 expression did not vary after treatment with metformin in all the cell lines analysed (Fig. 3b), suggesting the absence of autophagy. However, the finding that metformin increased both LC3B isoforms in HCT116 cells might support an induction of autophagy in this cell line as described by Buzzai et al.6. Finally, the fact that β-galactosidase staining showed no differences between untreated and metformin-treated cells of all three cell lines excluded the activation of senescence (Fig. 3c).

Metformin-induced inhibition of proliferation is reversible

To further investigate the absence of induced cell death after metformin treatment, we examined the clonogenic capacity of the three cell lines. Six days’ treatment with metformin significantly reduced the number and size of colonies in comparison with the untreated controls (Supplementary Fig. S6). This was confirmed after 12 days, and was maintained for up to 18 days of treatment (Fig. 4 and Supplementary Fig. S6). The ability to form colonies was least in the two HCT116 cell lines. After drug removal and its replacement with complete medium, the “rescued cells” resumed proliferating at all time points (Fig. 4 and Supplementary Fig. S6). Although this recovery was less marked in the HCT116 p53−/− cell line, this effect does not seem to depend on TP53 status, which is mutated also in HT29 cells. Treatment of “rescued cells” with metformin for 12 days reduced the number and size of the colonies, in comparison with the untreated control and, again, the ability to form colonies was least in the two HCT116 cell lines (Fig. 4). These results are also supported by the finding that the viability and the proliferation of metformin treated cells are comparable to those of rescued cells cultured for 6 days in fresh medium and then treated with metformin for 3, 6 and 12 days (see Supplementary Figs. S7 and S8). Finally, the expression patterns of p-Rb, cycD1, cycE and c-Myc and of untreated and “rescued cells” are similar (Supplementary Fig. S9).

Figure 4
Figure 4

Metformin (Met) reversibly inhibits colony formation. (a) One hundred cells were grown for 12 days in 6-well plates with or without Met (5 mM). The “rescued” colonies (R) are cells treated with Met for 12 days and grown for a further 12 days in fresh complete medium without Met. (b) Graphic representation of the results. The bars indicate the mean value ± SD of three independent experiments (**P < 0.01, ***P < 0.001). (c) One hundred “rescued” cells (R) were grown for 12 days in 6-well plates in presence or absence of Met. (d) Graphic representation of the results. The bars indicate the mean value ± SD of three independent experiments (*P < 0.05, ***P < 0.001).

These findings are in line with the absence of cell death, and suggest that metformin has an inhibitory effect on CRC cell proliferation that can be overcome after drug removal.

Metformin inhibits the mTOR pathway as a result of different mechanisms

At cellular level, metformin can inhibit mitochondrial respiratory complex I (CI)15, which leads to membrane depolarisation16, the release of ROS, and a decrease in the ATP/ADP ratio. The decrease in energy, resulting in AMPK activation, shuts down the mTOR/S6/4EBP1 axis and blocks protein synthesis17. After metformin treatment we observed respectively 3- and 2.5-fold increases in ROS production in the HCT116 and HCT116 p53−/− cells, but none in the HT29 cells (Fig. 5a). FACS analysis and JC-1 staining were used to investigate changes in mitochondria membrane potential. After 72 hours of metformin treatment, there was a significant shift in fluorescence, with a decrease in PE and an increase in Alexa-488 emission, indicating mitochondria depolarisation in the HCT116 (55%) and HCT116 p53−/− cells (65%); in line with the absence of ROS production, this effect was weaker in HT29 cells (28.04%) (Fig. 5b). Metformin activated AMPK by phosphorylation of Thr172 only in the HT29 cells (in which the mitochondria were only marginally impaired by the drug). Despite these differences, metformin inhibited mTOR (Ser2448), ribosomal protein S6 kinase (RPS6K, Thr389) and 4E-binding protein 1 (4EBP1, Thr37/46) in all of the cell lines (Fig. 6).

Figure 5
Figure 5

Metformin (Met) causes oxidative stress and induces mitochondrial depolarization. The cells were treated with 5 mM Met for 72 hours and a significant increase in ROS production was measured in HCT116 and HCT116 p53−/− cell lines. The bars represent the mean value ± SD of three independent experiments (***P < 0.001) (a). In line with ROS increase, HCT116 and HCT116 p53−/− also showed a stronger mitochondrial depolarisation respect to HT29 (b). Mitochondrial depolarization was revealed by decrease of JC-1 PE and increase of JC-1 Alexa-488 signal intensities measured in the related cells. Before harvesting the cells, 1 μl of carbonyl cyanide 3-chlorophenylhydrazone (CCCP) was added as a positive control.

Figure 6
Figure 6

Metformin (Met) reversibly inhibits the mTOR pathway and IGF1R protein. (a) Immunoblots of cells treated with 5 mM Met for 72 hours, or left untreated. The “rescued” cells (R) were grown for 72 hours with Met and for a further 72 hours in fresh complete medium without Met. Vinculin was used as the loading control (in HCT116 and HCT116 p53−/− cells vinculin is the same for AMPK, S6 and 4EBP1 since they were analysed on the same blots). (b) Tables showing densitometric quantification of the western blot bands normalized to vinculin. The ratio of each phosphorylated protein to the corresponding total protein band was calculated and quantified with respect to the untreated sample (−) set to 1.0. The results are representative of three independent experiments. Full-size blots are shown in Supplementary Figs S12S14.

Taken together, these findings indicate that inhibition of the mTOR/S6/4EBP1 pathway in CRC cell lines is only partially mediated by mitochondrial depolarisation and AMPK activation, and suggest that the cellular effects induced by metformin could be at least partially due to the mTOR-mediated reduction in protein synthesis. This view is supported by the recently published finding that inhibition of the mTOR/S6 branch significantly reduces the translation elongation step of protein synthesis in CRC mouse models18.

Interestingly, as previously found in other tumour histotypes19,20, metformin reduced the expression/phosphorylation (Tyr1135/1136) of IGF1R (Fig. 6), the activation of which induces the proliferation, differentiation and transformation of cancer cells21.

It is particularly interesting to note that drug removal reduced the phosphorylation of AMPK in HT29 cells and increased mTOR/S6/4EBP1/IGF1R phosphorylation profile in all “rescued” samples, except for mTOR in HCT116 cells (Fig. 6), thus confirming that the metformin-induced blockade of cell proliferation is transient.

Metformin reduces the expression of CRC stem cell markers

In addition to promoting cell proliferation, c-Myc is highly expressed in cancer cells with stem cell properties22, and the sharp reduction in c-Myc expression after treatment with metformin suggests that the drug has an effect on the stem cell component of CRC. We investigated this possibility by evaluating the expression of the CRC stem cell markers CD44 and LGR5, and found that metformin treatment reduced CD44 mRNA levels in all the cell lines (Fig. 7a, left). The biguanide also reduced LGR5, which was expressed only in HT29 cells23, as observed both by qRT-PCR (Fig. 7a, right) and ISH (Fig. 7b).

Figure 7
Figure 7

Metformin (Met) affects the expression of the CRC stemness markers LGR5 and CD44. (a) qRT-PCR of CD44 and LGR5 in HT29, HCT116 and HCT116 p53−/− cells. (b) In situ hybridisation of LGR5 in HT29 cells. Hematoxylin and eosin (H&E) staining is the same as in Fig. 2.

Discussion

We investigated the in vitro effects of metformin on CRC cells because, although previous studies have shown that metformin inhibits the proliferation of CRC cells5, the drug’s underlying mechanisms of action on CRC are still unclear. We found that metformin was cytostatic and reduced cell motility, but did not induce cell death; furthermore, its effects were reversed after drug removal. As has been observed in other types of cancer24,25,26,27,28,29, the cytostatic effect of metformin on CRC cells is due to a reduction in cell proliferation, an increased accumulation of cells in the G0/G1 phase, a decrease in the expression of proteins controlling cell cycle progression such as cycD1 and c-Myc, and the inhibited phosphorylation of the cyclin D1 target pRb. Metformin reduced cell motility by decreasing cell migration and invasion, thus confirming what has been found in other types of cancer20,30. We also observed a decrease in the mRNA expression of MMP9, a matrix metalloproteinase expressed in CRC31, and a reduction in the mRNA expression of the two established cancer stem cell (CSC) markers CD44 and LGR532,33.

There are no published findings showing that metformin induces apoptosis in CRC, and the absence of cell mortality after treatment with the drug has previously been observed in prostate and breast cancer cells24,34. It has been suggested that metformin may only induce apoptosis when cells are treated almost at confluence35. We worked under low density conditions (5 × 104 cells/cm2) because the drug induced a high degree of acidosis (a decrease in pH from 8 to 6.5) at a higher density (1.5 × 105 cells/cm2), which may interfere with its effects; however, we did not observe any apoptosis even under these conditions (data not shown).

A clonogenic assay showed that the cytostatic effect of metformin was transient and that cells treated with metformin for 18 days were still able to form new colonies in complete medium after drug removal. Moreover, “rescued cells” re-treated with metformin recapitulated the growth arrest. HCT116 p53−/− cells had more difficulty in reverting this transient effect, thus suggesting that metformin is more effective in the absence of TP53 as previously proposed in vivo and in vitro 6. The reversible effect of metformin was highlighted by the reactivation of the mTOR pathway and IGF1R protein after drug removal and three days’ growth in fresh complete medium. Similar results were obtained for p-Rb, cycD1, cycE and c-Myc.

The anti-proliferative action of metformin seems to be mediated by the suppression of the mTOR pathway and IGF1R inactivation36, which both play a critical role in cell growth and are often deregulated in patients with various cancers (including CRC), who have high expression levels of these proteins37. Metformin reduced the phosphorylation of mTOR, its downstream effectors (ribosome protein S6 and 4EBP1) and IGF1R. Interestingly, the inhibition of mTOR was also observed in the HCT116 and HCT116 p53−/− cells in which AMPK was not activated, thus suggesting that the antiproliferative effect of metformin on CRC cells was not only mediated by AMPK. There is evidence that metformin can inhibit mTOR by means of molecular mechanisms that inhibit the phosphoinositide 3-kinase/Akt/mTOR (PI3K/Akt/mTOR) signalling pathway (such as the inactivation of RAG GTPases), or activate the mTOR inhibitor REDD1 (REgulated in Development and DNA damage responses)38,39.

Metformin can also reversibly inhibit complex I of the mitochondrial respiratory chain, an important source of ROS production3. ROS are involved in the chemical damage of cell components that also participate in maintaining cell redox homeostasis and signal transduction pathways. ROS play a complex role in cancer, and can promote survival or apoptosis depending on their concentration and the type of cancer cell40. It has been suggested that metformin stimulates ROS production, thus providing an alternative mechanism to AMPK activation41. We found that ROS production was increased in HCT116 and HCT116 p53−/− cells, but not in HT29 cells, which suggests that metformin may act in these two cell lines by increasing ROS levels and inhibiting mTOR. The finding that an increase in ROS levels induces cell cycle arrest42 is in line with our data, and similar findings have been made in ovary43 and luminal breast cancer44.

The distinctive effects of metformin observed on the cellular models analysed may be due to the different genetic background of the cell lines used in the study. In fact, HCT116 and HCT116 p53−/− cells are KRAS-mutant (G13D), while HT29 cells are BRAF-mutant (V600E). Gain-of-function missense mutations in KRAS and BRAF genes are mutually exclusive CRC drivers and have similar but subtly different roles during initiation and tumor development. Their mechanistic differences in signal transduction could provide distinct differences in response to therapeutic interference45. However, we showed the same effects of metformin, although through different molecular mechanisms, in cells with different genetic driven mutations. These observations support the conclusion of a study investigating the association between metformin use and outcome in patients with stage III CRC in adjuvant chemotherapy who used or not metformin prior to CRC diagnosis, considering also their tumoral KRAS and BRAF status. This study did not find any association between metformin use and disease-free survival, time to recurrence, and overall survival46.

Metformin seems to selectively target the CSCs of a number of cancer types2, and combined treatment with metformin and chemotherapy can partially affect the viability of CSCs in chemo-resistant CRC cell lines47. We found that metformin reduced the RNA expression of two colorectal CSC markers involved in Wnt signalling32,33, CD44 and LGR5, thus confirming that it may affect cancer cells with “stem” characteristics. Interestingly, our findings of mitochondrial depolarisation (which was more marked in HCT116 and HCT116 p53−/− cells) and ROS release are in line with the observation that CSCs are more dependent on mitochondrial respiration respect to non CSCs48.

Metformin is typically administered at doses of 1 g/day, and its median plasma concentration is 330 μM49. The dose used in ours and other in vitro studies of CRC5 is therefore higher than the therapeutic dose. However, metformin accumulates in tissues at concentrations that are several times those found in blood49, and drug concentrations in the gut are higher than in the rest of the body50. In addition, CRC cells express high levels of organic cation transporter 1 (OCT1), the main metformin transporter, which may increase the uptake of metformin51.

Finally, Dowling et al. showed that intracellular accumulation levels of metformin in HCT116 cells were only 10%–15% of the drug present in the medium, and that the lowest dose of metformin required to activate AMPK in cell culture is 1 mM52. These observations may justify testing the use of higher metformin concentrations in the treatment of CRC.

Conclusion

Our findings suggest that metformin can transiently inhibit the growth and proliferation of CRC cells and of their stem fraction by activating AMPK or by promoting oxidative stress, thus suggesting that its use alone is not suitable for cancer treatment. Future studies should consider strategies that synergistically combine metformin with drugs that can benefit from the inhibition of cell growth functions and induce cell death.

Materials and Methods

Cell lines and culture conditions

The HT29 and HCT116 cell lines were obtained from the American Type Culture Collection (ATCC), and the HCT116 p53−/− isogenic cells in which the TP53 gene was inactivated by means of homologous recombination were kindly provided by Professor Bert Vogelstein (Johns Hopkins University, Baltimore, MD, USA). The cells were maintained at 37 °C with 5% CO2, and cultured in McCoy’s 5A (modified) medium, GlutaMAXTM supplement (ThermoFisher Scientific, Inc., Waltham, MA, USA) plus 10% fetal bovine serum (FBS) and 1% penicillin-streptamycin (ThermoFisher Scientific, Inc.).


Chemicals

Metformin (Sigma-Aldrich, St. Louis, MO, USA) was diluted in sterile water to make the 1 M stock solution that was used at a concentration of 5 mM in all of the experiments, with the exception of migration and invasion assays, in which scalar concentrations, ranging from 5 to 0.3 mM were tested.


Immunoblotting

The cells were lysed in Ripa buffer (50 mM Tris, pH 8.0, 50 mM NaCl, 0.5% Triton X-100, 0.1% sodium deoxycholate, 0.25% sodium dodecyl sulphate [SDS]) supplemented with protease inhibitors (Merck Millipore, Billerica, MA, USA) and a phosphatase inhibitor cocktail (Sigma-Aldrich, St. Louis, MO, USA), and their protein content was quantified using the BCA protein assay (ThermoFisher Scientific, Inc.). Cells treated with metformin for 72 hours, left untreated, or grown for 72 hours with metformin and for a further 72 hours in fresh complete medium without the drug (rescued cells) were used. For each sample, 40 μg of protein extract were separated on 4–12% polyacrylamide gels, transferred onto nitrocellulose membranes (Sigma-Aldrich, St. Louis, MO, USA), and incubated with primary antibodies (see Supplementary Table S1). The signals were detected using enhanced chemiluminescence, and protein levels were quantified using ImagelabTM software (Bio-Rad, Hercules, CA, USA). Each experiment was repeated at least three times.

A densitometry scan of the bands was performed to quantify the results. Finally data analysis was done by Image J software (NIH, Bethesda, MD), measuring integrated densities of obtained bands after background subtraction.


Cell proliferation assay by means of 5-bromo-2′-deoxyuridine (BrdU) incorporation

One thousand cells were seeded on 13 mm diameter coverslips (ThermoFisher Scientific, Inc.) and, after 24 hours, metformin was added for 72 hours. The cells were incubated with 33 μM BrdU (Sigma-Aldrich), anti-BrdU primary antibody (BD Biosciences, Franklin Lakes, NJ, USA) and secondary FITC-conjugated antibody (Jackson ImmunoResearch, West Grove, PA, USA). Their nuclei were visualised using 4′, 6-diamidino-2-phenylindole (DAPI; Sigma-Aldrich), and the images were acquired using an upright, automated Olympus BX-61 microscope (Shinjuku, Tokyo, Japan).


Annexin V assay

Cell apoptosis was evaluated by Annexin V assay. The cells were seeded in a 35 mm multi-well plates (2 × 105 cells per well) and, after 24 hours, were treated with metformin for 72 hours. They were then harvested, washed once with Annexin buffer (1 M HEPES, 5 M NaCl, 1 M MgCl2, 1 M CaCl2, 50 mM KCl) and stained with anti-Annexin V-APC conjugated primary antibody (BD Biosciences, Franklin Lakes, NJ, USA). Then the cells were washed once with Annexin buffer, stained with propidium iodide and analysed using a FACSCanto II system (BD Biosciences) and ModFit LT 3.0 software. Three independent experiments were carried out. Cells treated with 20 mM Metformin were used as control of apoptosis induction (Supplementary Fig. S5).


Cell motility assay

Cell migration was evaluated using a modified wound scratch assay. The cells were plated (5 × 104 for HT29; 3 × 104 for HCT116; 4 × 104 for HCT116 p53−/− cells) in a cell culture dish (Ibidi, Martinsried, Germany) and, when cell confluence was approximately 90%, the chambers were removed and the cells were treated with metformin. The images (magnification 10x) were acquired immediately after removing the chambers, and at the complete closure of the gap in the untreated cells using an inverted EVOSmicroscope (Thermofisher Scientific, Inc., Waltham, MA, USA).

Cell invasion was evaluated by plating 3 × 104 cells per well in a 24-well BDBioCoat growth factor-reduced MATRIGEL invasion chamber (BD Biosciences, Franklin Lakes, NJ, USA), and treating them with metformin. Inserts were placed in a well containing 750 µl of normal growth medium with 10% FBS. After 72 hours (HCT116 and HCT116 p53−/− cells) or 96 hours (HT29 cells), the cells in the upper compartment were removed, and the invading cells attached to the bottom side were fixed with cold methanol and stained with Crystal Violet (Sigma-Aldrich). Three independent experiments were carried out.


Cell cycle analysis

Cells were plated in 10 mm Petri dishes (5 × 105 cells/well) and, after 24 hours, were treated with metformin for 72 hours. They were then scraped, fixed in ice-cold 70% ethanol, and stained in accordance with the RNAse/propidium iodide protocol. Cell cycle distribution was analysed using a FACSCanto II system (BD Biosciences) and ModFit LT 3.0 software.


Immunohistochemistry (IHC)

HT29 and HCT116 cells were detached from plates using 1% trypsin, washed twice with PBS, fixed using 10% PBS-buffered formalin, and then inserted in 500 µl of pre-heated (60 °C) 2% agar solution (Cat. No. 50005, Lonza, Basel, Switzerland). After agar solidification, the cells were embedded in paraffin using standard procedures. The IHC experiments were carried out using 3 µm cell block sections. C-Myc (Cat. No. 32072, Abcam, Cambridge, UK) and phospho-histone H3 (Ser10, Cat. No. #9701, Cell Signaling Technology, Danvers, MA, USA) were respectively diluted 1:50 and 1:200, and antigen retrieval and development were carried out using a fully automated BenchMark ULTRA (Ventana Medical Systems, Inc., Tucson, AZ, USA) instrument in accordance with the manufacturer’s instructions.


In situ hybridisation (ISH)

ISH for leucine-rich repeat-containing G-protein coupled receptor 5 (LGR5) RNA expression was carried manually using the RNAScope 2.0 High Definition (Brown) detection kit (Advanced Cell Diagnostics, Inc., Hayward, CA, USA) in accordance with the manufacturer’s instructions, as previously described53.


Senescence-associated (SA) β-galactosidase activity

The cells were seeded in 35 mm multi-well plates (3.75–30 × 104 cells/well) and then treated with metformin for 72 hours. SA-β-galactosidase activity was detected using the Senescence β-galactosidase staining kit (Cell Signaling Technology, Inc.) in accordance with the manufacturer’s instructions. The cell images were acquired under reflected light using a motorised inverted Olympus IX81 microscope. The cells used as positive controls were obtained as previously described54.


Colony-formation assay

The cells were seeded in 35 mm multi-well plates (102 cells/well) and treated with metformin for 6, 12 and 18 days. The medium was refreshed every three days. In order to obtain the rescued cells, the drug was removed and the cells were allowed to grow for 6, 12 and 18 days in fresh medium. The rescued cells were seeded in 35 mm multi-well plates (102 cells/well) and treated with metformin, or left untreated, for 12 days. At the end of the treatments, the plates were stained with Crystal Violet. Only the clearly visible colonies were counted. The experiment was repeated at least three times with three replicates every time.


RNA extraction and quantitative real-time PCR (qRT-PCR)

Total RNA was extracted from the cells using the Trizol reagent (Invitrogen, Carlsbad, CA, USA) in accordance with the manufacturer’s protocol. TaqMan gene expression assays were used for qRT-PCR as previously described55, and the expression revealed by each assay (MMP9 Hs00234579_m1, LGR5 Hs00173664_m1 and CD44 Hs00153304_m1) was normalised to that of GAPDH (Hs99999905_m1). The data were analysed using Sequence Detector, version SDS 2.1.


Reactive oxygen species (ROS) detection assay

The cells were seeded in 96-well microplates (1.5 × 103 cells/well) and, after 24 hours, treated with metformin. Hydrogen peroxide (H2O2) levels were measured using the ROS-GloTM H2O2 assay (Promega, Fitchburg, WI, USA) in accordance with the manufacturer’s instructions. The bioluminescent readings were obtained from three independent experiments using a microplate reader (TECAN Group Ltd., Männedorf, Switzerland).


Mitochondrial membrane potential (Δψm) analysis

The mitochondrial membrane potential of the cells was detected using a MitoProbe™ JC-1 (5,5′,6,6′-tetrachloro-1,1′,3,3′-tetraethylbenzimidazolylcarbocyanine iodide) Assay Kit for flow cytometry (ThermoFisher Scientific, Inc.) in accordance with the manufacturer’s instructions. The cells were treated with metformin for 72 hours, and analysed using a FACSCanto II system. Before harvesting the cells, 1 μl of carbonyl cyanide 3-chlorophenylhydrazone (CCCP) was added as a positive control.


Statistical analysis

The data are presented as mean values ± standard deviation (SD) and statistically compared between groups using one-way analysis of variance followed by Student’s t-test. A P-value of <0.05 was considered statistically significant.

Additional information

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Pierotti, M. A. et al. Targeting metabolism for cancer treatment and prevention: metformin, an old drug with multi-faceted effects. Oncogene 32, 1475–1487, https://doi.org/10.1038/onc.2012.181 (2013).

  2. 2.

    Bednar, F. & Simeone, D. M. Metformin and cancer stem cells: old drug, new targets. Cancer Prev Res (Phila) 5, 351–354, https://doi.org/10.1158/1940-6207.CAPR-12-0026 (2012).

  3. 3.

    Bridges, H. R., Jones, A. J., Pollak, M. N. & Hirst, J. Effects of metformin and other biguanides on oxidative phosphorylation in mitochondria. Biochem J 462, 475–487, https://doi.org/10.1042/BJ20140620 (2014).

  4. 4.

    Salani, B. et al. Metformin impairs glucose consumption and survival in Calu-1 cells by direct inhibition of hexokinase-II. Sci Rep 3, 2070, https://doi.org/10.1038/srep02070 (2013).

  5. 5.

    Zakikhani, M., Dowling, R. J., Sonenberg, N. & Pollak, M. N. The effects of adiponectin and metformin on prostate and colon neoplasia involve activation of AMP-activated protein kinase. Cancer Prev Res (Phila) 1, 369–375, https://doi.org/10.1158/1940-6207.CAPR-08-0081 (2008).

  6. 6.

    Buzzai, M. et al. Systemic treatment with the antidiabetic drug metformin selectively impairs p53-deficient tumor cell growth. Cancer Res 67, 6745–6752, https://doi.org/10.1158/0008-5472.CAN-06-4447 (2007).

  7. 7.

    Algire, C., Amrein, L., Zakikhani, M., Panasci, L. & Pollak, M. Metformin blocks the stimulative effect of a high-energy diet on colon carcinoma growth in vivo and is associated with reduced expression of fatty acid synthase. Endocr Relat Cancer 17, 351–360, https://doi.org/10.1677/ERC-09-0252 (2010).

  8. 8.

    Garrett, C. R. et al. Survival advantage observed with the use of metformin in patients with type II diabetes and colorectal cancer. Br J Cancer 106, 1374–1378, https://doi.org/10.1038/bjc.2012.71 (2012).

  9. 9.

    Lee, J. H. et al. The effects of metformin on the survival of colorectal cancer patients with diabetes mellitus. Int J Cancer 131, 752–759, https://doi.org/10.1002/ijc.26421 (2012).

  10. 10.

    Spillane, S., Bennett, K., Sharp, L. & Barron, T. I. A cohort study of metformin exposure and survival in patients with stage I-III colorectal cancer. Cancer Epidemiol Biomarkers Prev 22, 1364–1373, https://doi.org/10.1158/1055-9965.EPI-13-0347 (2013).

  11. 11.

    Tsilidis, K. K. et al. Metformin does not affect cancer risk: a cohort study in the U.K. Clinical Practice Research Datalink analyzed like an intention-to-treat trial. Diabetes Care 37, 2522–2532, https://doi.org/10.2337/dc14-0584 (2014).

  12. 12.

    Bae, S. et al. Impact of Diabetes Status and Medication on Presentation, Treatment, and Outcome of Stage II Colon Cancer Patients. J Cancer Epidemiol 2015, 189132, https://doi.org/10.1155/2015/189132 (2015).

  13. 13.

    Shuman Moss, L. A. J.-T. S. & Stetler-Stevenson, W. G. Matrix metalloproteinases: changing roles in tumor progression and metastasis. The American journal of pathology 181(6), 1895–1899 (2012).

  14. 14.

    Pedersen, G., Saermark, T., Kirkegaard, T. & Brynskov, J. Spontaneous and cytokine induced expression and activity of matrix metalloproteinases in human colonic epithelium. Clin Exp Immunol 155, 257–265, https://doi.org/10.1111/j.1365-2249.2008.03836.x (2009).

  15. 15.

    Owen, M. R., Doran, E. & Halestrap, A. P. Evidence that metformin exerts its anti-diabetic effects through inhibition of complex 1 of the mitochondrial respiratory chain. Biochem J 348(Pt 3), 607–614 (2000).

  16. 16.

    Andrzejewski, S., Gravel, S. P., Pollak, M. & St-Pierre, J. Metformin directly acts on mitochondria to alter cellular bioenergetics. Cancer Metab 2, 12, https://doi.org/10.1186/2049-3002-2-12 (2014).

  17. 17.

    Dowling, R. J., Zakikhani, M., Fantus, I. G., Pollak, M. & Sonenberg, N. Metformin inhibits mammalian target of rapamycin-dependent translation initiation in breast cancer cells. Cancer Res 67, 10804–10812, https://doi.org/10.1158/0008-5472.CAN-07-2310 (2007).

  18. 18.

    Faller, W. J. et al. mTORC1-mediated translational elongation limits intestinal tumour initiation and growth. Nature 517, 497–500, https://doi.org/10.1038/nature13896 (2015).

  19. 19.

    Li, L. et al. Metformin restores crizotinib sensitivity in crizotinib-resistant human lung cancer cells through inhibition of IGF1-R signaling pathway. Oncotarget 7, 34442–34452, https://doi.org/10.18632/oncotarget.9120 (2016).

  20. 20.

    Kato, H. et al. Metformin inhibits the proliferation of human prostate cancer PC-3 cells via the downregulation of insulin-like growth factor 1 receptor. Biochem Biophys Res Commun 461, 115–121, https://doi.org/10.1016/j.bbrc.2015.03.178 (2015).

  21. 21.

    Sachdev, D. & Yee, D. Disrupting insulin-like growth factor signaling as a potential cancer therapy. Mol Cancer Ther 6, 1–12, https://doi.org/10.1158/1535-7163.MCT-06-0080 (2007).

  22. 22.

    Takahashi, K. & Yamanaka, S. Induction of pluripotent stem cells from mouse embryonic and adult fibroblast cultures by defined factors. Cell 126, 663–676, https://doi.org/10.1016/j.cell.2006.07.024 (2006).

  23. 23.

    Al-Kharusi, M. R. et al. LGR5 promotes survival in human colorectal adenoma cells and is upregulated by PGE2: implications for targeting adenoma stem cells with NSAIDs. Carcinogenesis 34, 1150–1157, https://doi.org/10.1093/carcin/bgt020 (2013).

  24. 24.

    Ben Sahra, I. et al. The antidiabetic drug metformin exerts an antitumoral effect in vitro and in vivo through a decrease of cyclin D1 level. Oncogene 27, 3576–3586, https://doi.org/10.1038/sj.onc.1211024 (2008).

  25. 25.

    Cantrell, L. A. et al. Metformin is a potent inhibitor of endometrial cancer cell proliferation–implications for a novel treatment strategy. Gynecol Oncol 116, 92–98, https://doi.org/10.1016/j.ygyno.2009.09.024 (2010).

  26. 26.

    Gotlieb, W. H. et al. In vitro metformin anti-neoplastic activity in epithelial ovarian cancer. Gynecol Oncol 110, 246–250, https://doi.org/10.1016/j.ygyno.2008.04.008 (2008).

  27. 27.

    Green, A. S. et al. The LKB1/AMPK signaling pathway has tumor suppressor activity in acute myeloid leukemia through the repression of mTOR-dependent oncogenic mRNA translation. Blood 116, 4262–4273, https://doi.org/10.1182/blood-2010-02-269837 (2010).

  28. 28.

    Isakovic, A. et al. Dual antiglioma action of metformin: cell cycle arrest and mitochondria-dependent apoptosis. Cell Mol Life Sci 64, 1290–1302, https://doi.org/10.1007/s00018-007-7080-4 (2007).

  29. 29.

    Memmott, R. M. et al. Metformin prevents tobacco carcinogen–induced lung tumorigenesis. Cancer Prev Res (Phila) 3, 1066–1076, https://doi.org/10.1158/1940-6207.CAPR-10-0055 (2010).

  30. 30.

    Hwang, Y. P. & Jeong, H. G. Metformin blocks migration and invasion of tumour cells by inhibition of matrix metalloproteinase-9 activation through a calcium and protein kinase Calpha-dependent pathway: phorbol-12-myristate-13-acetate-induced/extracellular signal-regulated kinase/activator protein-1. Br J Pharmacol 160, 1195–1211, https://doi.org/10.1111/j.1476-5381.2010.00762.x (2010).

  31. 31.

    Marshall, D. C. et al. Selective Allosteric Inhibition of MMP9 Is Efficacious in Preclinical Models of Ulcerative Colitis and Colorectal Cancer. PLoS One 10, e0127063, https://doi.org/10.1371/journal.pone.0127063 (2015).

  32. 32.

    Sahlberg, S. H., Spiegelberg, D., Glimelius, B., Stenerlow, B. & Nestor, M. Evaluation of cancer stem cell markers CD133, CD44, CD24: association with AKT isoforms and radiation resistance in colon cancer cells. PLoS One 9, e94621, https://doi.org/10.1371/journal.pone.0094621 (2014).

  33. 33.

    Barker, N. et al. Crypt stem cells as the cells-of-origin of intestinal cancer. Nature 457, 608–611, https://doi.org/10.1038/nature07602 (2009).

  34. 34.

    Alimova, I. N. et al. Metformin inhibits breast cancer cell growth, colony formation and induces cell cycle arrest in vitro. Cell Cycle 8, 909–915 (2009).

  35. 35.

    Silvestri, A. et al. Metformin Induces Apoptosis and Downregulates Pyruvate Kinase M2 in Breast Cancer Cells Only When Grown in Nutrient-Poor Conditions. PLoS One 10, e0136250, https://doi.org/10.1371/journal.pone.0136250 (2015).

  36. 36.

    Gwinn, D. M. et al. AMPK phosphorylation of raptor mediates a metabolic checkpoint. Mol Cell 30, 214–226, https://doi.org/10.1016/j.molcel.2008.03.003 (2008).

  37. 37.

    Weber, M. M. et al. Overexpression of the insulin-like growth factor I receptor in human colon carcinomas. Cancer 95, 2086–2095, https://doi.org/10.1002/cncr.10945 (2002).

  38. 38.

    Kalender, A. et al. Metformin, independent of AMPK, inhibits mTORC1 in a rag GTPase-dependent manner. Cell Metab 11, 390–401, https://doi.org/10.1016/j.cmet.2010.03.014 (2010).

  39. 39.

    Ben Sahra, I. et al. Metformin, independent of AMPK, induces mTOR inhibition and cell-cycle arrest through REDD1. Cancer Res 71, 4366–4372, https://doi.org/10.1158/0008-5472.CAN-10-1769 (2011).

  40. 40.

    Biswas, S., Chida, A. S. & Rahman, I. Redox modifications of protein-thiols: emerging roles in cell signaling. Biochem Pharmacol 71, 551–564, https://doi.org/10.1016/j.bcp.2005.10.044 (2006).

  41. 41.

    Mackenzie, R. M. et al. Mitochondrial reactive oxygen species enhance AMP-activated protein kinase activation in the endothelium of patients with coronary artery disease and diabetes. Clin Sci (Lond) 124, 403–411, https://doi.org/10.1042/CS20120239 (2013).

  42. 42.

    Marchi, S. et al. Mitochondria-ros crosstalk in the control of cell death and aging. J Signal Transduct 2012, 329635, https://doi.org/10.1155/2012/329635 (2012).

  43. 43.

    Chan, D. K. & Miskimins, W. K. Metformin and phenethyl isothiocyanate combined treatment in vitro is cytotoxic to ovarian cancer cultures. J Ovarian Res 5, 19, https://doi.org/10.1186/1757-2215-5-19 (2012).

  44. 44.

    Zhang, Y. et al. Involvement of metformin and AMPK in the radioresponse and prognosis of luminal versus basal-like breast cancer treated with radiotherapy. Oncotarget 5, 12936–12949, https://doi.org/10.18632/oncotarget.2683 (2014).

  45. 45.

    Morkel, M., Riemer, P., Blaker, H. & Sers, C. Similar but different: distinct roles for KRAS and BRAF oncogenes in colorectal cancer development and therapy resistance. Oncotarget 6, 20785–20800, https://doi.org/10.18632/oncotarget.4750 (2015).

  46. 46.

    Singh, P. P. et al. Relationship Between Metformin Use and Recurrence and Survival in Patients With Resected Stage III Colon Cancer Receiving Adjuvant Chemotherapy: Results From North Central Cancer Treatment Group N0147 (Alliance). Oncologist 21, 1509–1521, https://doi.org/10.1634/theoncologist.2016-0153 (2016).

  47. 47.

    Zhang, Y. et al. Effects of metformin on CD133+ colorectal cancer cells in diabetic patients. PLoS One 8, e81264, https://doi.org/10.1371/journal.pone.0081264 (2013).

  48. 48.

    Viale, A., Corti, D. & Draetta, G. F. Tumors and mitochondrial respiration: a neglected connection. Cancer Res 75, 3685–3686, https://doi.org/10.1158/0008-5472.CAN-15-0491 (2015).

  49. 49.

    Foretz, M., Guigas, B., Bertrand, L., Pollak, M. & Viollet, B. Metformin: from mechanisms of action to therapies. Cell Metab 20, 953–966, https://doi.org/10.1016/j.cmet.2014.09.018 (2014).

  50. 50.

    Pernicova, I. & Korbonits, M. Metformin–mode of action and clinical implications for diabetes and cancer. Nat Rev Endocrinol 10, 143–156, https://doi.org/10.1038/nrendo.2013.256 (2014).

  51. 51.

    Zhang, S. et al. Organic cation transporters are determinants of oxaliplatin cytotoxicity. Cancer Res 66, 8847–8857, https://doi.org/10.1158/0008-5472.CAN-06-0769 (2006).

  52. 52.

    Dowling, R. J. et al. Metformin Pharmacokinetics in Mouse Tumors: Implications for Human Therapy. Cell Metab 23, 567–568, https://doi.org/10.1016/j.cmet.2016.03.006 (2016).

  53. 53.

    Bozzi, F. M. A. et al. MIF/CD74axis is a target for novel therapies in colon carcinomatosis. Journal of Experimental & Clinical Cancer Research 36(1), 16, https://doi.org/10.1186/s13046-016-0475-z (2017).

  54. 54.

    Vizioli, M. G. et al. Oncogenic RAS-induced senescence in human primary thyrocytes: molecular effectors and inflammatory secretome involved. Oncotarget 5, 8270–8283, https://doi.org/10.18632/oncotarget.2013 (2014).

  55. 55.

    Sokolova, V. et al. The Effects of miR-20a onp21: Two Mechanisms Blocking Growth Arrest in TGF-beta-Responsive Colon Carcinoma. J Cell Physiol 230, 3105–3114, https://doi.org/10.1002/jcp.25051 (2015).

Download references

Acknowledgements

This research was supported in part by a grant [No. 14487] to MA Pierotti from AIRC (Associazione Italiana Ricerca Cancro), by the Italian Ministry of Public Health and by funds obtained through an Italian law that allows taxpayers to allocate 0.5 per cent share of their income tax contribution to a research institution. We thank the staff of the IFOM Imaging Facility and the staff of the IFOM Cell Culture Facility.

Author information

Affiliations

  1. Department of Experimental Oncology and Molecular Medicine, Fondazione IRCCS Istituto Nazionale dei Tumori, via G. Amadeo 42, 20133, Milan, Italy

    • Angela Mogavero
    • , Maria Valeria Maiorana
    • , Susanna Zanutto
    • , Luca Varinelli
    •  & Manuela Gariboldi
  2. Molecular Genetics of Cancer, Fondazione Istituto FIRC di Oncologia Molecolare, via Adamello 16, 20139, Milan, Italy

    • Angela Mogavero
    • , Maria Valeria Maiorana
    • , Susanna Zanutto
    • , Luca Varinelli
    • , Marco A. Pierotti
    •  & Manuela Gariboldi
  3. Department of Diagnostic Pathology and Laboratory Medicine, Fondazione IRCCS Istituto Nazionale dei Tumori, via G. Venezian 1, 20133, Milan, Italy

    • Fabio Bozzi
    • , Antonino Belfiore
    • , Chiara C. Volpi
    •  & Annunziata Gloghini

Authors

  1. Search for Angela Mogavero in:

  2. Search for Maria Valeria Maiorana in:

  3. Search for Susanna Zanutto in:

  4. Search for Luca Varinelli in:

  5. Search for Fabio Bozzi in:

  6. Search for Antonino Belfiore in:

  7. Search for Chiara C. Volpi in:

  8. Search for Annunziata Gloghini in:

  9. Search for Marco A. Pierotti in:

  10. Search for Manuela Gariboldi in:

Contributions

A.M., F.B., A.G., M.A.P. and M.G. conceived and designed the experiments. A.M., M.V.M., S.Z., F.B., A.B. and C.C.V. performed the experiments. A.M., F.B. and M.G. analysed the data and wrote the manuscript. L.V. and S.Z. performed all the experiments suggested by the reviewers. All the authors discussed and approved the manuscript.

Competing Interests

The authors declare that they have no competing interests.

Corresponding author

Correspondence to Manuela Gariboldi.

Electronic supplementary material

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41598-017-16149-z

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.