Recent studies have shown that online users tend to select information that adheres to their system of beliefs, ignore information that does not, and join groups that share a common narrative. This information environment can elicit tribalism instead of informed debate, especially when issues are controversial. Algorithmic solutions, fact-checking initiatives, and many other approaches have shown limitations in dealing with this phenomenon, and heated debate and polarization still play a pivotal role in online social dynamics (e.g. traditional vs. anti-establishment polarization). To understand the effect of different communication strategies able to smooth polarization, in this paper, together with Corriere della Sera, a major Italian news outlet, we measure the social response of users to different types of news framing. We analyse users’ reactions to 113 ad-hoc articles published on the newspaper’s Facebook page and the corresponding news articles on the topic of migration, published from March to December 2018. We examine different journalistic techniques and content types by analyzing their impact on user comments in terms of toxicity, criticism of the newspaper, and stance concerning migration. We find that visual pieces and factual news reports elicit the highest level of trust in the media source, while opinion pieces and editorials are more likely to be criticized. We also notice that data-driven pieces elicit an extremely low level of trust in the news source. Furthermore, coherently with the echo chambers behaviour, we find social conformity strongly affecting the commenting behaviour of users on Facebook.
Social media such as Facebook and Twitter provide direct access to an unprecedented amount of content. The direct path from content producers to consumers has changed the way users get information, debate, and develop their opinions (Del Vicario et al., 2016). Algorithms mediate and facilitate content promotion, and thus information spreading, by taking into account the preferences and attitudes of users. This paradigm shift profoundly impacts the construction of social perceptions and the framing of narratives, especially when issues are controversial, and it influences policy-making, political communication, and the evolution of public debate. Social media platforms, indeed, are mainly intended for ludic rather than for informative purposes and news consumption on social media may be incidental rather than deliberate. For example, the most liked image on Instagram is an egg (54 million likes).
Although social media content has increased in heterogeneity and quantity, quality has been adversely affected by content monetization and reduced monetary support of news gathering institutions. The result has been a loss of trust in traditional media (Authority for Communications Guarantees, 2018), and a growing interest in investigating the spread of fake news (Del Vicario et al., 2016; Lazer et al., 2018; Vosoughi et al., 2018). Efforts to limit the spread of fake news have ranged from debunking and fact-checking initiatives (Silverman, 2015; The Poynter Institute, 2019), to algorithmic-based solutions (Zhang et al., 2016, 2019; Castillo et al., 2011; Gupta et al., 2013; Kwon et al., 2013).
For example, Facebook is working with independent third-party fact-checking organizations and drawing on user feedback to its News Feed to reduce the prevalence of false news content (Facebook for Media, 2018). However, scientific efforts, both qualitative and quantitative, have produced conflicting definitions, results, and insights (Ruths, 2019), and online debunking attempts appear to be ineffective and may even backfire (Zollo et al., 2017; Swire-Thompson et al., 2020).
Recent studies indicate that although users have access to a large amount of content, they tend to fragment and clump into groups of like-minded individuals sharing a common narrative (Quattrociocchi, 2017; Sunstein, 2018). They enter an “echo chamber”, are driven by confirmation bias as they continue to select content (Del Vicario et al., 2016; Schmidt et al., 2017) (even when the content is false (Bessi et al., 2015)), and ignore any content that calls their beliefs into question (Zollo et al., 2017). Thus, the information pluralism in social media does not produce the positive results (Dutton et al., 2017) expected when individuals are exposed to different points of view (Authority for Communications Guarantees, 2018). Furthermore, exposure to opposing views can even increase polarization (Bail et al., 2018). More powerful than fake news, polarization itself is so pivotal in online social dynamics that it can serve as a useful proxy for the early detection of future misinformation (Del Vicario et al., 2019).
We here examine how news framing affects online social dynamics. We focus on the migration debate currently inflaming Italian public opinion, and we construct an experiment that measures how the differing ways of reporting news impacts user behaviour. We examine Italy’s most read newspaper, Corriere della Sera, analyse how different journalistic techniques and types of content impact user behaviour, and focus on user comments as they invoke toxicity, criticize the newspaper, and promulgate their stance on migration. We collect an original dataset of 113 posts (21,618 comments) and the corresponding news articles on the topic of migration published by Corriere on their Facebook page from March to December 2018. We find that content framing contributes to the modification and re-shaping of existing narratives and fosters the emergence of new narratives, and we find that visual items and news reports draw less criticism than editorial columns and opinion pieces. Data-driven pieces are also mistrusted but are less likely to elicit invective. We also find traces of another powerful force that affects user comments: social conformity.
The remainder of the paper is structured as follows. Section “Materials and methods” describes our dataset and introduces our methodology. Section “Results” presents and discusses our results. Section “Discussion” lists and discusses our conclusions and suggests future lines of possible research.
Materials and methods
We perform our data collection using the Facebook Graph API, which is publicly available and accessible through any personal Facebook user account, and we abide by the terms, conditions, and privacy policies specified for its use, i.e., all the data we download from Facebook pages is publicly available. On 5 February 2018 the Facebook API policy was updated such that information about the users who comment on Facebook content on public pages is only available to the Facebook page owners themselves. Thus, users with privacy restrictions are not included in our dataset, and all the data we use in our project is aggregated and anonymous. The Facebook IDs of the users interacting with the content of the page are anonymized using the hash algorithm SHA-512.
We download data from the Facebook page of the Italian newspaper Corriere della Sera. Dixon et al. (2018) indicates that Corriere is widely read by groups on different sides of the migration debate. The data includes 113 Facebook posts (corresponding to 113 articles published and selected by the journalists of the newspaper) plus all user reactions and comments. We classify the posts that link to articles on the Corriere della Sera website by (i) the journalistic technique (JT) used and (ii) the content type (CT).
We examine the following journalistic techniques (JTs):
News report: Standard objective reporting of what happened or what somebody said.
Article with context: Articles that contextualize an event or situation, e.g., a featured article that goes into more depth on a topic.
Editorial: Articles in which experts or journalists provide their opinion or offer comment on a particular event or situation.
Constructive news: Content that proposes possible solutions to migration issues.
Data-driven pieces: These include fact-checking and analytic pieces.
Human interest—individual: Content that focuses on the experiences of individuals.
Human interest—groups: Content that focuses on groups of people.
Article with pop-culture references: Articles focused on popular personalities such as football players, or actors.
We examine the following content types (CTs):
Article: A standard text with photographs.
Article/infographics: Articles that include figures presenting data.
Video/photo-gallery: Visual content with little or no text.
Multimedia: Usually longer news articles presenting a mixture of videos, photos, and text.
Table 1 provides a detailed breakdown of the data associated with the different journalistic techniques and content types and Table 2 presents a summary of the engagement metrics of the posts by content type and by journalistic technique.
We hired three Italian students to annotate the 22,236 comments on the 113 posts. Some of the comments were replicated in order to compare agreements between annotators with the self-agreement of the annotator. They annotated comments using Goldfinch, an online annotation platform from the Department of Knowledge Technologies of the Jožef Stefan Institute in Ljubljana. Using this platform, they annotated small sets of anonymous random comments without context, i.e., without knowing the post to which the comments were responding. The annotators annotated (i) the commenter’s position on migration, (ii) the presence of toxic language, and (iii) the presence of criticism directed toward the media source. For further details see Supplementary Materials.
We asked the annotators to determine whether each commenter was for or against the migration phenomenon in Italy. A third option was allowed for commenters who were neutral or whose position the annotators could not determine. To remove bias, we assumed that a comment supporting a known pro-migration or anti-migration figure did not automatically indicate the commenter’s position on migration.
We asked the annotators to also determine whether a comment contained vulgar, aggressive, rude, disrespectful, or unreasonable language that could lead other users to abandon the conversation, but we also asked them to disregard comments containing irony or sarcasm.
We also asked the annotators to identify comments criticizing the publisher, journalist, or newspaper serving as the source of information. The annotators were also to disregard irrelevant comments, including spam, empty comments, or those not in Italian.
The comment annotation produced 21,618 relevant annotated comments.
We first examine the relationship between the article news framing typologies (CT and JT) and their online engagement. We do this to determine whether the CTs considered (resp. JTs) differ in terms of engagement. The null hypothesis of the test is that the samples of likes (res. comments, shares) to articles showing different CTs (res. JTs) come from populations with the same distribution (McKight and Najab, 2010). This test is non-parametric and is used to compare different groups using a variable that is an ordinal measurement (Leon, 1998). We observe that the distributions of likes, comments, and shares are usually right-skewed (Zollo et al., 2017; Devineni et al., 2017). Thus, the Kruskal–Wallis test is appropriate because it relaxes the normality assumption typical of similar parametric tests.
To examine the association of news framing with user commenting behaviour, we exploit contingency tables, i.e. tables displaying the multivariate frequency distribution of two variables. In our case, the tables cross-sectionally represent the frequency of each commenting behaviour (toxic or non-toxic, critical or non-critical, anti-migration or non-anti-migration) given the associated articles’ framing typologies (both for what concerns CTs and JTs).
We finally use a set of multivariate logistic regression models (logit models) to examine commenter civility, trust in media sources, and position on migration in relation with our framing categories. Furthermore, we account for the possibility of observing some kind of social conformity in users commenting behaviour. To this respect, each Facebook commenter can potentially read and be influenced by previous comments from other users.
In logistic regressions the probability of observing a binary outcome is predicted in the form of logarithmic odds (Stoltzfus, 2011). Thus, our models indirectly predict the probability of observing the commenting behaviours considered given a set of explanatory variables.
The binary dependent variables we considered are the following:
Toxic, as proxy for comment civility, where Toxici = 1 when comment i is rude and disrespectful.
Critical, as proxy for trust in the news source, where Criticali = 1 when comment i criticizes the news source (Corriere della Sera).
Anti, as proxy for commenter position on the Italian migration issue, where Antii = 1 when comment i indicates that the commenter is against migration.
We take into account as explanatory variables:
CT, the set of CTs as dummy variables (baseline: Article);
JT, the set of JTs as dummy variables (baseline: News report);
per_toxic, where per_toxici is the percentage of toxic comments observed before comment i;
per_critical, where per_criticali the percentage of critical comments observed before comment i;
per_anti, where per_antii is the percentage of anti-migration comments observed before comment i.
As CT and JT are non-mutually exclusive, we alternate these sets of variables in our models. We also alternate per_toxic, per_critical and per_anti coherently with the dependent variables; Table 3 reports in summary the 6 models considered.
In implementing our models, we also take into account the p-value issue addressed in (Lin et al., 2013), given the magnitude of our dataset, and we report the CPS charts in Supplementary Materials that show the sensitivity of p-values to different sample sizes.
News framing and engagement
As a first step, we explore the interplay between news framing and online engagement to understand whether some news pieces may be more appealing than others when treating specific topics. In particular, we focus on articles—written by using different CTs and JTs- dealing with the Italian migration issue.
Table 2 provides summary statistics of article engagement aggregated by CT and JT and the results of the Kruskal–Wallis tests. Simply put, the null hypothesis of the test is that samples of articles, displaying different CTs and JTs, come from populations having the same distribution for what concerns engagement.
All tests fail to reject the null-hypothesis with a significance level α = 0.05. We thus find no clear-cut difference across the CTs and JTs in the distribution of likes, comments, and shares. We thus limit our examination to ranking CTs and JTs by their median engagement values.
Among CTs, multimedia pieces exhibit the highest engagement in terms of likes (214) and comments (132), while articles/infographics are the most shared CT (77.5). In contrast, video/photo-gallery pieces have the lowest aggregated engagement values (likes: 155.5, comments: 51, shares: 38.5).
Among JTs, editorials have the highest number of likes (389), articles with pop-culture references in terms of comments (163), and news reports in terms of shares (74.5). Human interest stories (G), are the least successful JT in terms of likes (78.5), comments (32.5), and shares (21), followed by human interest stories (I) in terms of likes (139), constructive journalism (56) in terms of comments, and focus-on-data pieces in terms of shares (30).
Users seem to prefer news pieces providing combinations of different ways of enjoying the reading. Dealing with the writing style, users preferences seem to be for articles presenting a softer approach to the migration topic, such as editorials or pop-culture pieces.
News framing and commenting behaviour
We proceed assessing the hypothesis that different commenting behaviours may be associated with different news framing typologies related to a fiercely debated topic, as the Italian migration issue. This aspect assumes a preliminary importance as it may offer useful insights to better understand which journalistic formats inhibit or trigger readers reactions. We assess whether the propensity to publish a toxic, critical, or anti-migration comment is different for different CTs and JTs.
Table 4 provides the results of the χ2 and Cramér’s V tests. The χ2 tests reject the null hypothesis of independence with a p-value < 2.2e−16, suggesting a possible interdependence between news framing and each of the commenting behaviours we observed. We use the Cramér V test to further examine the effect size and find that the effect size of this interdependence is small, as in the rule-of-thumb interpretation in (Cohen, 2013). As expected, news framing is not the only, nor the most important, variable associated with single user behaviour because it only factors in the process when a user decides whether to read a news piece (Schmidt et al., 2017) based on their already settled position on the topic (Del Vicario et al., 2016). Nonetheless the relevant, although weak, connection between news framing and commenting behaviour can be associated with audience segmentation in which different users sharing similar features can have similar preferences and exhibit similar behaviours. For example, different JTs can attract the attention of user groups with different propensities to publish toxic, critical, or anti-migration comments.
Notice that JT-based framing is more strongly associated with commenter civility, trust, and anti-migration positions than CT-based framing. This may be because of the differing relevance the two framing typologies may have for users, and a fine-grained framing categorization may better capture the differences in user commenting behaviour.
Framing effect and social conformity
We conclude our analysis examining more thoroughly the interaction between the different commenting behaviours (commenter civility, trust in media sources, and position on migration) and the news framing typologies considered. We also broaden the extent of our analysis taking into account within our models the possibility of observing commenters acting in accordance with social conformity, i.e. the fact that a user can be influenced by the actions of other users. We rely on the set of logistic regressions previously described in this section. Our models allow to observe the individual effects of news framing and social conformity on the probability of observing a given commenting behaviour.
Table 5 lists the parameters of our logistic regression models. Models 1–3 generate the probabilities of finding toxic, critical, or anti-migration comments, respectively, in four different CTs. Note that toxic comments are more prevalent in response to video and photo-gallery items than to plain vanilla articles, and that the opposite is true in response to article/info-graphic pieces. Comments in response to multimedia and video and photo-gallery items are less likely to criticize the source with respect to the baseline, and comments in response to article/info-graphics items are more likely to criticize the source than those to a plain article.
Note that only comments on video and photo-gallery items are less probable of being anti-migration with respect to the baseline.
Models 4–6 generate the probabilities of finding toxic, critical, or anti-migration comments, respectively, in eight different JTs. Note that toxic comments are more prevalent in response to articles with context with respect to news reports, but that the opposite is true in response to focus-on-data pieces. Comments in response to base-line pieces, i.e., news reports, are much less likely to criticize the news source, but comments in response to constructive journalism, human interest stories (G) and focus-on-data CTs are more likely to be critical. Note also that comments in response to articles with pop-culture references and editorials are less likely to be anti-migration than those to news reports.
It is also worth noting that the odds that a comment will be toxic, critical, or anti-migration increase as the percentage of toxic, critical, and anti-migration comments observed by the commenter increases.
Focusing on news framing, we find that the two framing typologies considered, CTs and JTs, are associated with commenting behaviour. Toxic comments are more likely in response to articles containing videos and photos and in those offering a deeper contextual description of the facts but are less likely to appear in response to data and info-graphics articles. Pieces containing videos and multimedia elicit comments with a higher level of trust than data-related articles and opinion pieces, which receive comments with the lowest level of trust in the source. Most of the CTs and JTs tested do not diverge in the way they encourage anti-migration comments. Only articles with pop-culture references and editorials elicits fewer anti-migration comments.
Figure 1 shows the statistical signature of the CT and JT we tested. The radar plots include the average number of comments exhibiting the behaviours we traced and add the pro-migration comments. The statistical signatures of the different CT and JT agree with the results of the models.
Moving to the hypothesis of social conformity, we assess whether commenters access to previous comments influences their commenting behaviour. We find that this is true for all the commenting behaviours considered. We also find that a comment is more likely to be toxic, critical, or anti-migration if the commenter has read previous comments that are toxic, critical, or anti-migration. This is in accord with the academic literature on social influences (Deutsch and Gerard, 1955; Bandura, 1986).
In summary, at least two mechanisms are active when dealing with the users commenting behaviours we considered: news framing and social conformity. On the one hand, a certain role seems to be played by the format and content of a piece of news. Dealing with the Italian migration issue, we find that visual pieces and factual news reports are associated with the highest level of trust in the media source, while editorials and opinion pieces are more subject to criticism. Furthermore, we note that data-driven pieces are associated with a low level of trust in the news source but are also less likely to generate toxic debate. On the other hand, social conformity as an element influencing users’ commenting.
To date the most liked image on Instagram is an egg (54 million likes). Social media platforms, indeed, are mainly intended for ludic rather than for informative purposes and news consumption on social media may be incidental rather than deliberate. Algorithms mediate and facilitate content promotion, and thus information spreading, by taking into account the preferences and attitudes of users. This paradigm shift profoundly impacts the construction of social perceptions and the framing of narratives, especially when issues are controversial, and it influences policy-making, political communication, and the evolution of public debate. Fact-checking initiatives showed limitations in approaching this phenomenon. Indeed, heated debating and polarization still play a pivotal role in online social dynamics (e.g., traditional vs. anti-establishment polarization).
In this paper, we measure the online social response to different journalistic techniques using the Facebook page of the major Italian news outlet Corriere della Sera. Since polarization seems to be one of the main drivers of online social interaction, the purpose of this work is to improve the understanding of the online disintermediated information environment by accounting for the reaction of users to a highly polarizing topic. We focus our analysis on attitudes toward migration, which in recent years has been a highly controversial topic among the Italian public. We collect an original dataset of 113 posts (21,618 comments) and the corresponding news articles divided according to their method of reporting information. We describe how this set of different articles impacts user commenting behaviour in terms of toxicity, criticism of the news source, and stance on migration.
Our findings suggest that visual pieces (among content types) and factual news reports (among journalistic techniques) elicit the highest level of trust in the media source, but that editorials and opinion pieces are more subject to criticism. We note that data-driven pieces elicit a low level of trust in the news source, but less likely to generate toxic debate. We also find another important influence on commenting behaviour: social conformity.
The datasets generated and analysed during the current study are not publicly available in accordance with EU GDPR law, but an anonymized version of the data will be made available from the corresponding author on reasonable request.
Authority for Communications Guarantees (AGCOM) (2018) News vs. fake in the information system. Interim Report: Sector Inquiry “Online platforms and the news system”. https://www.agcom.it/documents/10179/12791486/Allegato+25-1-2019/831ee043-55dd-41e2-b87d-4578016b9989
Bail CA, Argyle LP, Brown TW, Bumpus JP, Chen H, Hunzaker MF, Lee J, Mann M, Merhout F, Volfovsky A (2018) Exposure to opposing views on social media can increase political polarization. Proc Natl Acad Sci USA 115(37):9216–9221
Bandura A (1986) Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall
Bessi A, Coletto M, Davidescu GA, Scala A, Caldarelli G, Quattrociocchi W (2015) Science vs conspiracy: collective narratives in the age of misinformation. PLoS ONE 10(2):e0118093
Castillo C, Mendoza M, Poblete B (2011) Information credibility on twitter. In: Proceedings of the 20th international conference on world wide web. ACM, pp. 675–684
Cohen J (2013) Statistical power analysis for the behavioral sciences. Routledge
Del Vicario M, Bessi A, Zollo F, Petroni F, Scala A, Caldarelli G, Stanley HE, Quattrociocchi W (2016) The spreading of misinformation online. Proc Natl Acad Sci USA 113(3):554–559
Deutsch M, Gerard HB (1955) A study of normative and informational social influences upon individual judgment. J Abnorm Soc Psychol 51(3):629
Devineni P, Koutra D, Faloutsos M, Faloutsos C (2017) Facebook wall posts: a model of user behaviors. Soc Netw Anal Min 7(1):6
Dixon T, Hawkins S, Heijbroek L, Juan-Torres M, Demoures F (2018) Attitudes towards national identity, immigration and refugees in Italy. More in Common
Dutton WH, Reisdorf B, Dubois E, Blank G (2017) Search and politics: the uses and impacts of search in Britain, France, Germany, Italy, Poland, Spain, and the United States. Quello Center Working Paper No. 5-1-17
Facebook for Media (2018) Working to stop misinformation and false news. https://bit.ly/2tf2L5G.
Gupta A, Lamba H, Kumaraguru P, Joshi A (2013) Faking sandy: characterizing and identifying fake images on twitter during hurricane sandy. In: Proceedings of the 22nd international conference on World Wide Web. ACM, pp. 729–736
Kwon S, Cha M, Jung K, Chen W, Wang Y (2013) Prominent features of rumor propagation in online social media. In: Proceedings of the 2013 IEEE 13th international conference on data mining. IEEE, pp. 1103–1108
Lazer DM, Baum MA, Benkler Y, Berinsky AJ, Greenhill KM, Menczer F, Metzger MJ, Nyhan B, Pennycook G, Rothschild D, Schudson M (2018) The science of fake news. Science 359(6380):1094–1096
Leon AC (1998) Descriptive and inferential statistics. In: Schooler NR (ed) Comprehensive clinical psychology, vol. 3. Elsevier Science, pp. 243–285
Lin M, Lucas Jr HC, Shmueli G (2013) Research commentary—too big to fail: large samples and the p-value problem. Inf Syst Res 24(4):906–917
McKight PE, Najab J (2010) Kruskal–Wallis test. In: Weiner IB, Craighead WE (eds) The Corsini Encyclopedia of Psychology, https://doi.org/10.1002/9780470479216.corpsy0491
Quattrociocchi W (2017) Inside the echo chamber. Sci Am 316(4):60–63
Ruths D (2019) The misinformation machine. Science 363(6425):348–348
Schmidt AL, Zollo F, Del Vicario M, Bessi A, Scala A, Caldarelli G, Stanley HE, Quattrociocchi W (2017) Anatomy of news consumption on Facebook. Proc Natl Acad Sci USA 114(12):3035–3039
Silverman C (2015) Emergent: a real-time rumor tracker. www.emergent.info
Stoltzfus JC (2011) Logistic regression: a brief primer. Acad Emerg Med 18(10):1099–1104
Sunstein CR (2018) Is social media good or bad for democracy. SUR-Int’l J Hum Rts 27:83
Swire-Thompson B, Ecker UK, Lewandowsky S, Berinsky AJ (2020) They might be a liar but they’re my liar: source evaluation and the prevalence of misinformation. Pol Psychol 41(1):21–34
The Poynter Institute (2019) Poynter: the international fact checking network. www.poynter.org
Vicario MD, Quattrociocchi W, Scala A, Zollo F (2019) Polarization and fake news: Early warning of Potential misinformation targets. ACM Trans Web (TWEB) 13(2):10
Vosoughi S, Roy D, Aral S (2018) The spread of true and false news online. Science 359(6380):1146–1151
Zhang H, Kuhnle A, Zhang H, Thai MT (2016) Detecting misinformation in online social networks before it is too late. In: Proceedings of the 2016 IEEE/ACM international conference on advances in social networks analysis and mining. IEEE Press, pp. 541–548
Zhang Q, Lipani A, Liang S, Yilmaz E (2019) Reply-aided detection of misinformation via Bayesian deep learning. In: The world wide web conference. ACM, pp. 2333–2343
Zollo F, Bessi A, Del Vicario M, Scala A, Caldarelli G, Shekhtman L, Havlin S, Quattrociocchi W (2017) Debunking in a world of tribes. PLoS ONE 12(7):e0181821
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Schmidt, A.L., Peruzzi, A., Scala, A. et al. Measuring social response to different journalistic techniques on Facebook. Humanit Soc Sci Commun 7, 17 (2020). https://doi.org/10.1057/s41599-020-0507-3