Measuring social response to different journalistic techniques on Facebook

Recent studies have shown that online users tend to select information that adheres to their system of beliefs, ignore information that does not, and join groups that share a common narrative. This information environment can elicit tribalism instead of informed debate, especially when issues are controversial. Algorithmic solutions, fact-checking initiatives, and many other approaches have shown limitations in dealing with this phenomenon, and heated debate and polarization still play a pivotal role in online social dynamics (e.g. traditional vs. anti-establishment polarization). To understand the effect of different communication strategies able to smooth polarization, in this paper, together with Corriere della Sera , a major Italian news outlet, we measure the social response of users to different types of news framing. We analyse users ’ reactions to 113 ad-hoc articles published on the newspaper ’ s Facebook page and the corresponding news articles on the topic of migration, published from March to December 2018. We examine different journalistic techniques and content types by analyzing their impact on user comments in terms of toxicity, criticism of the newspaper, and stance concerning migration. We ﬁ nd that visual pieces and factual news reports elicit the highest level of trust in the media source, while opinion pieces and editorials are more likely to be criticized. We also notice that data-driven pieces elicit an extremely low level of trust in the news source. Furthermore, coherently with the echo chambers behaviour, we ﬁ nd social conformity strongly affecting the commenting behaviour of users on Facebook.


S
ocial media such as Facebook and Twitter provide direct access to an unprecedented amount of content.The direct path from content producers to consumers has changed the way users get information, debate, and develop their opinions (Del Vicario et al., 2016).Algorithms mediate and facilitate content promotion, and thus information spreading, by taking into account the preferences and attitudes of users.This paradigm shift profoundly impacts the construction of social perceptions and the framing of narratives, especially when issues are controversial, and it influences policy-making, political communication, and the evolution of public debate.Social media platforms, indeed, are mainly intended for ludic rather than for informative purposes and news consumption on social media may be incidental rather than deliberate.For example, the most liked image on Instagram is an egg (54 million likes).
Although social media content has increased in heterogeneity and quantity, quality has been adversely affected by content monetization and reduced monetary support of news gathering institutions.The result has been a loss of trust in traditional media (Authority for Communications Guarantees, 2018), and a growing interest in investigating the spread of fake news (Del Vicario et al., 2016;Lazer et al., 2018;Vosoughi et al., 2018).Efforts to limit the spread of fake news have ranged from debunking and factchecking initiatives (Silverman, 2015;The Poynter Institute, 2019), to algorithmic-based solutions (Zhang et al., 2016(Zhang et al., , 2019;;Castillo et al., 2011;Gupta et al., 2013;Kwon et al., 2013).
For example, Facebook is working with independent thirdparty fact-checking organizations and drawing on user feedback to its News Feed to reduce the prevalence of false news content (Facebook for Media, 2018).However, scientific efforts, both qualitative and quantitative, have produced conflicting definitions, results, and insights (Ruths, 2019), and online debunking attempts appear to be ineffective and may even backfire (Zollo et al., 2017;Swire-Thompson et al., 2020).
Recent studies indicate that although users have access to a large amount of content, they tend to fragment and clump into groups of like-minded individuals sharing a common narrative (Quattrociocchi, 2017;Sunstein, 2018).They enter an "echo chamber", are driven by confirmation bias as they continue to select content (Del Vicario et al., 2016;Schmidt et al., 2017) (even when the content is false (Bessi et al., 2015)), and ignore any content that calls their beliefs into question (Zollo et al., 2017).Thus, the information pluralism in social media does not produce the positive results (Dutton et al., 2017) expected when individuals are exposed to different points of view (Authority for Communications Guarantees, 2018).Furthermore, exposure to opposing views can even increase polarization (Bail et al., 2018).More powerful than fake news, polarization itself is so pivotal in online social dynamics that it can serve as a useful proxy for the early detection of future misinformation (Del Vicario et al., 2019).
We here examine how news framing affects online social dynamics.We focus on the migration debate currently inflaming Italian public opinion, and we construct an experiment that measures how the differing ways of reporting news impacts user behaviour.We examine Italy's most read newspaper, Corriere della Sera, analyse how different journalistic techniques and types of content impact user behaviour, and focus on user comments as they invoke toxicity, criticize the newspaper, and promulgate their stance on migration.We collect an original dataset of 113 posts (21,618 comments) and the corresponding news articles on the topic of migration published by Corriere on their Facebook page from March to December 2018.We find that content framing contributes to the modification and re-shaping of existing narratives and fosters the emergence of new narratives, and we find that visual items and news reports draw less criticism than editorial columns and opinion pieces.Data-driven pieces are also mistrusted but are less likely to elicit invective.We also find traces of another powerful force that affects user comments: social conformity.
The remainder of the paper is structured as follows.Section "Materials and methods" describes our dataset and introduces our methodology.Section "Results" presents and discusses our results.Section "Discussion" lists and discusses our conclusions and suggests future lines of possible research.

Materials and methods
Ethics statement.We perform our data collection using the Facebook Graph API, which is publicly available and accessible through any personal Facebook user account, and we abide by the terms, conditions, and privacy policies specified for its use, i.e., all the data we download from Facebook pages is publicly available.On 5 February 2018 the Facebook API policy was updated such that information about the users who comment on Facebook content on public pages is only available to the Facebook page owners themselves.Thus, users with privacy restrictions are not included in our dataset, and all the data we use in our project is aggregated and anonymous.The Facebook IDs of the users interacting with the content of the page are anonymized using the hash algorithm SHA-512.
Data description.We download data from the Facebook page of the Italian newspaper Corriere della Sera.Dixon et al. (2018) indicates that Corriere is widely read by groups on different sides of the migration debate.The data includes 113 Facebook posts (corresponding to 113 articles published and selected by the journalists of the newspaper) plus all user reactions and comments.We classify the posts that link to articles on the Corriere della Sera website by (i) the journalistic technique (JT) used and (ii) the content type (CT).
We examine the following journalistic techniques (JTs): 1. Comment annotation.We hired three Italian students to annotate the 22,236 comments on the 113 posts.Some of the comments were replicated in order to compare agreements between annotators with the self-agreement of the annotator.They annotated comments using Goldfinch, an online annotation platform from the Department of Knowledge Technologies of the Jožef Stefan Institute in Ljubljana.Using this platform, they annotated small sets of anonymous random comments without context, i.e., without knowing the post to which the comments were responding.The annotators annotated (i) the commenter's position on migration, (ii) the presence of toxic language, and (iii) the presence of criticism directed toward the media source.For further details see Supplementary Materials.
We asked the annotators to determine whether each commenter was for or against the migration phenomenon in Italy.A third option was allowed for commenters who were neutral or whose position the annotators could not determine.To remove bias, we assumed that a comment supporting a known pro-migration or anti-migration figure did not automatically indicate the commenter's position on migration.
We asked the annotators to also determine whether a comment contained vulgar, aggressive, rude, disrespectful, or unreasonable language that could lead other users to abandon the conversation, but we also asked them to disregard comments containing irony or sarcasm.
We also asked the annotators to identify comments criticizing the publisher, journalist, or newspaper serving as the source of information.The annotators were also to disregard irrelevant comments, including spam, empty comments, or those not in Italian.
The comment annotation produced 21,618 relevant annotated comments.
Statistical analysis.We first examine the relationship between the article news framing typologies (CT and JT) and their online engagement.We do this to determine whether the CTs considered (resp.JTs) differ in terms of engagement.The null hypothesis of the test is that the samples of likes (res.comments, shares) to articles showing different CTs (res.JTs) come from populations with the same distribution (McKight and Najab, 2010).This test is non-parametric and is used to compare different groups using a variable that is an ordinal measurement (Leon, 1998).We observe that the distributions of likes, comments, and shares are usually right-skewed (Zollo et al., 2017;Devineni et al., 2017).Thus, the Kruskal-Wallis test is appropriate because it relaxes the normality assumption typical of similar parametric tests.
To examine the association of news framing with user commenting behaviour, we exploit contingency tables, i.e. tables displaying the multivariate frequency distribution of two variables.In our case, the tables cross-sectionally represent the frequency of each commenting behaviour (toxic or non-toxic, critical or noncritical, anti-migration or non-anti-migration) given the associated articles' framing typologies (both for what concerns CTs and JTs).
We finally use a set of multivariate logistic regression models (logit models) to examine commenter civility, trust in media  sources, and position on migration in relation with our framing categories.Furthermore, we account for the possibility of observing some kind of social conformity in users commenting behaviour.To this respect, each Facebook commenter can potentially read and be influenced by previous comments from other users.
In logistic regressions the probability of observing a binary outcome is predicted in the form of logarithmic odds (Stoltzfus, 2011).Thus, our models indirectly predict the probability of observing the commenting behaviours considered given a set of explanatory variables.
The binary dependent variables we considered are the following: • Toxic, as proxy for comment civility, where Toxic i = 1 when comment i is rude and disrespectful.
• Critical, as proxy for trust in the news source, where Critical i = 1 when comment i criticizes the news source (Corriere della Sera).
• Anti, as proxy for commenter position on the Italian migration issue, where Anti i = 1 when comment i indicates that the commenter is against migration.
We take into account as explanatory variables: • CT, the set of CTs as dummy variables (baseline: Article); • JT, the set of JTs as dummy variables (baseline: News report); • per_toxic, where per_toxic i is the percentage of toxic comments observed before comment i; • per_critical, where per_critical i the percentage of critical comments observed before comment i; • per_anti, where per_anti i is the percentage of anti-migration comments observed before comment i.
As CT and JT are non-mutually exclusive, we alternate these sets of variables in our models.We also alternate per_toxic, per_critical and per_anti coherently with the dependent variables; Table 3 reports in summary the 6 models considered.
In implementing our models, we also take into account the p-value issue addressed in (Lin et al., 2013), given the magnitude of our dataset, and we report the CPS charts in Supplementary Materials that show the sensitivity of p-values to different sample sizes.

Results
News framing and engagement.As a first step, we explore the interplay between news framing and online engagement to understand whether some news pieces may be more appealing than others when treating specific topics.In particular, we focus on articles-written by using different CTs and JTs-dealing with the Italian migration issue.
Table 2 provides summary statistics of article engagement aggregated by CT and JT and the results of the Kruskal-Wallis tests.Simply put, the null hypothesis of the test is that samples of articles, displaying different CTs and JTs, come from populations having the same distribution for what concerns engagement.
All tests fail to reject the null-hypothesis with a significance level α = 0.05.We thus find no clear-cut difference across the CTs and JTs in the distribution of likes, comments, and shares.We thus limit our examination to ranking CTs and JTs by their median engagement values.
Among CTs, multimedia pieces exhibit the highest engagement in terms of likes (214) and comments (132), while articles/ infographics are the most shared CT (77.5).In contrast, video/ photo-gallery pieces have the lowest aggregated engagement values (likes: 155.5, comments: 51, shares: 38.5).
Among JTs, editorials have the highest number of likes (389), articles with pop-culture references in terms of comments (163), and news reports in terms of shares (74.5).Human interest stories (G), are the least successful JT in terms of likes (78.5), comments (32.5), and shares ( 21), followed by human interest stories (I) in terms of likes ( 139), constructive journalism (56) in terms of comments, and focus-on-data pieces in terms of shares (30).
Users seem to prefer news pieces providing combinations of different ways of enjoying the reading.Dealing with the writing style, users preferences seem to be for articles presenting a softer approach to the migration topic, such as editorials or pop-culture pieces.
News framing and commenting behaviour.We proceed assessing the hypothesis that different commenting behaviours may be associated with different news framing typologies related to a fiercely debated topic, as the Italian migration issue.This aspect assumes a preliminary importance as it may offer useful insights to better understand which journalistic formats inhibit or trigger readers reactions.We assess whether the propensity to publish a toxic, critical, or anti-migration comment is different for different CTs and JTs.
Table 4 provides the results of the χ 2 and Cramér's V tests.The χ 2 tests reject the null hypothesis of independence with a p-value < 2.2e−16, suggesting a possible interdependence between news framing and each of the commenting behaviours we observed.We use the Cramér V test to further examine the effect size and find that the effect size of this interdependence is small, as in the rule-of-thumb interpretation in (Cohen, 2013).As expected, news framing is not the only, nor the most important, variable associated with single user behaviour because it only factors in the process when a user decides whether to read a news piece (Schmidt et al., 2017) based on their already settled position on the topic (Del Vicario et al., 2016).Nonetheless the relevant, although weak, connection between news framing and commenting behaviour can be associated with audience segmentation in which different users sharing similar features can have similar preferences and exhibit similar behaviours.For example, different JTs can attract the attention of user groups with different propensities to publish toxic, critical, or anti-migration comments.
Notice that JT-based framing is more strongly associated with commenter civility, trust, and anti-migration positions than Table 3 Summary of the logistic regression models considered. (1) CT-based framing.This may be because of the differing relevance the two framing typologies may have for users, and a fine-grained framing categorization may better capture the differences in user commenting behaviour.
Framing effect and social conformity.We conclude our analysis examining more thoroughly the interaction between the different commenting behaviours (commenter civility, trust in media sources, and position on migration) and the news framing typologies considered.We also broaden the extent of our analysis taking into account within our models the possibility of observing commenters acting in accordance with social conformity, i.e. the fact that a user can be influenced by the actions of other users.We rely on the set of logistic regressions previously described in this section.Our models allow to observe the individual effects of news framing and social conformity on the probability of observing a given commenting behaviour.Table 5 lists the parameters of our logistic regression models.Models 1-3 generate the probabilities of finding toxic, critical, or anti-migration comments, respectively, in four different CTs.
Note that toxic comments are more prevalent in response to video and photo-gallery items than to plain vanilla articles, and that the opposite is true in response to article/info-graphic pieces.Comments in response to multimedia and video and photogallery items are less likely to criticize the source with respect to the baseline, and comments in response to article/info-graphics items are more likely to criticize the source than those to a plain article.
Note that only comments on video and photo-gallery items are less probable of being anti-migration with respect to the baseline.
Models 4-6 generate the probabilities of finding toxic, critical, or anti-migration comments, respectively, in eight different JTs.Note that toxic comments are more prevalent in response to articles with context with respect to news reports, but that the opposite is true in response to focus-on-data pieces.Comments in response to base-line pieces, i.e., news reports, are much less likely to criticize the news source, but comments in response to constructive journalism, human interest stories (G) and focus-ondata CTs are more likely to be critical.Note also that comments in response to articles with pop-culture references and editorials are less likely to be anti-migration than those to news reports.Table 5 Specification of the logistic regression models.

Dependent variable
Toxic (1) Critical ( 2) Anti ( 3) Toxic ( 4) Critical ( 5) Anti ( 6) It is also worth noting that the odds that a comment will be toxic, critical, or anti-migration increase as the percentage of toxic, critical, and anti-migration comments observed by the commenter increases.
Focusing on news framing, we find that the two framing typologies considered, CTs and JTs, are associated with commenting behaviour.Toxic comments are more likely in response to articles containing videos and photos and in those offering a deeper contextual description of the facts but are less likely to appear in response to data and info-graphics articles.Pieces containing videos and multimedia elicit comments with a higher level of trust than data-related articles and opinion pieces, which receive comments with the lowest level of trust in the source.Most of the CTs and JTs tested do not diverge in the way they encourage anti-migration comments.Only articles with popculture references and editorials elicits fewer anti-migration comments.
Figure 1 shows the statistical signature of the CT and JT we tested.The radar plots include the average number of comments exhibiting the behaviours we traced and add the pro-migration comments.The statistical signatures of the different CT and JT agree with the results of the models.
Moving to the hypothesis of social conformity, we assess whether commenters access to previous comments influences their commenting behaviour.We find that this is true for all the commenting behaviours considered.We also find that a comment is more likely to be toxic, critical, or anti-migration if the commenter has read previous comments that are toxic, critical, or anti-migration.This is in accord with the academic literature on social influences (Deutsch and Gerard, 1955;Bandura, 1986).
In summary, at least two mechanisms are active when dealing with the users commenting behaviours we considered: news framing and social conformity.On the one hand, a certain role seems to be played by the format and content of a piece of news.
Dealing with the Italian migration issue, we find that visual pieces and factual news reports are associated with the highest level of trust in the media source, while editorials and opinion pieces are more subject to criticism.Furthermore, we note that data-driven pieces are associated with a low level of trust in the news source but are also less likely to generate toxic debate.On the other hand, social conformity as an element influencing users' commenting.

Discussion
To date the most liked image on Instagram is an egg (54 million likes).Social media platforms, indeed, are mainly intended for ludic rather than for informative purposes and news consumption on social media may be incidental rather than deliberate.Algorithms mediate and facilitate content promotion, and thus information spreading, by taking into account the preferences and attitudes of users.This paradigm shift profoundly impacts the construction of social perceptions and the framing of narratives, especially when issues are controversial, and it influences policy-making, political communication, and the evolution of public debate.Fact-checking initiatives showed limitations in approaching this phenomenon.Indeed, heated debating and polarization still play a pivotal role in online social dynamics (e.g., traditional vs. anti-establishment polarization).
In this paper, we measure the online social response to different journalistic techniques using the Facebook page of the major Italian news outlet Corriere della Sera.Since polarization seems to be one of the main drivers of online social interaction, the purpose of this work is to improve the understanding of the online disintermediated information environment by accounting for the reaction of users to a highly polarizing topic.We focus our analysis on attitudes toward migration, which in recent years has been a highly controversial topic among the Italian public.We collect an original dataset of 113 posts comments) and the corresponding news articles divided according to their method of reporting information.We describe how this set of different articles impacts user commenting behaviour in terms of toxicity, criticism of the news source, and stance on migration.
Our findings suggest that visual pieces (among content types) and factual news reports (among journalistic techniques) elicit the highest level of trust in the media source, but that editorials and opinion pieces are more subject to criticism.We note that datadriven pieces elicit a low level of trust in the news source, but less likely to generate toxic debate.We also find another important influence on commenting behaviour: social conformity.

Fig. 1
Fig. 1 Radar plots displaying the characteristic features of each content type (panel a) and journalistic technique (panel b).Each dimension respectively displays the standardized average number of critical comments, toxic comments, anti-migration comments, pro-migration comments.Features' values have been normalized across CT (res.JT) to ease comparison.The corresponding number of posts and comments for each CT (res.JT) are reported above each radar plot.
Visual content with little or no text.Table 2 presents a summary of the engagement metrics of the posts by content type and by journalistic technique.
4. Multimedia: Usually longer news articles presenting a mixture of videos, photos, and text.Table 1 provides a detailed breakdown of the data associated with the different journalistic techniques and content types and

Table 1
Data breakdown of the posts, reactions and comments by their content type (above) and their journalistic technique (below).
a CT.b JT.

Table 2
Summary of the engagement metrics of the posts by content type (above) and by journalistic technique (below).

Table 4
Summary of the results of the χ 2 test and of the Cramér's V test.