Classification of unlabeled online media

This work investigates the ability to classify misinformation in online social media networks in a manner that avoids the need for ground truth labels. Rather than approach the classification problem as a task for humans or machine learning algorithms, this work leverages user–user and user–media (i.e.,media likes) interactions to infer the type of information (fake vs. authentic) being spread, without needing to know the actual details of the information itself. To study the inception and evolution of user–user and user–media interactions over time, we create an experimental platform that mimics the functionality of real-world social media networks. We develop a graphical model that considers the evolution of this network topology to model the uncertainty (entropy) propagation when fake and authentic media disseminates across the network. The creation of a real-world social media network enables a wide range of hypotheses to be tested pertaining to users, their interactions with other users, and with media content. The discovery that the entropy of user–user and user–media interactions approximate fake and authentic media likes, enables us to classify fake media in an unsupervised learning manner.

The idea that people may not be suitable to assess the authenticity of information without the aid of additional tools is widely explored by the forensics community 1,2 . Information sharing in social media is highly dependent on data modality, the behavior of fellow networkers, the habit of using online services, and the widespread presence of opinion leaders in posts and comments 3,4 . In some cases, information is doctored using deep fakes 5 , or manipulated by spreading misinformation relating to health outbreaks such as COVID-19 6,7 . Beyond affecting the conduit of social media networks, much of algorithmic fake news detection 2,8 or recommender systems 9 rely on data type identification and data labeling. Such information is typically manually compiled by content moderators in social media companies who may undergo post-traumatic stress disorder as a result of viewing disturbing media 10,11 . Data-driven supervised methods benefit from superior classification accuracy due to the availability of labels but may fail to generalize to unseen data and often rely on manual human labeling. Alternatively, unsupervised methods do not rely on labels to classify data and address the trade-off in accuracy with the help of additional features, feature representation techniques, or complex models. Thus, there is a necessity to shift towards unsupervised approaches for content discovery or fake media discovery [12][13][14] .
In this study, we consider (1) work that seeks to understand why misinformation spreads and (2) work that seeks to prevent the spread of misinformation through classification. Current literature has improved our understanding of truth and falsity diffusion through quantitative analyses on the spread of rumors such as the discovery of the Higgs boson 15 , and the spread of anti-vaccine campaigns that led to decreases in vaccination rates against measles 16,17 . Other studies have investigated rumor diffusion modeling 18 , credibility evaluation 19,20 and intervention strategies to curtail the spread of rumors 21 . Similar analyses have been conducted in social media networks such as Facebook, where Del Vicario et al. 22 analyze the diffusion of science and conspiracy-theory stories, and in Twitter where Bovet and Kakse 23 study the influence of fake news during the 2016 presidential election, and Vosoughi et al. 24 study the characteristics of fact-checked rumors.
The spread of information, authentic or fake, is however significantly impacted by the credibility of the users in the network and their connections 25 . In a social media network such as Twitter, users form opinions and beliefs 26 based on outside influences 23,27 , and use these opinions to test if information adheres to or opposes their beliefs 28 . Though the most direct measure of user credibility involves asking the other users 29 , such a measure is often subjective. Additionally, a subjective measure does not apply to sources or users previously unknown to a user 30 . However, such occasions are likely to occur during times of disasters such as COVID-19 7  www.nature.com/scientificreports/ measles outbreaks 16,17 where authentic information was provided by those physically attending the event 30 . Hence, we are interested in understanding how fake information spreads in a controlled social media network where users respond to unknown users with the help of an objective credibility measure. Node-edge homogeneity and the homogeneous sharing paths created by social media users during the evolution of a social media network, play a critical role in the spread of fake information 22,31,32 . It has been shown that users' opinions on media and their engagement behaviors on social media networks can be utilized to distinguish between authentic and fake information 8,33,34 . However, such work 8,33,34 typically makes use of supervised learning techniques to detect fake information, requiring a labeled dataset. Though these supervised methods have shown promising results, they suffer from a critical limitation of requiring a pre-annotated dataset to train a classification model. Further, Chu et al. 35 and Wang 33 have released Facebook and Twitter datasets that include user activity and interactions pertaining to a specific time period, thereby not capturing the organic growth of user-user connections from the time users join the network. In order to address the limitations of requiring human-annotated datasets to train supervised models, crowd-sourcing approaches make use of platforms such as Amazon Mechanical Turk to recruit workers as users or as data labelers to take part in social media experiments. Crowd-sourcing techniques typically leverage cost-effective workers to obtain annotations, hence alleviating the burden of expert checking 36 . Long et al. 37 and Rodrigues et al. 38 propose a Bayesian probabilistic approach coupled with active learning, wherein they use crowd-sourced data to estimate both the data authenticity as well as the credibility of each individual data labeler/user. Recent studies 39,40 have proposed replacing the Expectation-Maximization (EM) algorithm which is typically used for parameter estimation and inference with a deep generative model such as a variational auto-encoder (VAE). By replacing the EM algorithm with neural networks, the additional computational overhead can be avoided, which allows the models to generalize beyond classification settings 40 . Rodrigues et al. 40 introduce an additional layer (crowd layer) in their CNN such that the typical softmax output layer is transformed into a bottleneck layer while allowing the crowd layer to handle data labeler/user reliability and labeling noise. While deep neural networks have been shown to improve the classification/labeling accuracy of unsupervised probabilistic models that use EM, the notion of explainability is replaced by a black-box approach. A study by Yang et al. 14 proposes an unsupervised approach for detecting misinformation in online social media networks such as Twitter and Facebook using yet another probabilistic graphical approach. In their approach, Yang et al. consider the credibility of users as a latent random variable in addition to user opinions and use an efficient Gibbs sampling approach to estimate news authenticity. While supervised approaches such as 8,33,34 have leveraged user-user interactions, these unsupervised probabilistic approaches 14,38-40 explicitly assume that each user's opinion is independent of other users. Hence, they do not consider user-user interactions, which can be an influential source of recommendation in Twitter and Facebook [41][42][43] , especially if users know one another. Further, prior work 14,38-40 considers news features 39 as a latent random variable in addition to considering the authenticity of news/data and the credibility of the data labelers/users 37,40 as latent random variables.
When presented with uncertainty, human behavior in social media networks tends to react differently, employing unconventional social heuristics 44 . Hence, regardless of whether users know one another, there exists uncertainty in what a user likes and who they follow or befriend. The use of information-theoretic measures, such as entropy, avoids making assumptions about human behavior, thus allowing statistical characterization of uncertainty 45 in social media networks. Shannon's entropy has been used as a statistical feature in detecting rumors in social media 46 , abnormal activities or anomalies in IoT devices and sensor networks [47][48][49] . In addition to detecting anomalies, entropy has also been used in determining the credibility of machines in a machine-machine communications network in order to distinguish malicious nodes from honest nodes by iteratively updating the credibility value of each node for every message passed between the nodes 50 . Similarly, entropy has also been used to estimate the veracity of topics from tweets, such a measure has been reported to categorize tweets in the veracity spectrum in a consistent manner 51 .
Since it is evident that users in social media networks adhere to a notion of credibility before following a user and liking media, we derive a relationship between an objective credibility measure, user opinions (media likes), and the probability of following or establishing a connection with a user. We show that such a relationship helps understand the connection between user opinions and the credibility of users, and how they affect the probability of users making new connections. To that end, unlike prior approaches 14,[38][39][40] , in our work, we do not consider the authenticity/label of the data as a latent variable, but instead only consider user opinions as a latent variable. This avoids the assumption that users like and share media on the basis of truth when users could intentionally like or share media on the basis of satire 52 . Additionally, the derived proportionality between user credibility and user opinion allows us to use one in place of another, thereby decreasing the number of latent variables considered. Then, we compute the entropy of user-user connections given user opinions in order to map user interactions to a quantifiable value that expresses the uncertainty contributed to the network by each user. Further, by exploiting the principle of entropy, we select users based on a threshold entropy to take part in a majority voting in order to distinguish fake information from authentic information. This allows utilizing users and their interactions as features to detect misinformation while decoupling the estimation of credibility or opinion and classification of data.
The paper is presented as follows. First, we annotate the controlled experiment conducted to collect real-time social network data. Then, we derive a relationship between user credibility, media likes, and the probability of establishing a connection with a user, followed by a comprehensive analysis that shows how entropy explains the spread of fake information. Finally, we propose an unsupervised model that uses entropy to select users to classify information as fake or authentic.

Methods
Data collection. We conduct a real-life social experiment, wherein participants are recruited from the crowd-sourcing platform Amazon MTurk (AMT) to participate in the study. The study was conducted with the approval of and in accordance with the relevant guidelines and regulations of The Pennsylvania State University's Institutional Review Board (IRB). Prior to the conduct of the study, informed consent was obtained from all participants for participation in the study. Only human subjects of age 18 or above were included in the study. We created a mock online social media platform that replicates the functionality of Twitter as shown in Fig. 1 and follows their directional model of interactions. We restrict users from posting their own media and from unfollowing a followed user to conduct a controlled real-time, real-world social experiment. This web application is released for public use to replicate or conduct similar online social experiments. The focus of the data collection is to organically acquire the interactions between unknown users rather than from known users within the network in order to gain a fundamental understanding of how fake information can be classified using random user opinions. We populate the social network with authentic and fake videos from the FaceForensics data set 53 and verified fake YouTube videos. The users were monetarily incentivized to spend at least 45 min per Human Intelligence Task (HIT). To ensure that the human subjects are reliable in staying throughout the study and with a provision for new users to join, we only recruited subjects who had a HIT' (Human Intelligence Task) approval rate of greater than 90% in all requesters' HITs. A user was allowed to participate in any number of HITs over the period of study, ensuring that the same user population could log into the network and continue accruing credibility and followers. We chose to use 40 random videos, with 20 being authentic and the other 20 fake. The total number of enrolled users in the study was 620, which falls within the range of participants recruited for other social science experiments that utilize AMT 54,55 .
To simulate a real-world scenario in Twitter where users follow users or are followed by users thereby forming explicit connections 56 , we introduce the notion of user credibility. According to Resnick et al. 57 , reputation mechanisms provide an incentive for honest behavior and help users make decisions involving trust. In this study, we develop an approach to compute user credibility using trust modeling. Specifically, we use the Beta Reputation System (BRS) 58 used in e-commerce to calculate the credibility of users. We select this method as it is grounded in probability theory and uses expectations on probability distributions to derive the predicted benefit of trust to help users form explicit connections in a participatory media setting. The credibility of user i is C i and we assume the prior probability of C t i at time t to be generated from a Beta distribution with parameters a t (prior authentic media category counts) and b t (prior fake media category counts) such that C t i ∼ Beta(a t , b t ). We use the uniform distribution as prior to the Beta distribution such that every new user is assigned a credibility score of 0.1. As the user continues to form new links within the network, and like media, the credibility score gets updated. We use a Bayesian update rule to update the credibility score of the user at each timestamp. We state the credibility update rule for user i such that the parameter a t is updated as a t+1 i ← a t i + a δt i and b t is updated as b t+1 www.nature.com/scientificreports/ where δt is 20 min. To understand the relationship between credibility and follower count, we fit a regression model between follower counts and user credibility as shown in Fig. 2 and find the co-efficient of determination (R 2 ) as 0.56. This shows that there is a positive correlation between follower count and credibility score. Additionally, we also compute and illustrate the final credibility score distribution of the social media network at the last timestamp as shown in Fig. 3. We find that the distribution closely fits the Beta distribution given the uniform prior with E[C] = 0.1 . We also provide a monetary incentive to the top three users who earn the highest credibility score and get the highest user following by the end of the 2-day experiment.
Preliminaries and definitions. The social network under study is a directed network having links specifying the followers (in-degree) and the following (out-degree). A social network G (t) := {V , E } is an order 2 Tensor such that G t ∈ R N×N where N represents the number of users or V such that V ∈ R N , Relating entropy to user credibility. We assume that users recruited in our social network are not known to each other and by showing every user the credibility of all users, we assume that users tend to follow or be followed by other users on the basis of credibility. We observe a positive linear correlation between the number of links connected to a user (in-degree) and credibility as shown in Fig. 2, consistent with findings from other works 30 . However, in reality, such a credibility measure is often network-dependent, or may not be read- Figure 2. Fitted regression curve to follower counts against credibility score. www.nature.com/scientificreports/ ily available. Hence, we use links and media likes to approximate user credibility. We define a similarity metric, Media Node-Edge Homogeneity, which computes similarity as a function of common media liked by any two users, which have an edge connection at time t. We denote the media node-edge homogeneity (MNEH) metric as M t ij , where, In Eq. (2), M t ij between any two users i and j at time (t), is bounded between the interval [0, 1], where, 0 ≤ M ij ≤ 1 expresses the magnitude of similarity in media likes between any two users i and j. The node-edge homogeneity interval [0, 1] is discretized with a resolution of 0.01. The MNEH matrix (M t ) is a sum of matrices as shown in Eq. (3) with varying γ , such that γ = γ /100 , M t ij,γ = 1 if and only if the similarity in media liked by users i and j is γ%.
Given A t and M t from the in-degree and out-degree networks, we assess the Directed Acyclic Graph (DAG) connecting i, j and M t . We assume liking media and forming links with other users to be conditionally independent events since j = 1 is observed thereby breaking the dependence between the events. Hence, we compute the probability of a link between user i and j for each instant in time t as shown in Eq. (4).
where p(j = 1|i) = P t ij is a conditional probability which represents the probability of a link formed with user j given user i at time t such that P t ij is an element of the transition matrix (P t ) which is algebraically computed as (AD −1 ) t , and p(j=1,M tγ ) p(M tγ ) = P t ij,γ is computed as follows, where D AMγ is a diagonal matrix such that D AMγ ,ii = j∈N (A • M) ij,γ , D Mγ is a diagonal matrix such that D Mγ ,ii = j∈N M ij,γ and ( • ) denotes Hadamard product operation between matrices. For each media category l ∈ L , Z t jl is generated from a Bernoulli distribution with parameter C t j such that Z t jl ∼ Bernoulli(C t j ) and C t j is the probability of user j liking l. Hence, we assume the prior probability of C t j to also be the assumed credibility of a user j and sampled from a Beta distribution with parameters a t (prior authentic media category counts) and b t (prior fake media category counts) such that C t j ∼ Beta(a t , b t ). In a similar manner as Eq. (4), we compute the probability of a link between user i and j given the media matrix (Z t j ) for each instant in time t as shown inEq. (6) . Let p(j = 1|Z t j ) be denoted by �(Z) and p(j = 1|M t j ) be denoted by F (�(Z)) since M t j is a function of Z t j . Using the results from Eqs. (4) and (6), we derive the proportionality between C t j and P t ij,γ given γ for every timestamp as shown by Eq. (7).
Hence, from Eq. (4), it is evident that we are indeed computing a weighted transition probability with the weight being proportional to the user's credibility. Using these results, we now compute the Shannon's Information Entropy (joint entropy) generated by all users (i, j, k, ..., n) as the sum of entropy generated by each user at time t.

Results and discussion
Objective. The objective of this study is to use the entropy response of media node-edge homogeneity over time to perform unsupervised classification of fake media. We present a visual overview of our entropy computation approach as shown in Fig. 4, along with an algorithm box (Algorithm 1). We have organized this section to discuss hypotheses, hypotheses testing, and how the tests inform the selection of users that separate fake and authentic media in a majority voting classifier to classify fake media. Hereafter, the notations H 0 , H 0 * and H a , H a * refer to null-hypothesis and alternate hypothesis respectively. www.nature.com/scientificreports/ Entropy of media node-edge homogeneity. We compute the joint information entropy of the matrix for each value of γ and for each t using Eq. (7). The social media network is directional as it has an in-degree or followee network and an out-degree or follower network. Hence, we hypothesize that the entropy generated by a random variable (X ) representing social network users in the in-degree and the out-degree network is sampled from the same underlying node-edge distribution, formally stated as follows: where H(X) in and H(X) out denotes entropy distribution in in-degree and out-degree network respectively. We use H 0 and H a to denote the null hypothesis and alternate hypothesis following conventional notation. We perform a Kolmogorov-Smirnov (KS) test between the in-degree entropy distribution and out-degree entropy distribution to test our hypothesis. For all hypothesis tests, we use a significance level (α ) of 0.05. We find no statistically significant difference between the in-degree entropy distribution and out-degree entropy distribution with a KS statistic = 0.09 and p value = 0.81, thereby failing to reject the null hypothesis. Figure 5 illustrates entropy response and media likes distribution against varying media node-edge homogeneity for the in-degree network across all timestamps such that each timestamp is overlaid against the other. The comparison draws attention to the correlation between media distribution and entropy distribution in the in-degree network. To check for correlation, we compute the Pearson's correlation coefficient between M t and H t , where H t represents network entropy distribution at time t while M t represents media node-edge homogeneity of the network at time t. We compute the correlation for all t ∈ T and for all users in the in-degree and out-degree network. The average correlation across all timestamps is ρ = 0.82 , suggesting that entropy and media likes distribution are strongly positively correlated. Hence we hypothesize that the entropy in either network is sampled from the same distribution as the fake and authentic media distribution. Formally, we state our hypotheses as follows: where T(Z) and F(Z) denotes the authentic and fake media distribution of the random variable (Z) respectively. We find no statistically significant difference between entropy distribution and authentic media distribution (H 01 ) with a KS Statistic = 0.14 and p value = 0.28, thereby failing to reject the null hypothesis H 01 . We also find no statistically significant difference between entropy distribution and fake media distribution (H 02 ) with a KS statistic = 0.12, p value = 0.46, thereby failing to reject the null hypothesis H 02 . Thus, we observe that entropy response of media node-edge homogeneity and media likes distribution are strongly positively correlated. Though entropy response is individually correlated with authentic and fake media distribution, we hypothesize that it characterizes the superposition of both the media distributions. We assume and also empirically show (see Fig. 6a www.nature.com/scientificreports/ the underlying media likes distribution, as well as entropy distribution, are sampled from a Gaussian distribution such that media likes distribution is approximated as a linear combination of Gaussians, one describing the spread of fake media likes and the other describing the spread of authentic media likes as shown in Fig. 6d. Based on our hypothesis, we assume that H ∼ N(µ, σ 2 ) is decomposable as a mixture of Gaussians such that H authentic + H fake ∼ N(µ authentic + µ fake , σ 2 authentic + σ 2 fake ) , where each Gaussian component represents the underlying authentic and fake media likes distribution. Then we compute the Jensen-Shannon divergence (JSD) 59 between the actual joint entropy distribution (H ) and Gaussian approximated entropy distributions (H A and H B ) as follows, Hence, we formulate another pair of hypotheses:  To test our hypotheses, we use the Expectation-Maximization algorithm to estimate the Gaussian parameters and fit the data to the Gaussian components found using Gaussian mixture modeling. We perform a KS test between the information gain from authentic and fake media distribution as well as from the approximated Gaussian components. We reject the null hypothesis (H 01 ) that authentic media likes distribution is statistically significantly different from that of the fake media likes distribution with KS statistic = 1.0, p value = 9.99e−16 and the null hypothesis (H 02 ) that approximated authentic media likes distribution is statistically significantly different from that of the approximated fake media distribution with KS statistic = 1.0, p value = 8.61e−83 for the in-degree network. Now, we describe how our model uses the results from the hypotheses tests to select user opinions as features for the media classifier. Once we compute the joint entropy response of the network to media node-edge homogeneity for varying γ , we approximate joint entropy as a mixture of Gaussians. We then compute the information gain from actual entropy and either of the Gaussian approximated media distributions and select the distribution with the maximum information gain (Q ). Then, we define threshold (η ) as a percentage of maximum user entropy in the network such that Q is the set of all users above η . Since the maximum entropy generated by a user is 4.0 bits, >25% refers to all users who have generated greater than 1.0 bit of entropy. We then compute an aggregated histogram of media likes per category from all users above the threshold. This enables us to aggregate all the media opinions of users in Q who have been selected as features for the media classifier. Finally, we compute the average media likes per category and predict media categories with media likes above that of the www.nature.com/scientificreports/ average media likes per category as authentic media, else fake. Algorithm box 2 depicts the media classification process in a step-by-step manner.
Since it has been shown that there exists a positive correlation between users and credibility, we test our model on the users from only the in-degree network. Apart from selecting users based on our model, we validate our work by benchmarking against (1) all users from the network as a classifier and (2) Durall et al. method 60 which is a state-of-the-art unsupervised media classifier. From Table 1, we observe that using the combined human decision is similar to a random coin flip. Since we assume that media likes refers to user authentication 14 , other evaluation metrics such as F1 score, precision and recall, cannot be calculated from just media likes. In replicating Durall's method, we compute feature embeddings for all media used in this experiment and consider this www.nature.com/scientificreports/ the test set. We use the convolutional neural network (CNN) architecture proposed by Durall et al. to generate media embeddings for the test set, which we use as input to the k-means clustering method with k = 2 . We find the F1 score of the unsupervised classifier to be outperformed by our model, as shown in Table 1. We find that selectively filtering users on the basis of a network independent measure such as entropy is sufficient to classify fake media with an accuracy higher than random chance and unsupervised state-of-the-art model. In a network where users are initially unknown to one another, we find that it is possible to approximate the underlying objective credibility using user opinions. Further, by computing the entropy response of the in-degree users using media node-edge homogeneity, we compute the entropy within the network. We approximate entropy as a superposition of fake and authentic media likes distribution. Since the distribution of fake media is different from authentic media, we are able to use the Gaussian approximated cluster with the highest information gain to utilize user opinions as features to classify fake media.

Time invariance of the proposed model.
To study if our model is time-invariant to information classification, we perform an empirical experiment by introducing a sliding window parameter h. In Fig. 7, we illustrate the average performance of the classifier across all timestamps for every sliding window parameter in the range [2,50] and a stride parameter s = 3 which corresponds to the movement of the time window across time. We find that varying the time window varies the F1 score of the classifier with increasing time windows leading to increased F1, precision, recall and accuracy scores. When the sliding window parameter is increased by a value of 1 from 3 to 50, we approach the highest media classification accuracy and F1 score as shown in Table. 1. Since a time window of 2 corresponds to a period of 40 min in our study, a period of 17 h is required to achieve an F1 score of 0.68 which is comparable to the performance of the state-of-the-art unsupervised media classifier. Since social media networks have been operational for hundreds of thousands of hours, accessing prior time should not be an issue. In comparison, the state-of-the-art unsupervised approach proposed by Durall et al. requires neural network fine-tuning as new media is uploaded to the network or in the worst case, re-training, given the development of new media doctoring tools. Hence, while our model is not time-invariant, we show that it is possible for user-user and user-media interactions of a network over a period of time to be able to quantify the entropy needed to filter fake media. Further, if we consider the relationship between time and entropy, we find that user activity in different timestamps often differs as a result of varying interaction with media (some network interactions happen at various timestamps with huge period of inactivity), causing entropy distribution of individuals to fluctuate with high uncertainty. However, when we consider 50 timestamps as shown in Fig. 7, the uncertainty due to user entropy fluctuations across different timestamps averages out, yielding consistent metric scores. This further highlights the dependence between time and the evolution of entropy.

Conclusions
In this study, we design and conduct a controlled social media experiment with unknown human subjects in a network similar to Twitter. To characterize fake media using a network independent metric, we propose using media polarity and node-edge homogeneity, which are well-known detriments affecting the spread of fake media as a metric to compute the similarity between users in the network. In social media networks, interactions such as liking and sharing media, and forming/removing connections with other users, lead to the fluctuation of entropy over time. The entropy response of the in-degree users is captured using media node-edge homogeneity to compute the entropy within the network. In this paper, we compute entropy as a function of media polarized node-edge homogeneity to characterize the spread of fake and authentic media. Using our findings, we describe a majority voting classifier to classify online media using selective user opinions. However, our model is limited by the size of the moving window with the moving window size equal to that of the whole study improving the F1 score of the classifier. This shows that new users who join the network at different times are valuable for media classification. As future work, selective user opinions can be filtered using approaches such as active learning and combined with unsupervised media classifiers to further improve fake media classification in social media networks.
Apart from our findings, we highlight some of our assumptions that can possibly be grounded in theory using existing work or addressed as future work. We assume that no user knows each other prior to joining the network (random user network), media likes equal validation of authenticity, our 2-day study linearly maps to a longer-term study. Finally, future work may consider exploring the generalizability to other media types such as text and audio.

Data availability
The data along with the social media platform that was developed to collect the data as well as the corresponding algorithm have all been made available at the following repository: https:// github. com/ AiPEX-Lab/ Social-Netwo rk-Analy sis.