Using Data Science to Understand the Film Industry's Gender Gap

Data science can offer answers to a wide range of social science questions. Here we turn attention to the portrayal of women in movies, an industry that has a significant influence on society, impacting such aspects of life as self-esteem and career choice. To this end, we fused data from the online movie database IMDb with a dataset of movie dialogue subtitles to create the largest available corpus of movie social networks (15,540 networks). Analyzing this data, we investigated gender bias in on-screen female characters over the past century. We find a trend of improvement in all aspects of women`s roles in movies, including a constant rise in the centrality of female characters. There has also been an increase in the number of movies that pass the well-known Bechdel test, a popular--albeit flawed--measure of women in fiction. Here we propose a new and better alternative to this test for evaluating female roles in movies. Our study introduces fresh data, an open-code framework, and novel techniques that present new opportunities in the research and analysis of movies.

The film industry is one of the strongest branches of the media, reaching billions of viewers worldwide [12,18]. Now more than ever, the media has a major influence on our daily lives [44], significantly influencing how we think [28], what we wear [48], and our self-image [42]. In particular, the representation of women in media has an enormous influence on society. As just one example, a new study shows that "women who regularly watch The X-Files are more likely to express interest in STEM, major in a STEM field in college, and work in a STEM profession than other women in the sample" [17].
Movies are the fulfillment of the vision of the movie director, who controls all aspects of the filming. It is well known that movie directors are primarily white and male [8]. With such a gender bias, it is not surprising that there is a male gender dominance in movies [6]. Studies from the past two decades have confirmed that women in the film industry are both underrepresented [21,36] and portrayed stereotypically [49]. A recent study found that the underrepresentation is so sizeable that there are twice as many male speaking characters as female in the average movie [1].
While the gender gap in the film industry is a well-known issue [1,10,20,36,49], there is still much value in researching this topic. Most previous gender studies can be categorized into two types: the first type offers simple statistics from the data to emphasize the gender gap [36]; and the second type introduces more advanced analytical methods, yet generally uses only a small amount of data [23,30].
In this study, we employed the latest data science tools to comprehensively analyze gender in movies (see Figure 1 1 ). This is the largest study to date that uses social network analysis (SNA) to investigate the gender gap problem in the film industry and how it evolved.
The study's primary goals are to answer the following four questions: Question 1: Are there movie genres that do not exhibit a gender gap?
Question 2: What do characters' relationships reveal about gender, and how has this changed over time?
Question 3: Are women receiving more central movie roles today than in the past?
Question 4: How has the fairness of female representation in movies changed over the years?
To answer these questions, we first analyzed movie subtitles using textprocessing algorithms and a list of movie characters' names (see Figure 2). We then developed a novel algorithm to construct a movie character's social network. We created an open-source code framework to collect and analyze movie data, and we used this framework to construct the largest existing open movie social network dataset that exists today.
Using the constructed movie social networks, we extracted dozens of topological features that characterized each movie. By analyzing these features, we could observe the gender gap across movie genres and over the last 99 years. Moreover, by utilizing the dataset, we developed a machine-learning classifier, which is able to assess, with precision at 200 of 0.94, how fairly women are represented in movies (i.e., if a movie passes the Bechdel test [24]).
Our results demonstrate that it is possible to classify a movie character's gender solely by the character's metadata. In addition, in most movie genres there is a statistically significant difference between men and women in centrality features like betweenness and closeness. These differences indicate that men are getting more central roles in movies than women (see Figures 3a and 3b, and Section 4). Another sign of the underrepresentation of women in movies is found by analyzing interactions among three characters: only 3.57% of the interactions are among three women, while 40.74% are among three men. These results strengthen previous studies' results that women play fewer central roles [23,36], and indicates that on average women have more minor roles. Our results highlight how and where gender bias manifests in the film industry and provides an automatic way to evaluate it over time.
The key contributions presented in this paper are fivefold: • A novel algorithm (see Section 3) which utilizes movie subtitles and character lists to automatically construct a movie's social network (see Section 3.1 and Figure 2).
• The largest open movie social network dataset, 21 times larger than the previous dataset [34] (see Section 3.2). Our dataset contains 15,540 dynamic networks of movies (937 of these networks are networks of biographic movies, which have information about real world events).
• An open-source framework for movie analysis. The code contains a framework to generate additional social networks of movies, facilitating research by creating and analyzing larger amounts of data than ever before.
• A machine-learning classifier that can predict if a movie passes the Bechdel test (see Section 3.4.4) and can evaluate the change in gender bias in thousands of movies over several decades (see Section 4).
• Our new and alternative automated Bechdel test to measure female representation in movies. This new test overcomes the weaknesses of the original Bechdel test.
Our study demonstrates that inequality is still widespread in the film industry. In movies of 2018, a median of 30% women and a mean of 33% were found in each movie's top-10 most central roles. That being said, there is evidence that the gender gap is improving (see Figure 6).
The remainder of this paper is organized as follows: In Section 2, we present an overview of relevant studies. In Section 3, we describe the datasets, methods, algorithms, and experiments used throughout this study. In Section 4, we present our results. Then, in Section 5, we discuss the obtained results. Lastly, in Section 6, we present our conclusions from this study and offer future research directions.
2 Related Work

Movie Social Networks
In the past decade, the study of social networks has gained massive popularity. Researchers have discovered that social network analysis techniques can be used in many domains that do not have explicit data with a network structure. One such domain is the film industry. Researchers have applied social network analysis to analyze movies, gaining not only new insights about specific movies but also about the film industry in general. For example, using social networks makes it possible to empirically analyze social ties between movie characters.
In 2009, Weng et al. [47] presented RoleNet, a method to convert a movie into a social network. The RoleNet algorithm builds a network by connecting links between characters that appear in the same scene. RoleNet is based on using image processing for scene detection and face recognition to find character appearances. Weng et al. evaluated their method on 10 movies and 3 TV shows. The method was used to perform semantic analysis of movies, find communities, detect leading roles, and determinine story segmentation.
In 2012, Park et al. [41] developed Character-net, another method to convert movies to networks. Character-net builds the social network based on dialog between characters, using script-subtitle alignment to extract who speaks to whom in the scene. Park et al. evaluated their method on 13 movies [41]. Similar to RoleNet, Character-net was used to detect leading roles and to cluster communities.
In 2014, Agrawal et al. [22] presented a method for parsing screenplays by utilizing machine-learning algorithms instead of using regular expressions. Their study showed that the parsed screenplay can be used to create a social network of character interactions. In 2015, Tran and Jung [45] developed the CoCharNet, a method which adds weight to a link in the interaction network, where the weight is a function of the number of times two characters appear together. Tran and Jung used CoCharNet to evaluate the importance of characters in movies. They demonstrated that network centrality features such as closeness centrality, betweenness centrality, and weighted degree can be used to classify minor and main characters in a movie. For instance, they detected the main characters using closeness centrality with a precision of 74.16%.
In 2018, Lv et al. [38] developed an algorithm to improve the accuracy of creating social networks of movies. They presented StoryRoleNet, which combines video and subtitle analysis to build a more accurate movie social network. The subtitles were used to add additional links that the video analysis might miss. Similar to RoleNet and Character-net, Lv et al. used the movie social networks to cluster communities and to detect important roles. They evalu-ated the StoryRoleNet method on 3 movies and one TV series, for which they manually created baseline networks [38].
Also in 2018, a dataset from Moviegalaxies [34] 2 was released. Moviegalaxies is a website that displays social networks of movie characters. The dataset contains 773 movie social networks that were constructed based on movie scripts. However, Moviegalaxies did not disclose the exact methods which were used for the construction of the networks.

Evaluating the Gender Gap
In recent years, there have been many studies that attempt to evaluate the gender gap between males and females across various domains [33,35,36,46]. For example, in 2018 the World Bank evaluated that the costs of gender bias are vast; gender inequality results in an estimated $160.2 trillion loss in human capital wealth [19].
Over the years, researchers have discovered many manifestations of the gender gap in our society. Lariviere et al. [35] discovered that scientific articles with women in dominant author positions receive fewer citations. Wagner et al. [46] observed that men and women are covered equally on Wikipedia, but they also discovered that women on Wikipedia are portrayed differently from men. Jia et al. [33] found that in online newspapers, women are underrepresented both in text and images.
The state of women in the film industry is similar to other domains: women are underrepresented and badly portrayed [36,49]. The Boxed In 2017-18 report [36] observed a 2% decline in female major characters across all platforms, compared to the previous year.
To tackle the underrepresentation of women in movies in 1985, the cartoonist Alison Bechdel published a test in her comic strip Dykes to Watch Out For to assess how fairly women are presented in filmed media. The Bechdel-Wallace test [24] (denoted as the Bechdel test) has three rules that a movie has to pass to be considered "women friendly": 1. It has to have at least two women in it.
2. The women have to talk to each other.
3. The women must talk about something besides a man.
Only 57% of current movies pass this test. To Bechdel's surprise, the media adopted her joke, and today it is a standard for female representation in movies [2, 3, 5, 9, 11].
The Bechdel test is also used by researchers. In recent years, studies have utilized the test to evaluate gender bias in movies. In 2014, Garcia et al. [30] quantified the Bechdel test and also applied it to social media. They joined YouTube trailers, movie scripts, and Twitter data, which resulted in 704 trailers for 493 movies and 2,970 Twitter shares. Garcia et al. created a social network of dialogues for these movies. Additionally, they constructed a network of dialogues between Twitter users who discussed the trailers. They mapped dialogues between men who were referring to women and between women who were referring to men. This mapping was used to calculate the Bechdel score. They found that trailers of movies which are male biased are more popular. Also, they discovered that Twitter dialogues have a similar bias to movie dialogues [30].
In 2015, Agarwal et al. [23] studied the differences between movies that pass and fail the Bechdel test. Similar to Garcia et al., Agarwal et al. also constructed social networks using screenplays. They created a classifier to automate the Bechdel test, which was trained on 367 movies and evaluated on 90. In the evaluation, they discovered that network-based features perform better than linguistic features. Additionally, they discovered that movies that fail the Bechdel test tend to have women in less central roles [23].

Graph Features and Named Entity Recognition
Data science tools and techniques have evolved rapidly in the past couple of years [27]. In this study, we primarily utilized data science algorithms from the domains of Natural Language Processing (NLP) and Social Network Analysis to computationally analyze movie content, movie social network structure, and how movie features change over time.
Namely, we used NLP to extract character names from the movie subtitles by utilizing Named Entity Extraction (NER) algorithms [40]. We used both Stanford Named Entity Recognizer [29] and spaCy Python Package [32] to find where characters appear in the text.
To match characters' names in the subtitles with characters' full names, we utilized FuzzyWuzzy [15], a Python package for fuzzy string matching. Specifically, we used FuzzyWuzzy's WRatio [7], a method for measuring the similarity between strings. WRatio uses several different preprocessing methods that rebuild the strings and compare them using Levenshtein distance [37]. Also, WRatio takes into account the ratio between the string lengths.
After extracting the movie characters, we constructed the movie social networks and used various graph centrality algorithms, such as closeness, betweenness, degree centrality, and PageRank [25] to identify the most central characters in each constructed movie network.

Constructing Movie Social Networks
One of this study's primary goals was to develop a straightforward algorithm that would construct the social network of character interaction within a given movie. We achieved this goal by utilizing movie subtitles 3 and a list of movie character names. Namely, given a movie, we constructed the movie social network G :=< V, E >, where V is the network's vertices set, and E is the set of links among the network's vertices. Each vertex v ∈ V is defined to be a character in the movie. Each link e := (u, v, w) ∈ E is defined as the interaction between two movie characters u and v, w times. For a movie with a given subtitle text and a given character list, we constructed the movie's social network using the following steps (see Figure 2): 1. First, we detected when each character appeared in the subtitles. To extract the characters from the subtitles we used NER, extracting all the entities which were labeled as a person or an organization. Additionally, for each entity, we stored the time the entity appeared in the movie.
2. Next, we matched the entities found in the subtitles with the character list. It worth mentioning that it is not possible to map one-to-one between the characters in the character list and the characters extracted from the subtitle. For example, in the movie The Dark Knight, Bruce Wayne was referred to as "Bruce Wayne" 3 times, as "Bruce" 16 times, and as "Wayne" 20 times.
3. To address the matching problem, we proposed the following mapping heuristic (see Algorithm 1). First, we split all the roles into first and last names and linked them to the actor and the character's full name (line 2). Then, if there was only one character with a certain first or last name (one-to-one match), we linked to the character all its occurrences in the subtitles (lines 3-5). However, if we had several characters with the same first or last name, we did not always know who was referred to in the text. For example, in the movie Back to The Future there are three characters with the last name McFly; where only "McFly" was mentioned in the text, we could not determine which character was referenced. Another challenge we encountered was when only part of the character's name was used. For instance, in the movie The Godfather, the main character is Don Vito Corleone, but he was never mentioned once by his full name because he usually was referred to as "Don Corleone." Moreover, there are other Corleone family members in the movie. To overcome this challenge, we used WRatio to compare strings and match parts of a name to the full name. Using WRatio, we chose the highest matching character that received a score higher than T hreshold (line 6). 4. In fact, we were able to overcome many of these problems by using hearingimpaired subtitles. In many hearing-impaired subtitles, the name of the speaking character is part of the text. This property allowed us to avoid most the problems we described earlier and gain additional information.
For instance, the movie The Matrix has a scene in which Morpheus calls Neo, and we can know this only because of the tag [PHONE RINGS]. Afterward, there is an annotation "MORPHEUS:" which tells us that Morpheus is the one calling. Without this annotation, we could not know who is on the other end of the line (see Figure 3). 5. Using the matched characters, we created a link between characters u and v if they appeared in the movie in a time interval less than t seconds. For each such appearance, we increased the weight w between u and v by one.
Since in subtitles we do not have an indication of when each scene begins and ends, we used a heuristic to model the interaction between characters. We assumed that two characters who appear one after another in a short period of time probably relate. For example, in Figure 2 we have part of the subtitles from the movie The Matrix. Morpheus introduces himself to Neo, and we know that Morpheus and Neo are talking within an interval of 5 seconds. If t is larger than 5 seconds, we increase the link weight between Morpheus and Neo by one.
6. To reduce the number of false positive edges, we filtered all the edges with weight lower than w min . There were two main reasons for the formation of edges that did not exist in the movie. The first case was when we matched an entity to the wrong character. The second case happened when in the interval of t seconds there was more than one scene. These kinds of false positive links add noise to the graph. Most of these links have a very low weight; hence, filtering edges with weight lower than w min helps remove false positive links.

Datasets
To evaluate and test our movie social network construction algorithm described above on real-world data, we assembled large-scale datasets of movie subtitles and movie character lists. In addition, we collected movie character lists from the IMDb (Internet Movie Database) website 4 and movie subtitles from 15,540 movies. Furthermore, we also used data from Bechdel test scores of 4,658 movies. In the following subsections, we describe in detail the datasets we used.

IMDb Dataset
To collect movie and actor data, we used IMDb, which is an online site that contains information related to movies, TV series, video games, etc [13]. IMDb data is contributed by users worldwide. It contains 5,487,394 titles from which 505,380 are full-length movies [14]. In this study, we used the official IMDb dataset. 5 From the IMDb dataset, which contains only a subset of the IMDb database, we mainly used movies' titles, crews, and ratings data.

Subtitle Dataset
To analyze movies' content, we decided to extract information out of subtitles. Subtitles are freely and widely available online on numerous sites. For instance, OpenSubtitles.org 6 alone hosts more than 500,000 English subtitles [16] that were manually created by the community. We collected the subtitles using Subliminal 7 , a Python library for searching and downloading subtitles. Subliminal downloads subtitles from multiple sources, and using an internal scoring method, it decides which subtitles are the best for a specific movie. Using Subliminal, we downloaded subtitles for 15,540 movies.

Bechdel Test Dataset
Bechdel test data is available at Bechdel Test Movie List 8 , which is a communityoperated website where people can label movies' Bechdel scores. Using the Bechdel Test Movie List API, we downloaded a dataset that contains 7,871 movies with labeled Bechdel scores, from which only 7,322 are full-length movies. Even for humans, it is a challenging task to determine if a movie actually passes the Bechdel test; Bechdeltest.com has a comments section where users discuss the scores and their disagreements [23]. For example, according to Bechdeltest.com, the movie The Dark Knight Rises failed the test. However, by taking a closer look at the community comments, 9 we noticed users arguing regarding the test results, which are hard to determine.

Dataset Preprocessing
The most critical part of building a social network of characters' interaction is mapping correctly between the characters in subtitles and the characters in the character list. The IMDb character data includes data on even the most minor roles such as a nurse, guard, and thug #1. These nameless minor characters are almost impossible to map correctly to their subtitle appearances. Usually, they just add false positive edges and do not add additional information.
To clean the data from nameless characters, we created a blacklist of minor characters (for a detailed explanation of the blacklist construction process see Section A.1). Additionally, to validate the characters' names we used TMDb (The Movie Database) 10 , another community-built movie database. For each character, we matched the IMDb and TMDb data by the actor name. Then, we compared the lengths of the character names and kept the longer one.

Network Features
To study gender bias in movies, we calculated five types of features: vertex features, network features, movie features, gender representation features, and actor features. Through the study, we analyzed how these features change over time. Additionally, we used these features to construct machine-learning classifiers. In the rest of this section, we supply the definitions of these features.
Vertex Features: For a given v ∈ V , a neighborhood is defined as a set of v friends, Γ(v). Following are the formal definitions of the vertex-based features: • Total Weight -the total weight of all the edges, which represents the number of character v appearances in the movie, • Closeness Centrality -the inverse value of the total distance to all the nodes in the graph. It is based on the idea that a node closer to other nodes is more central, is the shortest distance between v and u.
• Betweenness Centrality -represents the number of times that a node is a part of the shortest path between two nodes [25]. A junction (node) that is part of more paths is more central, σ(s,t) [25], where v = s = t, σ(s, t) is the number of shortest paths between s and t , and (s, t|v) is the number of those paths passing through some node v.
• Degree Centrality -a node that has a higher degree is considered more • Clustering -measures link formation between neighboring nodes, [43], where T (v) is defined as the number of triangles through vertex v where a triangle is a closed triplet (three vertices that each connect to the other two).
• Pagerank -a node centrality measure that takes into account the number and the centrality of the nodes pointing to the current node [25].
Network Features: • Edge Number -the number of edges in the network |E|.
• Vertex Number -the number of vertices in the network |V |.
• Number of Cliques -the number of maximal cliques in the network [25].
• Statistical Network Features -set of features which are based on the vertex features. From these features, we calculate statistical features for the entire network. We calculate the mean, median, standard deviation, minimum, maximum, first quartile, and third quartile.

Gender Representation Features:
• Gender Count -the number of actors of a specific sex in the movie, G c (G, Sex) = |{v|v ∈ V gender(v) = Sex}.
• Triangles with N Women -the number of triangles that contain N females and 3-N males, where N ∈ 1, 2, 3.
• Percent of Triangles with N Women -the percent of triangles that contain N females and 3-N males, where N ∈ 1, 2, 3.
• Females in Top-10 Roles -the number of females in top-10 roles ordered by PageRank.
• Male Count -the number of male actors in the movie.
• Female Count -the number of female actors in the movie.
Movie Features: • Release Year -the year when the movie was first aired.
• Movie Rating -the rating the movie has on IMDb.
• Runtime -the movie total runtime in minutes.
• Genres -the movie genre by IMDb.
• Number of Votes -number of votes by which the rating was calculated on IMDb.
Actor Features: • Actor Birth Year -the year the actor was born.
• Actor Death Year -the year the actor died.
• Actor Age Filming -the age of the actor when the movie was released (ReleaseY ear − ActorBirthY ear).

Network Feature Analysis
To examine the state of the gender gap, in movies generally and by genre in particular, we analyzed only the most popular movies (movies which had more than n votes on IMDb). To answer our first research question -if there are genres that do not show a gender gap (see Section 1) -we calculated vertex and actor features (see Section 3.4.1) for all the roles. Next, we split the data by gender and movie genre. Finally, we utilized a Mann-Whitney U [39] test on these features to check if there are statistical differences between the male and female roles in different genres.
To study relationships in movies, and to answer our second question regarding what relationships reveal about gender, we calculated all the relationship triangles in the network and grouped them by the number of women in each triangle. Afterward, we segmented the triangles by genres and how they changed over time.
To investigate the role of centrality by gender, our third research question regardong the centrality of female roles, we calculated PageRank for the nodes in all our movie networks. We analyzed the number of men and women in the top-10 characters in movies and examined how this number has changed over the years.

Constructing the Gender Classifier
To explore the difference in representation of men and women in the film industry, we trained a classifier that predicts the character's gender using the role data (vertex and actor features) and movie data (IMDb features without male and female count). The base hypothesis is that if men and women are represented similarly, we should not be able to distinguish between male and female roles. However, if we can predict the character's gender, then we can learn which features are the most descriptive.
To build the dataset, we collected data from the movies with the most popular actors. We defined a popular actor as an actor who had participated in at least n movies with more than m votes on IMDb. We wanted to avoid analyzing niche movies and unknown actors with only minor roles that would not have meaningful features.
For each actor's movie appearance, we extracted the features described above (see Section 3.4.1). Since we had data on each actor's movie participation from multiple movies, to evaluate the model we had to split all the data on the same actor into either the training set or the validation set. To solve this issue for the evaluation, we have used leave one group out cross-validation. Hence, each time we left out one actor and all their rows in the dataset.
To create a ground truth for the classifier, we had to determine each actor's gender. For most of the characters, we extracted the gender from IMDb similarly to [26]. IMDb has an attribute of "actor" or "actress," which allowed us to identify gender. As we mentioned earlier, the IMDb dataset is only partial, so to overcome this issue we used a dataset that maps the first name to the gender. 11

Constructing the Bechdel Test Classifier
As we described in Section 2, the Bechdel test is used to assess how fairly women are represented in a movie. The test has three criteria: 1. Are there at least two named women in the movie? 2. Do the women talk to each other?
3. Do the women talk about something other than men? These criteria are hierarchical; hence, if a movie passes the last test, it has passed all of the tests.
To train the classifier, we extracted all the network, vertex, and gender representation features (see Section 3.4.1). For testing the trained model, we used the n newest movies in the Bechdel test dataset. The rest of the movies were used as the training set. Additionally, to standardize metrics such as AUC, we evaluated the classifier performance by the Precision at k (P@k) metric. P@k presents how many of the results the classifier is confident it classified correctly.
To answer the fourth research question regarding the fairness of female representation, we analyzed the change in the average probability of a movie passing the Bechdel test over time. Finally, we analyzed the change over time by genre.

Results
To analyze the gender gap in the film industry, we analyzed subtitles of movies that had at least 1,000 votes on IMDb. This resulted in a dataset containing 15,540 movies, which is a dataset 20 times bigger than the largest movie dataset currently available [34].
First, we predicted the actor's gender from movie and vertex features. We evaluated our classifier using leave-one-group-out cross-validation and found that we could predict gender by roles significantly better than random. Our classifier achieved an AUC of 0.83, and precision and recall of 0.72 and 0.77, respectively. We also examined which features were the most important (see Table 1); in other words, these features are more biased.
Second, we analyzed the gender gap, in general, and by genres, in particular (see Tables 6-10). We found that the genre with the largest number of features that are distributed similarly between men and women is Musicals. In musical movies, 9 out of 10 features distribute similarly; only the clustering coefficient distributes differently between men and women. In terms of features, Total Weight is the feature that distributes most similarly between the genders, with 9 out of 21 genres distributing the same. On the other side of the scale, Age Filming is the feature that distributes least similarly, with 0 out of 21 genres distributing similarly. 14 Third, to examine relationships among characters, we analyzed relationship triangles in the networks. We found that most triangles have three men, and triangles with three women are the least common (see Table 2). Out of 21 genres, in 8 genres the most common type of triangle is 3 men (without any women) and in all the others it is 2 men and a woman. According to the results, Romance is the genre with the most interaction among women and War is the genre where women have the least interaction. Inspecting the change in the number of triangles over time (see Figure 4), we can observe that in many genres there is an equalizing improvement over the years, but there are genres like Sport without a significant change. Fourth, we analyzed how characters are ranked in terms of centrality (see Tables 3a and 3b). We found that among central roles, there are considerably more men than women. For example, men have about twice the roles that ranked in the top-10 most central roles than women. In all top-10 most central roles, the female percentage is the same except for the most central role.
Fifth, we analyzed the gender composition of the top-10 central roles in movies (see Figure 5). We discovered that most of the movies have more men in central roles than women. Moreover, from the data, we can observe that there are almost no movies with no men and 10 women in the top-10 roles. Also, there are a considerable number of movies where the majority of the top-10 most central roles are men. Sixth, we wanted to observe how the percentage of women in top 1, 3 and 10 most central roles has evolved over time. We analyzed the change in this metric over almost a century (see Figure 6). It can be seen from the network that there is a constant rise in the number of women in top-10 most central roles.
Finally, to create an automatic classifier that can assess the fairness of female representation in movies, we created the Bechdel test classifier. Our classifier achieved an AUC of 0.81. To check real-world usage of the classifier, we calculated precision at K (P@K) (see Figure 9). We can observe that in the first 200 movies in the validation set, we achieved high precision above 0.9. We also   inspected which feature was more important (see Table 5). Seven of ten features were triangle-based features. Moreover, all the features in the table are a subset of the Gender Representation Features (see Section 3.4.1) Next, we trained our automated Bechdel test classifier on all the labeled data and calculated the average probability of the classifier by decade on all the unlabeled data (see Figure 7). We can see that there is a trend of growth. Also, we examined how the probability changed by genres (see Figure 8). Comparing our results to Agarwal et al. [23] (see Table 4), we found that our classifier performs better than Agarwal's in terms of F1 score.

Discussion
In this study, we present a method that converts movie subtitles into social networks, and we analyze these networks to study gender disparities in the film industry. Using this method, we created the largest available corpus of movie character social networks. The method and the corpus are available for use by other researchers to study additional movies and even TV shows, and it has the potential to revolutionize the study of filmed media. The ability to classify the gender of characters based on features unrelated to gender demonstrates that there is enough of a difference between male and female roles that they can be distinguished. We can see that the most important feature for gender classification is the actor's birth year. This result corresponds to previous studies [31] that have found that female actors, on average, are much younger than male actors. Moreover, the Mann-Whitney U test results show that the only feature in films that is consistently different between men and women over all genres is age.
When looking at relationship triangles, we can see that in 77% of all triangles men are in the majority. In an equal society, we would expect to find that the number of triangles with three men, with three women, and with two men and  two women would be the same. However, we discovered that, on average, there are 11.4 times more triangles with three men than with three women, and almost twice as many triangles with two men than two women. At a deeper level of granularity, we can see a difference in the number of triangles between different movie genres. The Romance genre has the highest number of triangles that have two and three women. On the other side of the scale, 90.6% of triangles in the War genre have a majority of men. This result makes sense intuitively. By looking at Figure 10, we can see that genres with a higher percentage of movies that pass the Bechdel test also have a higher percentage of triangles with a majority of women.
In terms of centrality (see Table 3a), we can see that men have more central  roles than women. We expected to find more females in less central roles, but the percentage of females distributes evenly in the top-10 most central roles. We believe that these results correspond to the total percentage of women in the dataset, which is 32.3%. This number is still lower than the total percentage of female roles in IMDb, which is 37.2%. We also analyzed how many roles in a movie's top-10 most central roles are those of women. Unsurprisingly, there is a dominance of movies with a majority of men. For instance, all Lord of the Rings movies have 10 men in the top-10 roles. We found only 5 films where all top-10 roles were female, and each of these featured only women (one of these films is called The Women, another movie Caged is about a women's prison, and the movie The Trouble with Angels is about a girls' school). We also presented an automated Bechdel test classifier that can help assess the fairness of how women are presented in movies. We trained our model on data collected from bechdeltest.com, and we have indications that our model is even more accurate than the above presented results. We found that many movies on bechdeltest.com are misclassified. For example, The Young Offenders passes the test on bechdeltest.com (although the site does state this result is 'dubious'), but our work classifies it as a fail. The reverse is true for the movie Never Let Go. Based on these observations and on the P@K metric, we believe that our classifier can automatically classify movies with high confidence in the classification. Moreover, while the Bechdel test is certainly a useful and important test, it fails to account for many parameters such as the centrality of the characters, repression, etc. Basically, if there is a movie with only two women who appear in one scene and talk about something other than men for 2 seconds, then the movie will pass the traditional Bechdel test. However, this is the only test that has data that can be used to train a classifier. Our classifier partially tackles this problem since it calculates a score of how strongly the movie passes the test.
To truly solve the issues of the Bechdel test, a better test should be created. We believe that a good test can be created by comparing the number of interactions according to each gender. Hence, we propose an interaction test by comparing the total degree of male and female nodes in our movie social networks. In only 16.7% of movies do female characters have an equal or higher total degree than male characters. Moreover, in 55.8% of movies the total degree of male characters is at least twice as high as female characters. We think that a good rule of thumb for a movie should be 0.8 < T otalDegree F T otalDegree M < 1.2. Such a test would not be male nor female biased; sadly today only 12% of all movies pass this test. In future work, we are planning to perform statistical tests to compare the distributions of the degrees of male and female nodes and present a more accurate test.
We also calculated the average probability of passing the Bechdel test for all the movies in our dataset that do not have a Bechdel test score. Afterward, we inspected the change in the average probability of movies passing the test over a long period of time and by different genres. In almost all genres there is a trend of improvement, and there is a correlation between relationship triangles and the Bechdel score. Looking at Figure 8, we see that historically war movies have the lowest probability of passing the Bechdel test.
As we described in Section 2, there are several ways to analyze movie content. The first option is to analyze by watching movie videos as Weng et al. did. [47]. The problem with video analysis is that it is an expensive process which requires high computational power, especially when the plan is to analyze thousands of full-length movies. Moreover, most movies are copyrighted and not freely available online. The second option is to use speech recognition to extract information, which is what Park et al. did. [41]. However, this option has similar drawbacks. The third option is to utilize movie scripts, as has been done in several similar studies [23,30]. Scripts have a lot of information, such as character names, scenes, etc. However, currently the Internet Movie Script Database (IMSDb) 12 is relatively small and only has 1198 scripts. We decided to use movie subtitles for this study since subtitles are free and widely available.
From manually inspecting our movie social networks, we found them to be precise; however, there are many factors that affect our method's accuracy. The most critical factor is the quality of both the subtitles and the cast information from IMDb. In movies where the name of the character in the subtitles does not correspond to IMDb data, the actor cannot be linked to a character. During our study, we stumbled upon subtitles with spelling mistakes and other inconsistencies. Also, in some movies like superhero movies, we did not know how to link the different identities of a character with names such as "Captain America," that potentially could be filtered because it looks like a nameless character. In addition, nameless characters like "Street Pedestrian" sometimes eluded our cleaning process. There is a balance between cleaning the IMDb data too much and not enough. We observed that more accurate networks were in movies that had hearing-impaired subtitles since they have additional data and are less affected by the NER accuracy. Some of these limitations will be addressed in future research.
Our results demonstrate that a gender gap remains in nearly all genres of the film industry. For instance, 3.5 times more relationship triangles in movies have a majority of men. In terms of top-10 most central movie roles, again there is a majority of men. However, we also saw an improvement in equality over the years. Today, women have more important movie roles than in the past, and our Bechdel test classifier quantifies this improvement over time by calculating a movie's overall score. In a future study, we plan to analyze TV series, actors' careers, and directors' careers in a similar in-depth manner. We also plan to implement the tests that were proposed in [4] as well as develop new tests to gain further insight into how genders are represented in the film industry.

Data and Code Availability
This study is reproducible research. Therefore, the anonymous versions of the social network datasets and the study's code, including implementation, are available on the project's website 13 and repository 14 .
5. Finally, we removed all character names that appeared only once. Generic character names like Mom, Dad, Policeman, etc., appear in multiple unrelated movies. A character name that appears only in one film has a higher probability of being an important character.
6. All the remaining character names were blacklisted.