Data from a pre-publication independent replication initiative examining ten moral judgement effects

We present the data from a crowdsourced project seeking to replicate findings in independent laboratories before (rather than after) they are published. In this Pre-Publication Independent Replication (PPIR) initiative, 25 research groups attempted to replicate 10 moral judgment effects from a single laboratory’s research pipeline of unpublished findings. The 10 effects were investigated using online/lab surveys containing psychological manipulations (vignettes) followed by questionnaires. Results revealed a mix of reliable, unreliable, and culturally moderated findings. Unlike any previous replication project, this dataset includes the data from not only the replications but also from the original studies, creating a unique corpus that researchers can use to better understand reproducibility and irreproducibility in science.

campaign, but $115 million helped a great many needy people and perhaps the company received some credit for that.
This study tested the moral inversion hypothesis that charitable acts are nullified when companies spend more money promoting their donation activities than on the actual donation amount. The weak version of the moral inversion hypothesis predicts that self-promotion cancels out charitable acts; the strong version predicts that exploiting charitable acts is perceived even more negatively than making no charitable contribution at all.

Methods
One hundred thirty participants (64% female; M age = 34) (REPLICATION: 3133 participants, 53.8% female, M age = 26.51, SD = 11.05) were recruited from Amazon.com's Mechanical Turk (MTurk) service in return for a small cash payment. Participants were randomly assigned to one of four between-subjects conditions: charity only, publicized charity, charity + furniture advertising, or no contribution. Data were not analyzed until after data collection had terminated, no participants were excluded for any reason, and all conditions and dependent measures are described below in full.
Participants in the charity only condition read that Farrell Incorporated, a large home furnishing company, recently donated $200,000 to support research on cancer. In the publicized charity condition, Farrell Incorporated donated $200,000 to cancer research and subsequently spent $2 million publicizing its charitable contribution. In the charity + furniture advertising condition, the company donated $200,000 for cancer research and subsequently spent $2 million to advertise its furniture. In the no contribution condition, the company did not donate any money to charity (thus serving as a baseline/control condition).
Comprehension check items asked "Did the company donate money to cancer research?" (1 = Yes,2= No) and "Did the company also spend money on an advertising campaign about its donation for cancer research?" (1 = Yes,2= No). However no participants were removed from analyses based on their responses to these items (REPLICATION: did not include these items).
Finally, we asked participants to report their age, political orientation (1= very liberal, 7 = very conservative), gender, and nationality.
These scenarios and questionnaire items are provided at the end of this study report. The original data collection occurred in 2009, and in 2014 we noticed three items of unclear origin in the datafile (labeled "friends" "sweater" and "taxes") that used a different scale (-3 to +3) from the moral evaluations and trust DVs, and more importantly were not in the word version of the materials we had on file. These items appear to have been added in at the last minute and then forgotten entirely. In sum, a company that aggressively advertised its charitable acts not only squandered the good will it might have earned, but was judged even more harshly than a company that made no

--Michael Kinsley
Consider the case of a scientist who runs a study, then deletes the 95% of the sample that failed to support the research hypothesis. Clearly this is scientific fraud. But what about the case of a scientist who runs 20 very similar studies, then reports only the one that worked? Not only is this is not legally fraud, it is not necessarily even grounds for a correction to the publication. Yet, the actual truth value of the published work would seem to be equally nil in the two cases.
The difference, it seems, lies not in objective truth value, but in the underlying intentions of the agent. The former agent knowingly acted nefariously; the latter could have engaged in psychological rationalizations but acted with legitimate scientific goals in mind (e.g., fine-tuning the experimental paradigm). The present research explored whether there is a "moral cliff" of unambiguously bad intentions beyond which agents are seen to condemn themselves irrevocably.
Perhaps even more interestingly, just short of the cliff's edge behaviors that are in many respects just as objectively damaging can be treated with paradoxical leniency.
This initial study examined whether a moral cliff exists in the domain of false advertising. We tested the hypothesis that a cosmetics company that Photoshopped the model in its advertisement would be judged much more harshly than a company that simply hired a more attractive model (eliminating the need to digitally enhance her appearance). The effectiveness of the cosmetics would seem to be equally misportrayed in the Photoshopped and non-Photoshopped advertisement. Yet only the digitally manipulated ad, we argue, stumbles across the moral cliff.

Participants and Design
One hundred and fourteen participants (REPLICATION: 3592, 55.1% female, M age = 24.99, SD = 9.62) were recruited from Amazon.com's Mechanical Turk (MTurk) service and took part in the study in return for a small cash payment. The study employed a 2 (Photoshop vs. control) x 2 (counterbalancing order of the two scenarios) design, with the first factor manipulated within-subjects and the second factor between-subjects. Data were not analyzed until after data collection had terminated, no participants were excluded for any reason, and all conditions and dependent measures are described below in full.

Material and Procedures
Scenarios. All participants respond to the two target scenarios in counterbalanced order.
In the Photoshop scenario, a cosmetics company hired a model to appear an advertisement for their skin cream. The model was one in a thousand in terms of the beauty of her skin. An artist who worked for the cosmetics company then used Photoshop to make her skin appear "one in a million." In the control scenario, the company hired a model who already looked one in a million in terms of the beauty of her skin.
Accuracy. Participants were asked how accurately the company's advertisement portrayed the effectiveness of their skin cream (1= extremely inaccurately 7= extremely accurately) and whether the ad created a correct impression regarding the product (1= extremely incorrect 7= extremely correct). These items formed a reliable index in both the control and Photoshop conditions (α Control = .87 and α Photoshop = .78) (REPLICATION: α Control = .86 and α Photoshop = .76).
Dishonesty. Three items asked whether the ad was dishonest (1= not at all dishonest, 7 = extremely dishonest), fraudulent (1= not at all fraudulent, 7 = extremely fraudulent), and a case of false advertising (1= definitely false advertising, 7 = definitely truthful advertising) (reverse Intentionality. An item asked if the company had intentionally misrepresented their product (1= definitely not, 7 = definitely yes).

Rationalizability.
A further item assessed how easy it was for the company to justify their behavior to themselves as legitimate (1 = extremely difficult, 7 = extremely easy). We had hoped this would form a reliable "bad faith" index with the intentionality item, but as responses to the two items were practically uncorrelated (r Control = -.04 and r Photoshop = -.11) (REPLICATION: In sum, a company that digitally manipulated its advertisement was judged more harshly than a company that simply hired a more beautiful model. The Photoshopped ad was perceived as guided by a deliberate intent to deceive, as fraudulent, and grounds for punishing the company through fines and a ban on its advertisement. Contrary to predictions, participants did not even acknowledge that hiring a model who already had perfect skin portrayed the effectiveness of the skin cream just as inaccurately as digitally manipulating a model to appear to have perfect skin. Although speculative, this could be a case of belief overkill (Baron, 2009;Jervis, 1976) or moral coherence (Liu & Ditto, 2012), in which moral condemnation of the deceptive company distorted perceptions of their advertisement's objective truth value. Future studies will examine this possibility empirically, and test the moral cliff hypothesis in domains such as academic misconduct and accounting fraud.
NOTE: Participants respond to both scenarios in counterbalanced order, completing the same dependent measures twice.

PHOTOSHOP CONDITION
A cosmetics company hires a model to appear in an advertisement for their skin cream. She is one in a thousand in terms of the beauty of her skin. An artist who works for the cosmetics company then uses Photoshop to make her skin appear one in a million in terms of beauty. The skin cream advertisement with the model appears in magazines and on billboards all over the world.

CONTROL CONDITION
A cosmetics company hires a model to appear in an advertisement for their skin cream. She is one in a million in terms of the beauty of her skin. The skin cream advertisement with the model appears in magazines and on billboards all over the world.

DEPENDENT MEASURES
How accurately or inaccurately does the company's advertisement portray the effectiveness of their skin cream? This study examined whether concerns about unfairness predict the perceived material consequences of economic variables. Such a correlation would raise the possibility that people perceive certain economic variables as bad for the economy because they are unfair-in other words, that moral concerns distort logically unrelated perceptions of economic processes. Such a distortion effect with regards to economic beliefs would constitute an interesting case of the moral general phenomenon of moral coherence, in which factual beliefs shift to fall in line with moral evaluations (Liu & Ditto, 2012).
Notably, the Survey of Americans and Economists on the Economy (SAEE) reveals some interesting differences between laypeople and economists when it comes to perceived economic effects (Blendon et al., 1997;Caplan, 2001Caplan, , 2002 Demographics. Participants further reported demographic characteristics including political orientation (1 = very liberal, 7 = very conservative), gender, nationality, and the number of economics classes they had taken. The complete study materials are provided at the end of this report.

Results and Discussion
As expected, participants viewed variables that violated common sense notions of fairness (e.g., high corporate salaries) as bad for the economy. Indeed, as seen in column two of Pre-Publication Independent Replication (PPIR) 39 the Table, the zero order correlation between perceived fairness and economic effects was   significant for all 21 variables taken from the SAEE (REPLICATION: same result).
The causal influence could of course go in either direction-i.e., from perceived economic effects to fairness, or from perceived fairness to economic effects. Because our theoretical interest is in the latter possibility, in subsequent analyses for each variable all participants who indicated that their judgments of fairness were based on economic effects were removed from the sample. Only participants who indicated a 5, 6, or 7 on the relevant "in principle" item remained in the analysis (REPLICATION: this measure not included, so this analysis was not done). For these remaining participants, it is comparatively more likely that assessments of fairness distort perceived economic effects. Notably, even participants who met this criterion exhibited positive correlations between their assessments of fairness and economic effects (see Table, column 3). In sum, participants clearly viewed economic variables that violate common sense notions of fairness as also bad for the economy. This is consistent with the idea that perceived unfairness shapes assessments of economic effects, and more generally with the phenomenon of moral coherence (Liu & Ditto, 2012). However, the evidence from the present study is correlational and therefore cannot identify causal relationships.    ---------------------   ---------------------          ---------------------     This study examined whether a positive reputation and laudable goals can cause an organization and its leader to be held to a higher standard, leading to more severe censure for moral transgressions. Specifically, even minor inappropriate expenses by the leader of a charity may be morally condemned and viewed as a violation of trust . Trust violations undermine the conviction the world is a just and orderly place and thus represent both a threat to the social order and a psychological threat (Koehler & Gershoff, 2003). We therefore investigated whether frivolous perks accorded to the leader of a charity would lead participants to feel the world is unstable, chaotic, and unfair.

Methods
Two hundred sixty five participants were recruited from Amazon.com's Mechanical Turk (MTurk) (REPLICATION: 2888 participants) service in return for a small cash payment. The study utilized a 2 (type of organization: charity or company) × 3 (requested compensation: small perk, large perk, or cash only) between subjects design. Data were not analyzed until after data collection had terminated, no participants were excluded from the analyses, and all conditions and dependent measures are described below in full.
Scenario. Participants read that an organization was deciding between two job candidates for a top management position. The two candidates, henceforth referred to the target candidate and control candidate, had comparable backgrounds and employment histories, and this information was counterbalanced across participants. The names of the candidates ("Lisa" and "Karen"; two names equated for a number of connotations by  were also counterbalanced.
All candidates in all conditions requested compensation packages of the same total financial value. The only difference was that in some conditions, the target candidate requested a perk of a certain value as opposed making an equivalent salary request. In the large perk condition, the target candidate requested a chauffeured limousine on weekends. In the small perk condition, the target candidate requested expensive mineral water. We further manipulated the type of organization in question. In the company condition, the organization was called "The Jens Shoes Corporation." In the charity condition, the organization was called "Somalian Hunger Relief." Candidate evaluations. After reading the scenario, participants were asked whether a series of characteristics was more true of Lisa or Karen on a scale ranging from 1 (definitely Lisa) to 7 (definitely Karen). Participants rated the candidates in terms of their responsibility, moral character, selfishness, and willingness to act in the best interests of the organization. In the company condition they further indicated who they would invest money with, and in the charity condition who they would donate money with. In all conditions they reported who they would prefer to see hired. These items were adapted from .
Candidate evaluations along these dimensions were highly correlated and (after reverse scoring the selfishness item) were averaged into a reliable composite (α = .91) (REPLICATION: α = .92).
Informational value. Two items assessed the perceived informational value of each candidate's request (see also . These items asked how much each Pre-Publication Independent Replication (PPIR) 53 person's requested compensation "tell you about who she really is and what she is really like" (1 = nothing, 7 = a great deal) (REPLICATION: not included).
Evaluations of organization. Next participants were told to imagine that the organization had decided to hire the target candidate. They then evaluated the organization on seven-point scales on the dimensions bad-good, unfavorable-favorable, and negative-positive (α = .94) (REPLICATION: not included).

Trust in organization.
On similar seven-point scales, participants further reported whether they felt the company was trustworthy, dependable, and reliable (α = .86) (REPLICATION: not included).
Betrayal. A further item read "I feel betrayed by the organization's choice for President" (1 = strongly disagree, 7 =strongly agree). We had originally intended for this betrayal item to be part of the trust in organization index, but it only correlated weakly (r = -.33) with the other items and was therefore analyzed separately. It is unclear whether the weak correlation is due to the betrayal item being more strongly worded that the other trust items, or negatively worded (REPLICATION: not included).
Petition item. A stand-alone item read "I would sign an online petition to display my support for the organization" (1 = strongly disagree, 7 = strongly agree) (REPLICATION: not included).
Social threat. Items adapted from Koehler and Gershoff's (2003) social threat measure asked participants whether each candidate being chosen would lead them to feel the world is an unfair, disorderly, and uncertain place (1 = strongly disagree, 7 = strongly agree). These Pre-Publication Independent Replication (PPIR) 54 measures proved reliable for both the control candidate (α = .95) and target candidate (α = .94) (REPLICATION: not included).
Attention checks. Follow-up items asked participants if they had engaged in other activities during the survey and if they had read the instructions. However no participants were removed from the analyses based on their responses to the attention check items.
Demographics. Participants reported demographic characteristics including their age, political orientation, gender, and nationality.
Comprehension checks. Finally, participants were asked to recall whether the organization was a company or charity and whether a candidate had requested a perk. However no participants were removed from the analyses based on their responses.
The full study materials are provided at the end of this report.

Results and Discussion
Candidate evaluations. For ease of analysis and presentation, all candidate evaluation items were recoded such that positive scores reflected positive evaluations (and negative scores reflected negative evaluations) of the target candidate relative to the control candidate. An ANOVA revealed the hypothesized interaction between the type of organization (company vs.  However the main effect of pay did not reach significance separately for either the company, F(2, 136) = 2.00, p = .14, or the charity, F(2, 122) = 2.56, p = .08 (REPLICATION: not included). Social threat. As the control candidate's compensation did not vary by condition, our theoretical hypotheses related only to feelings of threat elicited by the target candidate's compensation. The expected interaction between type of organization and compensation did not reach significance when it came to feelings of social threat caused by the target candidate, F(2,  In sum, some noteworthy differences emerged in the reputational consequences of frivolous perks when it came to the leader of a company versus a charity. Participants tolerated a comparatively small perk (i.e., expensive mineral water) in the case of a corporate leader, but balked at a large one (i.e., a chauffeured limousine). In contrast, for the head of a charity, even a small perk was regarded very negatively: the expensive mineral water elicited perceptions of a charitable organization's leader that were just as negative as a chauffeured limousine. Moreover, granting a top leader a frivolous perk was seen as a trust violation only for the charity. Reading that a charity had agreed to provide its leader with expensive mineral water further elicited feelings of social threat (Koehler & Gershoff, 2003).

COMPANY + CASH CONDITION
Instructions: Please read the hiring scenario below and then answer the questions.
The Jens Shoes Corporation is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a sneakers company. She was promoted after developing successful partnerships with several shoe companies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an online shoe company. She was promoted after designing a new capital campaign that raised significantly more investments than her predecessor. As part of her proposed contract, Karen is asking for a salary of $400,000.

COMPANY + LARGE PERK CONDITION
Instructions: Please read the hiring scenario below and then answer the questions.
The Jens Shoes Corporation is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a sneakers company. She was promoted after developing successful partnerships with several shoe companies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an online shoe company. She was promoted after designing a new capital campaign that raised significantly more investments than her predecessor. As part of her proposed contract, Karen is asking for a salary of $350,000 plus $50,000 per year for rental of a chauffeurdriven limo on the weekends.

Pre-Publication Independent Replication (PPIR) 63
Instructions: Please read the hiring scenario below and then answer the questions.
The Jens Shoes Corporation is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a sneakers company. She was promoted after developing successful partnerships with several shoe companies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an online shoe company. She was promoted after designing a new capital campaign that raised significantly more investments than her predecessor. As part of her proposed contract, Karen is asking for a salary of $395,000 plus $5,000 per year for luxury water flown from Sweden.

CHARITY + CASH CONDITION
Instructions: Please read the hiring scenario below and then answer the questions.
The Somalia Hunger Relief Charity is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a children's non-profit. She was promoted after developing successful partnerships with several international charity agencies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an advocacy non-profit. She was promoted after designing a new fundraising campaign that raised significantly more donations than her predecessor. As part of her proposed contract, Karen is asking for a salary of $400,000.

CHARITY + LARGE PERK CONDITION
Instructions: Please read the hiring scenario below and then answer the questions.
The Somalia Hunger Relief Charity is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a children's non-profit. She was promoted after developing successful partnerships with several international charity agencies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an advocacy non-profit. She was promoted after designing a new fundraising campaign that raised significantly more donations than her predecessor. As part of her proposed contract, Karen is asking for a salary of $350,000 plus $50,000 per year for rental of a chauffeur-driven limo on the weekends.

CHARITY + SMALL PERK CONDITION
Instructions: Please read the hiring scenario below and then answer the questions.
The Somalia Hunger Relief Charity is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a children's non-profit. She was promoted after developing successful partnerships with several international charity agencies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an advocacy non-profit. She was promoted after designing a new fundraising campaign that raised significantly more donations than her predecessor. As part of her proposed contract, Karen is asking for a salary of $395,000 plus $5,000 per year for luxury water flown from Sweden.

DEPENDENT MEASURES
Please use the scale below to indicate whether the following characteristics are more true of Lisa or Karen. 1  2  3  4  5  6  7 ___Who is a more responsible person?

Definitely Lisa Definitely Karen
___Who is probably a more morally upstanding human being? ___Who do you predict will make more responsible decisions as leader?
___Who do you predict will act in the best interests of the organization?
___Who is a more selfish person? How much does Lisa's requested compensation tell you about who she really is and what she is really like?
How much does Karen's requested compensation tell you about who she really is and what she is really like?
Please rate your agreement with the following statements

If Somalia Hunger Relief Charity [CHARITY CONDITION]/ Jens Shoes Corporation [COMPANY CONDITION] picked Karen as its President …
Please use the following questions to rate the organization: Please rate your agreement with the following statements using the scale provided below.

Strongly Disagree
Strongly Agree  Even publicly supported behaviors can send negative signals about an agent's moral character (e.g., "It's a dirty job, but someone has to do it"). Perhaps some praiseworthy actssuch as sacrificing innocents in order to save a greater number of lives-require people who are deficient in generally positive moral traits such as empathy (Uhlmann, Zhu, & Tannenbaum, 2013). This study tested for an act-person dissociation where people view one act as more praiseworthy than another, but also more revealing of negative character traits.

Participants and Design
Seventy-nine participants (REPLICATION: 3016 participants) were recruited using Mechanical Turk and took part in the survey in return for a small cash payment. The study featured a joint evaluation design in which participants read about two targets and evaluated them relative to one another. Pairing of names (Karen and Lisa) with the two targets (medical research assistant and pet store assistant) was counterbalanced between-subjects. Data were not analyzed until after data collection had terminated, no conditions or participants were excluded, and all dependent measures are described below in full. This study was run together in a packet with another study, but this particular study was always presented first.

Materials and Procedure
Scenario. Participants read about two target persons, "Karen" and "Lisa," two names identified by  as similar in intelligence, age, and other connotations. The medical research assistant was described as working in a center for cancer research. Her job was to expose mice to radiation to induce tumors, and then give them injections of experimental cancer drugs. The pet store assistant worked in a store for expensive pets. Her job was to give gerbils a grooming shampoo and then tie bows on them. The pairing of the names Karen and Lisa with the target descriptions was counterbalanced across participants.
Moral actions. Participants were asked "Whose actions make a greater moral contribution to the world?", "Whose actions benefit society more?", "Whose job is more morally praiseworthy?", and "Whose job duties make a greater moral contribution to society?" (1 = definitely Karen, 7 = definitely Lisa). Items were scored and aggregated so that lower numbers reflected viewing the medical research assistant's actions as more praiseworthy (α = .85) (REPLICATION: α = .87).
Moral traits. Participants also assessed who was more caring, coldhearted, aggressive, and kind-hearted (1 = definitely Karen, 7 = definitely Lisa). Items were scored and aggregated so that lower numbers reflected more positive trait attributions regarding the medical research assistant (α = .74) (REPLICATION: α .83).
Animal testing. Participants were also asked if testing cancer drugs on mice is morally wrong (1 = definitely wrong, 4 = not sure, 7 = definitely OK).
Comprehension check. To see if participants were paying careful attention to the scenario, we asked them to identify which of the two women worked in the pet store. However no participants were removed from analyses based on their responses to this item.
Demographics. Finally, participants reported their age, gender, ethnicity, and political orientation. The complete study materials are provided at the end of this report.

Results and Discussion
Responses on all outcome measures were tested against the scale midpoint of 4 (on a scale of 1-7) since participants made comparative judgments of Karen and Lisa. As expected, the medical research assistant's actions were seen as more praiseworthy than those of the pet store NOTE: A conceptual replication of this effect that used separate as opposed to joint evaluation was reported in a footnote by Uhlmann, Zhu, & Tannenbaum (2013).

INSTRUCTIONS: Please read the paragraphs about the individuals below and answer the questions that come after.
Karen works as an assistant in a medical center that does cancer research. The laboratory develops drugs that improve survival rates for people stricken with breast cancer. As part of Karen's job, she places mice in a special cage, and then exposes them to radiation in order to give them tumors. Once the mice develop tumors, it is Karen's job to give them injections of experimental cancer drugs.
Lisa works as an assistant at a store for expensive pets. The store sells pet gerbils to wealthy individuals and families. As part of Lisa's job, she places gerbils in a special bathtub, and then exposes them to a grooming shampoo in order to make sure they look nice for the customers.
Once the gerbils are groomed, it is Lisa's job to tie a bow on them.
Please use this scale for the following items: Definitely Karen Definitely Lisa 1 2 3 4 5 6 7 _____ Whose actions benefit society more? _____ Whose job duties make a more moral contribution to society? _____ Whose job is more morally praiseworthy? _____ Whose actions make a greater moral contribution to the world?
Who is more likely to have the following traits?
Definitely States (Blendon et al., 1997;Caplan, 2001Caplan, , 2002. Therefore, participants were asked to indicate the percentage of top corporate executives they believed would burn in hell (given hell exists).
Burn-in-hell estimates for corporate executives were compared with those from one positively regarded group (social workers) and an array of groups defined by immoral behaviors (e.g., car thieves, drug dealers, vandals).

Participants and Design
A hundred and fifty-eight students (REPLICATION: 3430 individuals) participated in the study. Participants were recruited from two dining halls at Yale University (45%) and public campus areas at Northwestern University (55%) and paid $2 for their time. Data were analyzed twice, first between the Yale and Northwestern data collections and then again after data collection was complete. No conditions or participants were excluded from the analyses, and all measures are described below in full.

Materials and Procedure
Who will burn in hell? Participants estimated the percentage of individuals from a variety of social categories who would burn in hell (given that hell exists). The categories were: social workers, drug dealers, shoplifters, non-handicapped people who park in the handicapped spot, top executives at big corporations, people who sell prescription pain killers to addicts, people who kick their dog when they've had a bad day, car thieves, and vandals who spray graffiti on public property.
Arguments for and against capitalism. As an exploratory measure, participants were further asked to provide free responses indicating the best arguments in favor of and against capitalism. The order in which the arguments and burn-in-hell measures appeared was different between the two samples (capitalism arguments were always first at Northwestern and always second at Yale) (REPLICATION: not included).

Demographic measures.
Participants were asked to report their religion, religiosity ( Because there were more liberal than conservative participants in our sample, we also examined burn-in-hell estimates selecting only participants who scored 5 or higher on our 1-7 point political orientation measure (i.e., true conservatives). While more lenient toward corporate executives than liberals were, conservatives did consider them (

Study Materials
Assume for a moment that hell exists. What percentage of people in the following categories would go to hell when they die? Acts of everyday racial bigotry may provoke moral outrage in large part because they are perceived as strong signals of poor character (Uhlmann, Zhu, & Diermeier, 2014; see also Pizarro & Tannenbaum, 2011;Uhlmann, Pizarro, & Diermeier, in press). In this study, participants evaluated either a CEO who was selectively rude only to Black employees or a CEO who was indiscriminantly hostile and rude to all of his employees. Our prediction was that participants would view the bigot as a worse person than the misanthrope, despite the fact that the misanthrope mistreated a greater number of people. We further expected that the bigoted CEO's behavior, compared to the misanthrope, would be seen as more informative about his moral character. Finally, we predicted that participants would express greater willingness to affiliate with the misanthrope than the bigot, and also that they would expect the misanthrope to act more prosocially than the bigot in future interactions.

Participants and Design
Forty-six participants (REPLICATION: 3040 participants) were recruited from Amazon's Mechanical Turk and took part in the study in return for a small cash payment. The study featured a simple joint evaluation design in which participants read about two targets and evaluated them relative to one another. Pairing of names (Robert and John) with the two targets (Bigot and Misanthrope) was counterbalanced between-subjects. Data were not analyzed until after data collection had terminated, no participants were excluded from the analyses, and all conditions and dependent measures are described below in full.

Materials and Procedures
Scenario. Participants were asked to give their impressions of two CEOs, "Robert" and "John," who worked at similar but different companies. John did not say "hi" or engage in friendly small talk with any of his employees. Robert always said "hi" and engaged in friendly small talk with his White employees, but not his Black employees. John and Robert were selected as names because they were identified by  as similar in intelligence, age, and other connotations.
After reading the scenario, participants responded to a series of relative evaluation items on seven-point scales ranging from 1 (Definitely John) to 7 (Definitely Robert)..

Person judgments.
To assess character-based judgments, participants were asked whether John or Robert was the more immoral and blameworthy person (α = .91) (REPLICATION: α = .75). Responses were coded so that lower numbers reflected relatively greater condemnation of the bigot's moral character.
Informational value. To assess how informative they found each behavior, participants were asked to determine which person's behavior "tells you more about their moral character" and "tells you more about their personality" (α = .68; items adapted from Tannenbaum et al., 2011) (REPLICATION: α = .43). Responses were coded so that lower numbers indicated that participants viewed the bigot's behavior as more informative than the misanthrope's.

Affiliation.
Participants were asked who they would rather have as a close personal friend, date their daughter, have as a co-worker, and whose unlaundered sweater they would Free responses. Participants were told "If you had a preference for either John or Robert, please briefly tell us why" and were provided with space to respond in their own words.
Comprehension check. We asked participants to identify which CEO was selectively rude to his employees, with the options Robert, John, and Neither provided. However no participants were removed from the analyses based on their answer.
Demographics. Finally, participants reported their age, gender, ethnicity, nationality, and political orientation. The complete study materials are provided at the end of this report.

Results and Discussion
Because all items involved providing relative evaluations of the two targets, average responses to each measure were compared against the scale midpoint of 4 (scales ranged from 1 Pre-Publication Independent Replication (PPIR) 87

Study Materials
NOTE: Pairing of names (Robert and John) with the bigoted vs. misanthropic targets was counterbalanced between-subjects.
Instructions: We would like to get your impressions about two CEOs, Robert and John, who work at similar but different companies.
John is a CEO at Company X. John does not say "hi" or engage in friendly small talk with any of his employees. When an employee says "hi", John never responds.
Robert is a CEO at Company Y. Robert always says "hi" and engages in friendly small talk with his White employees. But when an African American employee says "hi," Robert never responds.
( Our previous work finds that some acts are seen as strong signals of poor moral character even when the act itself is viewed as relatively benign Uhlmann, Pizarro, & Diermeier, in press). Minor acts of everyday incivility seem like a context in which individuals can communicate negative information about themselves without causing much material harm to others. We therefore expected that leaving a restaurant tip entirely in pennies would be seen as highly informative of poor character, even though the act would not be viewed as morally blameworthy in-and-of itself.

Participants and Design
We recruited a sample of 79 participants (REPLICATION: 3706 participants) from Mechanical Turk, who each completed the survey in return for a small cash payment. Data were not analyzed until after data collection had terminated, no participants or conditions were excluded for any reason, and all dependent measures are described below in full. The study featured two between-subjects conditions. We administered this study as part of a packet of several studies; participants always completed this particular study after first responding to another study.

Materials and Procedures
Scenario. Participants read about a restaurant patron named Jack who was satisfied with his meals and service. Given the bill, the expected tip would be $15. In the bills condition, Jack left $14 in bills, thus paying less than what was appropriate. In the pennies condition, Jack paid the full gratuity of $15 by leaving a bag of pennies.
Person judgments. To assess character-based judgments, participants were asked whether Jack was a disrespectful person, had a good moral conscience, was a good person, and was the type of person they would want as a friend (1 = Not at all,7 = Definitely). For the analyses, these items were coded such that higher scores indicated more negative person judgments (α = Demographics. Finally, participants reported their age, gender, ethnicity, nationality, and political orientation. All study materials are provided below this report.

Results and Discussion
Jack As expected, a person who paid the full tip with a bag of pennies was judged more negatively than a person who tipped less well but in bills. Tipping in pennies was also viewed as relatively more informative about moral character. However, a dissociation between and act and person judgments  in press) did not emerge, as the act of tipping in pennies was also seen as more blameworthy than tipping in bills. Although speculative, tipping in pennies might be seen as causing harm because it inconveniences and upsets the waiter or waitress, making the act itself morally wrong. Future research will examine this possibility, and explore moral judgments of everyday incivility in other contexts.

Belief-Act Inconsistency Study (Uhlmann, Tannenbaum & Diermeier)
Do people disapprove of moral hypocrisy? The answer seems to be a straightforward Yes. Many instances of hypocrisy, however, are conflated with behavior that we find unacceptable even when hypocrisy is absent. Take the example of a politician who prosecutes criminals only to engage in corruption himself, or a religious leader who chastises sexually impropriety from the church pulpit and is later discovered having sex with a prostitute. In such cases our moral reactions may reflect our genuine distaste for hypocrisy, or it may simply reflect distaste for corruption and the solicitation of prostitutes. This study examined whether people have a direct distaste for hypocrisy even when they find the underlying behavior perfectly acceptable.

Participants and Design
One hundred ninety two Northwestern students (REPLICATION: 3708 participants) took part in the study, and each participant was randomly assigned to one of three conditions ( The complete study measures are provided at the end of this report.

Results and Discussion
Consistent with a direct aversion to moral hypocrisy, we found a significant effect of Compared to the hunter who was an advocate for an unrelated charity (doctors without borders), the animal rights activist was seen as more blameworthy ( In sum, an animal rights activist who was caught hunting was seen as an untrustworthy and bad person, even by participants who believed that hunting was morally acceptable. This suggests that an inconsistency between a person's moral beliefs and behaviors may be sufficient to elicit moral condemnation, even when the behavior is not actually seen as immoral in-and-of itself. People, it appears, have a direct aversion to moral hypocrisy.

Pre-Publication Independent Replication (PPIR) 101
Footnote 1 In the unrelated study, participants were randomly assigned to read either about an accident caused by a reckless driver, an accident caused by a negligent company, or a control condition in which no accident occurred (see the study materials below this report). They then filled out thirteen word completions designed to measure the automatic accessibility of words related to lawsuits. Coding of the word stem completion measure was discontinued after the first 142 participants due to its poor psychometric properties.

ANIMAL RIGHTS ACTIVIST CONDITION
Bob Hill has worked for 20 years as an animal rights activist and president of the non-profit organization Furry Friends Forever (FFF), which advocates for the ethical treatment of domestic and wild animals. FFF works through public education, cruelty investigations, research, animal rescue, legislation, special events, celebrity involvement, and protest campaigns.
Recently, the Associated Press news service reported that Hill had participated in a wild game hunting safari in South Africa. The report indicated that this is the fourth big game hunting safari that Hill has done in the last five years. Below is a picture that accompanied the press release, showing Hill with a Kudu antelope that he shot down with a .338 Winchester Magnum hunting rifle.

BIG GAME HUNTERS ASSOCIATION CONDITION
Bob Hill has worked for 20 years as an avid hunter and president of the American Big Game Hunters Association (ABGA), which advocates for big game trophy hunting throughout North America and the world. ABGA serves the hunting community through the sharing of experiences, knowledge and technology, promoting the education of youth in securing the future of the hunting tradition, and extending the goodwill of members through community outreach.
Recently, the Associated Press news service reported that Hill had participated in a wild game hunting safari in South Africa. The report indicated that this is the fourth big game hunting safari that Hill has done in the last five years. Below is a picture that accompanied the press release, showing Hill with a Kudu antelope that he shot down with a .338 Winchester Magnum hunting rifle.

DOCTORS WITHOUT BORDERS CONDITION
Bob Hill has worked for 20 years as a human right activist and president of doctors without borders (DWB), which provides medical aid in nearly 60 countries to people whose survival is threatened by violence, neglect, or catastrophe, primarily due to armed conflict, epidemics, malnutrition, exclusion from health care, or natural disasters. DWB provides independent, impartial assistance to those most in need. DWB is committed to bringing quality medical care to people caught in crisis regardless of race, religion, or political affiliation.
Recently, the Associated Press news service reported that Hill had participated in a wild game hunting safari in South Africa. The report indicated that this is the fourth big game hunting safari that Hill has done in the last five years. Below is a picture that accompanied the press release, showing Hill with a Kudu antelope that he shot down with a .338 Winchester Magnum hunting rifle. Not at all Definitely (6) How do you feel about the activity of hunting wild (non-endangered) animals?

FRIVOLOUS LAWSUIT CONDITION
Instruction: Please read the paragraph below. Later you will be tested on your memory for it.
Tom Patton was recently driving at double the speed limit on the highway, steering his car with his feet and shooting up heroin. On a sharp bend, he failed to turn in time and crashed his car into the highway railing. The railing, manufactured by Highland Road Company, gave way and his car fell down a steep hill. Tom was left with severe neck and back pain and is now unable to keep his job.

LEGITIMATE LAWSUIT CONDITION
Instruction: Please read the paragraph below. Later you will be tested on your memory for it.
Tom Patton was recently driving his car on the highway at the speed limit. He was unable to turn in time on a sharp bend where there are frequent accidents and crashed his car into the highway railing. The railing, manufactured by Highland Road Company, gave way and his car fell down a steep hill. Tom was left with severe neck and back pain and is now unable to keep his job.

NEUTRAL CONDITION
Instruction: Please read the paragraph below. Later you will be tested on your memory for it.
Tom Patton was recently driving his car on the highway at the speed limit. He turned on a sharp bend. The railing on the highway at the sharp bend was manufactured by Highland Road Company. In the first scenario you read, please describe the type of organization that Bob belonged to:

WORD STEM ACTIVATION DV FOR LITIGIOUSNESS STUDY
In the second scenario you read, did Tom crash his car? (please circle one)

Yes No
In the second scenario you read, was Tom shooting up heroin while he was driving? (please circle one)

Yes No
How do you feel about protecting wild animals (please check one) People should only undertake this action if it leads to some benefits that are great enough.
People should do this no matter how small the benefits.
Not undertaking the action is acceptable if it saves people enough money. My parents' occupations are:

SUPPLEMENT 3: REPLICATION MATERIALS
This packet includes the following materials: between-subjects design with random assignment to one of the four conditions. Chicago, Ill., December 2, 2013 -The Locks Corporation, based in Rockford, Illinois, today was accused that several of their food products contain a substance known as Gloactimate, which may be harmful to people's health. Gloactimate is an additive in processed foods and is used to increase the shelf life of foods. A recent series of studies found that Gloactimate raises "bad" cholesterol, lowers "good" cholesterol, and increases risk for heart disease.

Corporate Response: The Company Allows an Independent Investigation
The Locks Corporation announced that it is confident in its adherence to government standards regarding Gloactimate and would allow independent investigators into any of their nationwide locations to test their products. The company emphasized that with food products in stores and warehouses throughout the country, there would be no feasible way the Gloactimate would go undetected.
An independent group of scientists from the Advanced Science Institute (ASI) has conducted an independent investigation. ASI formed a team of investigators that included physicians, nutritionists, chemists, health inspectors and several senior members of ASI. The Locks Corporation agreed to allow ASI access into any of its facilities. This group of scientists has concluded that the food from the Locks Corporation does not contain Gloactimate. Chicago, Ill., December 2, 2013 -The Locks Corporation, based in Rockford, Illinois, today was accused that several of their food products contain a substance known as Gloactimate, which may be harmful to people's health. Gloactimate is an additive in processed foods and is used to increase the shelf life of foods. A recent series of studies found that Gloactimate raises "bad" cholesterol, lowers "good" cholesterol, and increases risk for heart disease.

Corporate Response: The Company Allows an Independent Investigation
The Locks Corporation announced that it is confident in its adherence to government standards regarding Gloactimate and would allow independent investigators into any of their nationwide locations to test their products. The company emphasized that with food products in stores and warehouses throughout the country, there would be no feasible way the Gloactimate would go undetected.
An independent group of scientists from the Advanced Science Institute (ASI) has conducted an independent investigation. ASI formed a team of investigators that included physicians, nutritionists, chemists, health inspectors and several senior members of ASI. The Locks Corporation agreed to allow ASI access into any of its facilities. This group of scientists has concluded that the food from the Locks Corporation does contain Gloactimate.

NOTE: This is the "moral inversion study", condition *1* of 4. The study uses a betweensubjects design with random assignment to one of the four conditions.
Farrell Incorporated is a multi-billion dollar home furnishing company.
Farrell Incorporated is:

NOTE: This is the "moral inversion study", condition *2* of 4. The study uses a betweensubjects design with random assignment to one of the four conditions.
Farrell Incorporated is a multi-billion dollar home furnishing company.
Recently the company donated 200,000 dollars to a charity for cancer research.
Farrell Incorporated is:

NOTE: This is the "moral inversion study", condition *3* of 4. The study uses a betweensubjects design with random assignment to one of the four conditions.
Farrell Incorporated is a multi-billion dollar home furnishing company.
Recently the company donated $200,000 dollars to a charity for cancer research.
The company then spent 2 million dollars on an advertising campaign about its donation for cancer research.
Farrell Incorporated is: Immoral

NOTE: This is the "moral inversion study", condition *4* of 4. The study uses a betweensubjects design with random assignment to one of the four conditions.
Farrell Incorporated is a multi-billion dollar home furnishing company.
Recently the company donated 200,000 dollars to a charity for cancer research.
The company also spent 2 million dollars on an advertising campaign about its home furnishings.
Farrell Incorporated is: NOTE: This is the "higher standard" study. This is condition *1* of 6 between-subjects conditions. Please further note that the sixth DV item says "invest money" in conditions 1-3 and "donate" in conditions 4-6; thus the DV items are not perfectly identical across conditions.

Instructions: Please read the hiring scenario below and then answer the questions.
The Jens Shoes Corporation is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a sneakers company. She was promoted after developing successful partnerships with several shoe companies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an online shoe company. She was promoted after designing a new capital campaign that raised significantly more investments than her predecessor. As part of her proposed contract, Karen is asking for a salary of $400,000.
Please use the scale below to indicate whether the following characteristics are more true of Lisa or Karen.
___Who is probably a more morally upstanding human being? ___Who do you predict will make more responsible decisions as leader?
___Who do you predict will act in the best interests of the organization?
___Who is a more selfish person?
___Who would you invest money with? ___Who would you hire as President?
Pre-Publication Independent Replication (PPIR) 120 NOTE: This is the "higher standard" study. This is condition *2* of 6 between-subjects conditions. Please further note that the sixth DV item says "invest money" in conditions 1-3 and "donate" in conditions 4-6.

Instructions: Please read the hiring scenario below and then answer the questions.
The Jens Shoes Corporation is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a sneakers company. She was promoted after developing successful partnerships with several shoe companies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an online shoe company. She was promoted after designing a new capital campaign that raised significantly more investments than her predecessor. As part of her proposed contract, Karen is asking for a salary of $350,000 plus $50,000 per year for rental of a chauffeur-driven limo on the weekends.
Please use the scale below to indicate whether the following characteristics are more true of Lisa or Karen. 1  2  3  4  5  6  7 ___Who is a more responsible person?

Definitely Lisa Definitely Karen
___Who is probably a more morally upstanding human being? ___Who do you predict will make more responsible decisions as leader?
___Who do you predict will act in the best interests of the organization?
___Who is a more selfish person?
___Who would you invest money with? ___Who would you hire as President?

Pre-Publication Independent Replication (PPIR) 121
NOTE: This is the "higher standard" study. This is condition *3* of 6 between-subjects conditions. Please further note that the sixth DV item says "invest money" in conditions 1-3 and "donate" in conditions 4-6.

Instructions: Please read the hiring scenario below and then answer the questions.
The Jens Shoes Corporation is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a sneakers company. She was promoted after developing successful partnerships with several shoe companies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an online shoe company. She was promoted after designing a new capital campaign that raised significantly more investments than her predecessor. As part of her proposed contract, Karen is asking for a salary of $395,000 plus $5,000 per year for luxury water flown from Sweden.
Please use the scale below to indicate whether the following characteristics are more true of Lisa or Karen. 1  2  3  4  5  6  7 ___Who is a more responsible person?

Definitely Lisa Definitely Karen
___Who is probably a more morally upstanding human being? ___Who do you predict will make more responsible decisions as leader?
___Who do you predict will act in the best interests of the organization?
___Who is a more selfish person?
___Who would you invest money with? ___Who would you hire as President?
Pre-Publication Independent Replication (PPIR) 122 NOTE: This is the "higher standard" study. This is condition *4* of 6 between-subjects conditions. Please further note that the sixth DV item says "invest money" in conditions 1-3 and "donate" in conditions 4-6.

Instructions: Please read the hiring scenario below and then answer the questions.
The Somalia Hunger Relief Charity is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a children's non-profit. She was promoted after developing successful partnerships with several international charity agencies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an advocacy non-profit. She was promoted after designing a new fundraising campaign that raised significantly more donations than her predecessor. As part of her proposed contract, Karen is asking for a salary of $400,000.
Please use the scale below to indicate whether the following characteristics are more true of Lisa or Karen. 1  2  3  4  5  6  7 ___Who is a more responsible person?

Definitely Lisa Definitely Karen
___Who is probably a more morally upstanding human being? ___Who do you predict will make more responsible decisions as leader?
___Who do you predict will act in the best interests of the organization?
___Who is a more selfish person?
___Who would you prefer to donate money with? ___Who would you hire as President?
Pre-Publication Independent Replication (PPIR) 123 NOTE: This is the "higher standard" study. This is condition *5* of 6 between-subjects conditions. Please further note that the sixth DV item says "invest money" in conditions 1-3 and "donate" in conditions 4-6.

Instructions: Please read the hiring scenario below and then answer the questions.
The Somalia Hunger Relief Charity is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a children's non-profit. She was promoted after developing successful partnerships with several international charity agencies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an advocacy non-profit. She was promoted after designing a new fundraising campaign that raised significantly more donations than her predecessor. As part of her proposed contract, Karen is asking for a salary of $350,000 plus $50,000 per year for rental of a chauffeur-driven limo on the weekends.
Please use the scale below to indicate whether the following characteristics are more true of Lisa or Karen. 1  2  3  4  5  6  7 ___Who is a more responsible person?

Definitely Lisa Definitely Karen
___Who is probably a more morally upstanding human being? ___Who do you predict will make more responsible decisions as leader?
___Who do you predict will act in the best interests of the organization?
___Who is a more selfish person?
___Who would you prefer to donate money with? ___Who would you hire as President?

Pre-Publication Independent Replication (PPIR) 124
NOTE: This is the "higher standard" study. This is condition *6* of 6 between-subjects conditions. Please further note that the sixth DV item says "invest money" in conditions 1-3 and "donate" in conditions 4-6.

Instructions: Please read the hiring scenario below and then answer the questions.
The Somalia Hunger Relief Charity is deciding between two candidates for President.
Lisa has an MBA from Harvard Business School and eight years of managerial experience at a children's non-profit. She was promoted after developing successful partnerships with several international charity agencies that cut overhead and administrative costs substantially. As part of her contract, Lisa is requesting a salary of $400,000 a year.
Karen has an MBA from Ross Business School at the University of Michigan and eleven years of managerial experience at an advocacy non-profit. She was promoted after designing a new fundraising campaign that raised significantly more donations than her predecessor. As part of her proposed contract, Karen is asking for a salary of $395,000 plus $5,000 per year for luxury water flown from Sweden.
Please use the scale below to indicate whether the following characteristics are more true of Lisa or Karen. 1  2  3  4  5  6  7 ___Who is a more responsible person?

Definitely Lisa Definitely Karen
___Who is probably a more morally upstanding human being? ___Who do you predict will make more responsible decisions as leader?
___Who do you predict will act in the best interests of the organization?
___Who is a more selfish person?
___Who would you prefer to donate money with? ___Who would you hire as President?

Pre-Publication Independent Replication (PPIR) 125
NOTE: This is the "belief-act inconsistency study", condition *1* of 3. The study uses a between-subjects design with random assignment to one of the three conditions.
Bob Hill has worked for 20 years as an animal rights activist and president of the non-profit organization Furry Friends Forever (FFF), which advocates for the ethical treatment of domestic and wild animals. FFF works through public education, cruelty investigations, research, animal rescue, legislation, special events, celebrity involvement, and protest campaigns.
Recently, the Associated Press news service reported that Hill had participated in a wild game hunting safari in South Africa. The report indicated that this is the fourth big game hunting safari that Hill has done in the last five years. Below is a picture that accompanied the press release, showing Hill with a Kudu antelope that he shot down with a .338 Winchester Magnum hunting rifle.
(1) How morally blameworthy or morally praiseworthy do you find Bob as a person?

Extremely Blameworthy Extremely Praiseworthy
(2) How much warmth or coldness do you feel personally towards Bob?

Incredibly cold Incredibly warm
(3) How trustworthy do you personally find Bob to be?
Incredibly untrustworthy Incredibly trustworthy (4) Do you find Bob to be a hypocrite? Very Wrong Perfectly Okay NOTE: This is the "belief-act inconsistency study", condition *2* of 3. The study uses a between-subjects design with random assignment to one of the three conditions.
Bob Hill has worked for 20 years as an avid hunter and president of the American Big Game Hunters Association (ABGA), which advocates for big game trophy hunting throughout North America and the world. ABGA serves the hunting community through the sharing of experiences, knowledge and technology, promoting the education of youth in securing the future of the hunting tradition, and extending the goodwill of members through community outreach.
Recently, the Associated Press news service reported that Hill had participated in a wild game hunting safari in South Africa. The report indicated that this is the fourth big game hunting safari that Hill has done in the last five years. Below is a picture that accompanied the press release, showing Hill with a Kudu antelope that he shot down with a .338 Winchester Magnum hunting rifle.
(1) How morally blameworthy or morally praiseworthy do you find Bob as a person?

Extremely Blameworthy Extremely Praiseworthy
(2) How much warmth or coldness do you feel personally towards Bob?

Incredibly cold Incredibly warm
(3) How trustworthy do you personally find Bob to be?
Incredibly untrustworthy Incredibly trustworthy (4) Do you find Bob to be a hypocrite?

NOTE: This is the "belief-act inconsistency study", condition *3* of 3. The study uses a between-subjects design with random assignment to one of the three conditions.
Bob Hill has worked for 20 years as a human right activist and president of doctors without borders (DWB), which provides medical aid in nearly 60 countries to people whose survival is threatened by violence, neglect, or catastrophe, primarily due to armed conflict, epidemics, malnutrition, exclusion from health care, or natural disasters. DWB provides independent, impartial assistance to those most in need. DWB is committed to bringing quality medical care to people caught in crisis regardless of race, religion, or political affiliation.
Recently, the Associated Press news service reported that Hill had participated in a wild game hunting safari in South Africa. The report indicated that this is the fourth big game hunting safari that Hill has done in the last five years. Below is a picture that accompanied the press release, showing Hill with a Kudu antelope that he shot down with a .338 Winchester Magnum hunting rifle.
(1) How morally blameworthy or morally praiseworthy do you find Bob as a person?

Extremely Blameworthy Extremely Praiseworthy
(2) How much warmth or coldness do you feel personally towards Bob?

Incredibly cold Incredibly warm
(3) How trustworthy do you personally find Bob to be?
Incredibly untrustworthy Incredibly trustworthy (4) Do you find Bob to be a hypocrite? Very Wrong Perfectly Okay NOTE: These are the materials for the "moral cliff" study. Each participant does both of these scenarios+follow-up DVs, with page order counterbalanced between-subjects.
A cosmetics company hires a model to appear in an advertisement for their skin cream. She is one in a million in terms of the beauty of her skin. The skin cream advertisement with the model appears in magazines and on billboards all over the world.
How accurately or inaccurately does the company's advertisement portray the effectiveness of their skin cream? Extremely easy A cosmetics company hires a model to appear in an advertisement for their skin cream. She is one in a thousand in terms of the beauty of her skin. An artist who works for the cosmetics company then uses Photoshop to make her skin appear one in a million in terms of beauty. The skin cream advertisement with the model appears in magazines and on billboards all over the world.
How accurately or inaccurately does the company's advertisement portray the effectiveness of their skin cream? Extremely easy NOTE: This is the "cold-hearted prosociality study." This is *1* of 2 between subjects conditions.

INSTRUCTIONS: Please read the paragraphs about the individuals below and answer the questions that come after.
Karen works as an assistant in a medical center that does cancer research. The laboratory develops drugs that improve survival rates for people stricken with breast cancer. As part of Karen's job, she places mice in a special cage, and then exposes them to radiation in order to give them tumors. Once the mice develop tumors, it is Karen's job to give them injections of experimental cancer drugs.
Lisa works as an assistant at a store for expensive pets. The store sells pet gerbils to wealthy individuals and families. As part of Lisa's job, she places gerbils in a special bathtub, and then exposes them to a grooming shampoo in order to make sure they look nice for the customers. NOTE: This is the "cold-hearted prosociality study." This is *2* of 2 between subjects conditions.

INSTRUCTIONS: Please read the paragraphs about the individuals below and answer the questions that come after.
Lisa works as an assistant in a medical center that does cancer research. The laboratory develops drugs that improve survival rates for people stricken with breast cancer. As part of Lisa's job, she places mice in a special cage, and then exposes them to radiation in order to give them tumors.
Once the mice develop tumors, it is Lisa's job to give them injections of experimental cancer drugs.
Karen works as an assistant at a store for expensive pets. The store sells pet gerbils to wealthy individuals and families. As part of Karen's job, she places gerbils in a special bathtub, and then exposes them to a grooming shampoo in order to make sure they look nice for the customers. NOTE: This is the "intuitive economics study". This is *1* of 2 betweensubjects conditions (4 pages of questions).           ---------------------          NOTE: This is the "intuitive economics study". This is *2* of 2 betweensubjects conditions (4 pages of questions).           ---------------------

Analytic approach
There is currently no single, fixed standard to evaluating replication results, and we will therefore apply a number of criteria to determine whether the replications successfully reproduced the original findings or not (see Brandt et al., 2014). These will include: 1. Whether the original and replication effects are in the same direction 2. Whether the replication effect was statistically significant 3. Whether meta-analyzing the original and replication effect results in a significant effect 4. Whether the replication effect size is significantly smaller than the original effect 5. Whether the replication effect size is too small to have been reliably detected in the original study (Simonsohn, 2013).
We will further employ Verhagen and Wagenmakers's (2014) suite of Bayesian tests for evaluating replications. These Bayesian tests parallel criteria 2, 3, and 4, and further test 6) whether the replication results suggest the original effect size or the null is more likely to be true.
In order to provide some additional assessments of the strength of evidence in the original studies, we will: • The final project report will feature a summary figure displaying the effect sizes observed in the original and replication labs (e.g., see Klein et al., 2014, Figure 1).
We will also conduct additional, more fine-grained comparisons of effect sizes based on the type of subject population in the replication. Specifically, we will compare original and replication effect sizes separately by: • Whether the study came first vs. did not (to address the participant fatigue issue, and potential interference effects from running multiple studies together) • Online data collections (MTurk, Moral Sense website, Your Morals Website) vs.
university participants (undergraduate students, MBAs) • Student population: psychology undergraduates vs. business undergraduates vs. MBAs • Computer vs. paper-pencil administration of materials • USA sample vs. non-USA sample • Whether the original location vs. a different location was used for the replication. (For the "Presumption of guilt study," "Belief-act inconsistency study," "Intuitive economics study," and "Burn in hell study" the original location was Northwestern University. For the other original studies it was Mechanical Turk) We will be inclusive and test for all effects in each original study in the relevant replications.

Data collection
There will be a total of three survey packets containing a total of 10 original studies to be replicated.
We will conduct self-replications on Amazon's Mechanical Turk using each of the three packets. We will collect 1000 participants in each packet for a total of 3000 participants. Data will be checked at an early stage to make sure it is collecting properly, but data collection will continue until 1000 subjects have been run in each packet.
Each replication team will be asked to collect at least 100 participants in at least one survey packet (containing 3 to 4 brief studies each). Replication teams will have until March 1 to collect data.
Replication teams using paper-pencil administration (e.g., for on-campus surveys) will receive a packet with either 3 short studies or 1 longer study and be asked to collect at least 100 participants using their packet.
This process will be flexible, however, based on the resources of individual labs, and some replication teams may collect fewer (or more) subjects or replicate fewer (or more) studies.
If replication teams have difficulties in collecting enough data by the original March 1st deadline, or it appears there will be too much data to analyze and write it up by the original manuscript deadline of April 1st, we may extend the deadline for data collection to June 15th (i.e., the end of the semester at most participating universities) and analyze the data and write up the paper over the summer.
NOTE: A replication of six of the original studies at HEC Paris conducted by Anne-Laure Sellier took place prior to the creation of this document, and those data were also analyzed prior to the pre-registration. However we simply repeated all of the analyses from the original study in the HEC Paris replication dataset, as we will do for all replications.

Addendum: Departures from preregistered analysis plan
We did not report the V statistic (Davis-Stober & Dana, 2014) for each of the original effects because Professors Davis-Stober and Dana determined the designs of the original studies were poorly suited to this statistical test.
We did not carry out the planned Type M and Type S error analyses (Gelman & Carlin, 2014) because both Professor Gelman and the Pipeline Projects' statistical experts expressed doubts about their suitability to the original studies targeted for replication.
Subject population (general population, MBA students, or undergraduates) turned out to be confounded with mode of study administration. All of the replications that recruited subjects from the general population collected the data online rather than in the laboratory, and paperpencil questionnaires were only used with one undergraduate sample. We therefore analyzed only subject population as a potential moderator of replication results, not the method by which the study materials were administered to subjects. Due to the limited number of samples available, we also collapsed across student populations in our analyses, and simply compared results in the general population vs. student samples.
As stipulated in the pre-registration document, we exercised the option to continue data collection until June 15 to increase the sample sizes and statistical power of the replications. In a departure from the original plan, we further extended the deadline to July 15 th to give a graduate student project coordinator more time to prepare for second year exams. Figure S5. Small telescopes results. The figure includes each original effect size, the corresponding aggregated replication effect size, and the d33% line indicating the smallest effect size that would be reasonably detectable with the original study design. Note that the original "Higher Standard" study reported one significant effect and one nonsignificant one, and that the "Presumption of Guilt" effect was originally a null finding.

IV: mi_condition DV: MI_moralgood
Original analysis: ANOVA Moderator analyses: Ran ANOVAs/regression analyses to examine how the various moderators might interact with the main effect.

Moderator 1: USA vs. non-USA replication location
USA (1) Non-USA (0) No Contribution (1)  There is a main effect of condition, no main effect of USA, and a marginally-significant interaction. There is a difference between the no contribution and charity condition for both the USA, t(1538) = -10.08, p < .001, and the non-USA samples, t(1538) = -3.04, p = .002.

Moderator 2: Student sample vs. general population
Student (1) General (0) No Contribution (1)  There is a main effect of condition, a main effect of student versus general population sample, and no interaction.

Moderator 3: Same vs. different location
Same (1) Different (0) No Contribution (1)  There is a only a main effect of condition.

Intuitive Economics
Variables: ie12com_htxfair and ie12comb_htxgood Original Analysis: a correlation between ie12com_htxfair and ie12comb_htxgood Moderator analyses: Selected cases by moderator variable, recorded the r, and performed t-tests on the rs.
To test the differences between these correlations, we used the Hausman Test to test the z-score: The main effect of Exec_Vandal Remains. There is also an interaction such that the difference in the USA sample is larger than the difference in the Non-USA sample. Main effect of Exec_Vandal remains. There is also an interaction such that the difference in the general population sample is larger than the difference in the student sample. There is a only a main effect of condition.

Presumption of Guilt
IV: presumption_condition (only Conditions 1 (no investigation) and 4 (guilty)) DV: PG_companyevaluation Original Analysis: T-test between Conditions 1 and 4 Moderator analyses: Rn ANOVAs/regressions to see if the main effect is moderated by the moderator variables.

Moderator 1: USA vs. non-USA sample
Non-USA (0) USA (1) Do nothing (1)  Contrary to the original study, there is a significant main effect of condition, such that doing nothing actually leads to significantly worse reputation ratings than being found guilty (the original study found no difference between the two conditions). No interaction with USA vs. non-USA sample.

Moderator 2: Student sample vs. general population
General (0) Student (1) Do Nothing (1) (1,1909) = 0.14, p = .71, η p 2 < .001 Contrary to the original study, there is a significant main effect of condition, such that doing nothing actually leads to significantly worse reputation ratings than being found guilty. This does not vary by student samples vs. the general population.

Moderator 3: Same vs. different location
Same (1) Different (0) Do Nothing (1)  There is a main effect for condition and a main effect of order such that ratings for both dependent measures are higher when the study appears earlier in the study packet.

Moral Cliff
Variables: mc_ps_dishonesty and mc_dishonesty Original Analysis: t-test to see if ratings of mc_ps_dishonesty were higher than ratings of mc_dishonesty.
Moderator analyses: As the original analysis was a paired, within subjects t-test, ran a repeated measures ANOVA with moderator variables. The original difference between Photoshop and Control replicates. But there is also significant moderation effect, such that this "Moral Cliff" effect is smaller in the non-USA samples than in the USA samples. The original difference between Photoshop and Control replicates. But there is also a moderation effect, such that this "Moral Cliff" effect is larger in the general population than it is for the student samples. The original difference between Photoshop and Control replicates. But there is also a moderation effect, such that the difference between the two conditions is smaller when the study was done in a different location than when it was done in the same location as the original study. There was a main effect for condition and a main effect of order such that ratings for both dependent measures are higher when the study appears later in the study packet.

Bad Tipper
IV: tipper_condition (1 (penny) vs. 2 (less tip)) DV: tipper_personjudge Original Analysis: T-test between Conditions 1 and 2 Moderator analyses: Ran ANOVAs/regressions to see if the main effect is moderated by the moderator variables.

Moderator 1: USA vs. non-USA sample
Non-USA (0) USA (1) Pennies (1)  The original main effect of pennies vs. less tip replicates. But there is also an interaction with USA versus non-USA sample. The difference between the Pennies and Less Tip condition is significant for both the non-USA samples, t(3643) = -5.04, p < .001, and USA samples, t(3643) = -19.99, p < .001, but the difference is larger for the USA samples.

Moderator 2: General vs. Student
General (0) Student (1) Pennies (1)  The original main effect of pennies versus less tip replicates. There is also an interaction with student sample vs. general population. The difference between the Pennies and Less Tip condition is significant for both the general population samples, t(3643) = -17.86, p < .001, and student samples, t(3643) = -10.19, p < .001, but the difference is larger in the general population.

Moderator 3: Same vs. different location
Different (0) Same (1) Pennies (1)  The original main effect of pennies versus less tip holds. But there is also an interaction with different population vs. same population. The difference between the Pennies and Less Tip conditions is significant for both the different locations, t(3643) = -12.34, p < .01, and same location, t(3643) = -16.41, p < .001, samples. However, the magnitude of difference is larger in the same subject population than in the other populations. There is only a main effect of condition.

Pre-Publication Independent Replication (PPIR) 164
Higher Standards: Company Conditions IV: standard_condition DV: standard_eval_7items Original Analysis: T-test between Conditions 3 (small perk) and 1 (monetary-salary only) Moderator analyses: Ran ANOVAs/regressions to see if the main effect was moderated by the various moderator variables.

Moderator 1: USA vs. non-USA sample
Non-USA (0) USA (1) No Perk (1)  Contrary to the findings of the original study, there is a significant main effect of no perk versus small perk for a company. There is also an interaction between USA vs. non-USA samples. The difference between the No Perk and Small Perk conditions holds for both the non-USA sample, t(910) = -3.84, p < .001, and USA sample, t(910) = -14.94, p < .001. However, the magnitude of the difference is larger in the USA sample.

Moderator 2: Student sample vs. general population
General (0) Student (1) No Perk (1)  Contrary to the original findings, there is a significant main effect of no perk versus small perk for a company. There is no interaction with type of sample (student vs. general population).

Moderator 3: Same vs. different location
Different (0) Same (1) No Perk (1)  Contrary to the findings of the original study, there is a significant main effect of no versus small perk for a company. There is also an interaction between same versus different location. The difference between the No Perk and Small Perk conditions holds for both the different location, t(910) = -9.38, p < .001, and same location, t(910) = -13.16, p < .001, samples. However, the magnitude of the difference is larger in the same location sample. There is only a main effect of condition.

Higher Standard: Charity Conditions
Original Analysis: T-test between Conditions 4 (monetary-salary only) and 6 (small perk) Moderator analyses: Ran ANOVAs/regressions to see if the main effect was moderated by the various moderator variables.

Moderator 1: USA vs. non-USA sample
Non-USA (0) USA (1) No Perk (4)  Only the original main effect of no perk versus small perk holds.

Moderator 2: Student sample vs. general population
General (0) Student (1) No Perk (4)  Only the original main effect of no versus small perk holds.

Moderator 3: Same vs. different location
Different (0) Same (1) No Perk (4)  There is only a main effect of condition.

Variables: cold_moral & cold_traits
Original Analysis: t-test comparing ratings of cold_moral with ratings of cold_traits Moderator analyses: As the original study used a paired, within subjects t-test, to test moderators we used a repeated measures ANOVA with various moderator variables. The original difference between Moral Acts and Traits replicates. But there is also a moderation effect, such that the effect is smaller in the non-USA samples than in the USA samples.

Moderator 2: General vs. Students
General ( The original difference between Moral Acts and Traits replicates. But there is also a moderation effect, such that the difference between the two conditions is larger in the general population than in the student samples. There is only a main effect of condition.

Variables: bigot_personjudge
Original Analysis: t-test comparing ratings of bigot_personjudge with the scale midpoint of 4. Moderator analyses: One-sample t-tests against the midpoint of the scale for each level of the moderators to examine whether effect holds at each level of the moderator. Between subjects ttest with moderator as the independent variable to examine whether the effect is moderated. The effect replicates in both samples, but the non-overlapping 95% confidence intervals also suggest a moderation effect, such that the bigot-misanthrope effect is weaker in the USA sample than in the non-USA sample. The effect replicates in both samples, but the non-overlapping 95% confidence intervals also suggest a moderation effect, such that the bigot-misanthrope effect is weaker in the general population than the student sample. The effect replicates in both samples, but the non-overlapping 95% confidence intervals also suggest a moderation effect such that the bigot-misanthrope effect is weaker in the same location than in a different location. Oneway ANOVA with study order as independent variable: F(3, 2602) = 1.68, p < .17. There is no moderating effect of study order.
Moderator analyses: Run ANOVAs/regressions to see if the main effect is moderated by our various moderator variables.

Moderator 1: USA vs. non-USA
Non-US (0) US (1) Animal Rights (1)  Main effect of condition still stands. Also a main effect of location such that USA samples provide lower ratings than non-USA samples. No interaction effect.

Moderator 2: Student samples vs. general population
General (0) Students (1) Animal Rights (1 There is a main effect of condition and an interaction effect such that the hypothesized effect is stronger when the study appears later in the packet rather than earlier.