Humor, a unique human characteristic, is critical in thought, communication and social interaction. Successful jokes involve a cognitive juxtaposition of mental sets, followed by an affective feeling of amusement; we isolated these two components of humor by using event-related fMRI on subjects who listened to auditorily presented semantic and phonological jokes (puns) and indicated whether or not they found the items amusing. Our findings suggest that whereas there are modality-specific pathways for processing the juxtaposition of mental sets necessary for the appreciation of jokes, a common component of humor is expressed in activity in medial ventral prefrontal cortex, a region involved in reward processing.