Evidence for a reversal of the neural information flow between object perception and object reconstruction from memory

Remembering is a reconstructive process. Surprisingly little is known about how the reconstruction of a memory unfolds in time in the human brain. We used reaction times and EEG time-series decoding to test the hypothesis that the information flow is reversed when an event is reconstructed from memory, compared to when the same event is initially being perceived. Across three experiments, we found highly consistent evidence supporting such a reversed stream. When seeing an object, low-level perceptual features were discriminated faster behaviourally, and could be decoded from brain activity earlier, than high-level conceptual features. This pattern reversed during associative memory recall, with reaction times and brain activity patterns now indicating that conceptual information was reconstructed more rapidly than perceptual details. Our findings support a neurobiologically plausible model of human memory, suggesting that memory retrieval is a hierarchical, multi-layered process that prioritizes semantically meaningful information over perceptual detail.

what the room looked like years ago, who was there at the time, and even an emotional 38 conversation with his old friend and coach Michael. Perceptual details like colours, however, are 39 initially missing in the scene, like in a faded photograph, and only gradually saturate over time. This 40 common way to depict memories in pop culture nicely illustrates that the memories we bring back 41 to mind are likely not unitary constructs, and also not veridical copies of past events. Instead, they 42 suggest that remembering is a reconstructive process that might prioritize more meaningful 43 components of an event over other more shallow aspects (Schacter, 2012;Schacter, Guerin, & St 44 Jacques, 2011). We here report three experiments that shed light onto the temporal information 45 flow during memory retrieval. Once a reminder has elicited a stored memory trace, are the different 46 features of this memory reconstructed in a systematic, hierarchical way? 47 Considering our vast knowledge about the information processing hierarchy during visual 48 perception, surprisingly little is known about the time course of memory recall. In the object 49 recognition literature, it is generally agreed that the presentation of an external stimulus initiates a 50 processing cascade that starts with low-level perceptual features in early visual areas, and 51 progresses to increasingly higher levels of semantic integration and abstraction along the inferior 52 temporal cortex ( & Poggio, 2007). However, mental representations can also be re-created from memory, without 55 much external stimulation: retrieving a scene from the movie Rocky V will elicit semantic knowledge 56 about the film (e.g. that the actor is called Sylvester Stallone), but also mental images that can 57 include fairly low-level details (e.g. whether the scene was in colour or in grey scale). How the brain 58 manages to bring back each of these features when reconstructing an event from memory remains 59 an open question. The present series of experiments tested our central working hypothesis that the 60 stream of information processing is reversed during memory reconstruction compared with the 61 perception of an external stimulus. 62 Over the last years, multivariate neuroimaging methods have made it possible to isolate brain 63 activity patterns that carry information about externally presented stimuli, but also about internally 64 generated mnemonic representations. Importantly, it has been shown that the neural trace that an 65 event produces during its initial encoding is reinstated in brain activity during its later retrieval (Chen 66 et al. However, because all existing studies only focused on a single feature of a memory representation 79 (e.g., its semantic category), the fundamental question whether memory reconstruction follows a 80 hierarchical information processing stream, similar to perception, has not been investigated. 81 We hypothesize that such a processing hierarchy does exist, and that the information flow is 82 reversed during memory reconstruction compared with perception. That is, based on the widely 83 accepted idea that memory reconstruction depends on back-projections from the hippocampus to 84 neo-cortex (Moscovitch, 2008), we expect that those areas that are anatomically closer to the 85 hippocampus (i.e. high-level conceptual processing areas along the inferior temporal cortex) should 86 be involved in the reactivation cascade faster that areas that are relatively remote (i.e., low-level 87 perceptual processing areas in earlier visual cortices). Therefore, we assume that once a reminder 88 has initiated the reactivation of an associated event, higher-level abstract information will be 89 reconstructed before lower-level perceptual information, producing an inverse temporal order of 90 processing compared with perception. 91 We tested this reverse reconstruction hypothesis in a series of two behavioural and one EEG 92 experiment (see Fig. 1b, c, and Fig. 3a). All experiments used a simple associative memory paradigm 93 where participants learn a series of arbitrary associations between word cues and everyday objects, 94 and are later cued with the word to recall the object. In order to test for a processing hierarchy, it is 95 important to independently manipulate the perceptual and conceptual contents of these objects. 96 Therefore, objects varied along two orthogonal dimensions: one perceptual dimension, where the 97 object can be either presented as a photograph or a line drawing; and a semantic dimension where 98 the object represents an animate or inanimate entity (Fig. 1a). The two behavioural experiments 99 measure reaction times while participants make perceptual or semantic category judgments for 100 objects that are either visually presented on the screen, or reconstructed from memory. during perception and memory reconstruction ( Fig. 3b and c). Our behavioural and 106 electrophysiological findings consistently support the idea that memory reconstruction is not an all-107 or-none process, but rather progresses on each single trial from higher-level semantic features to 108 lower-level perceptual details. 109

Behavioural experiments 111
Our two behavioural experiments used reaction times (RTs) to test our central hypothesis that the 112 information processing hierarchy reverses between the visual perception of an object, and its 113 reconstruction from memory. We assumed that the time required to answer a question about low-114 level perceptual (photograph vs. drawing) compared to high-level semantic (animate vs. inanimate) 115 features of an item would reflect the speed at which the brain gains access to these types of 116 information. If so, we expected that reaction time patterns would reverse depending on whether the 117 object is visually presented or reconstructed from memory: during visual perception, RTs should be 118 faster for perceptual compared with semantic questions to mirror the forward processing hierarchy,  119  while during retrieval RTs should be faster for semantic compared with perceptual questions if there  120 is a reversal of that hierarchy. 121 Both experiments used a 2 x 2 mixed design ( Fig. 1b and c), where all participants answered 122 perceptual and semantic questions (factor question type, within-subjects) about the objects. 123 Importantly, one group of participants was visually presented with the objects while answering 124 these questions, whereas the other group recalled the same objects from memory (factor task, 125 between-subjects). The main difference between the two experiments was that in Experiment 1, 126 both types of features were probed for a given object, and that in Experiment 2, object were 127 presented  were asked to create word-object associations (a total of 8 per block). Reaction times were then measured during the 143 retrieval phase, where subjects were presented with a reminder word, and asked to recall and categorize the associated 144 object according to its perceptual (photograph vs. line drawing) or semantic (animate vs. inanimate) features. Button press 145 symbols indicate at which moment in a trial RTs were collected. 146

Reaction times show the expected reversal in Experiments 1 and 2 147
To directly test for a reversal of the reaction time pattern between visual perception and memory 148 reconstruction, we performed an analysis of variance comparing the RTs to perceptual and semantic 149 questions during visual object presentation and during the cued-recall task. As predicted, we found a 150 significant interaction between task (visual vs. memory group) and question type (i.e. perceptual vs. 151 semantic) in Experiment 1 (F 1, 42 = 11.142, P = .002) and in Experiment 2 (F 1, 46 = 10.876, P = .002). 152 There was no main effect of question type (Experiment 1: F 1, 42 = 3.816, P = .057; Experiment 2: F 1, 46 153 = 3.184, P = .081), suggesting that participants were not generally faster or slower at answering one 154 type of question compared to the other ( Fig. 2a and b). 155 Post-hoc RT analyses were then performed for each task to confirm that this interaction was 156 produced by differences in the expected direction. In Experiment 1, participants in the visual 157 perception group were significantly faster when responding to perceptual (M = 795ms; SD = 235ms) 158 compared to semantic (M = 842ms, SD = 185ms) questions (t 22 = 3.68, P = .001). Importantly, these 159 differences reversed in the memory retrieval group, where RTs to semantic questions (2334ms; SD = 160 534 sec) were now significantly faster than RTs to perceptual questions (M = 2502ms; SD = 561; t 20 = 161 2.35, P = .029). This pattern was fully replicated in Experiment 2, where again the visual RT group 162 answered perceptual questions (M = 733ms; SD = 211ms) significantly faster than semantic 163 questions (M = 797ms, SD = 235; t 23 = 2.46, P = .022), whereas the memory group was significantly 164 faster at responding to semantic questions (M = 3133ms, SD = 660ms)) compared with perceptual 165 questions (M = 3348ms, SD = 754; t 23 = 2.67, P = .014). 166 Since reaction times are not necessarily normally distributed, we also wanted to confirm the results 167 using a Wilcoxon signed rank test. The significant RT differences between perceptual and conceptual 168 questions were also present using this non-parametric statistic in the visual perception group 169 (Experiment 1: z = 3.16, P = .002; Experiment 2: z = 2.57, P = .010) and in the memory group 170 (Experiment 1: z = 2.48, P = .013; Experiment 2: z = 2.42, P = .015). Reaction time analyses thus 171 support our central hypothesis that the speed of information processing for different object features 172 reverses between perception and memory, and this pattern fully replicated between Experiments 1 173 and 2. 174

Accuracy results support a reversal between perception and memory, and suggest a 175
directional dependency in the processing hierarchy 176 Next we investigated whether a similar pattern was, at least qualitatively, also present in terms of 177 accuracy. We found a significant interaction between task (visual vs. memory group) and question 178 type (i.e. perceptual vs. semantic question) in both experiments (Experiment 2: F 1, 42 = 14.467, P = 179 .001; Experiment 2: F 1, 46 = 9.698, P = .003). Post-hoc accuracy analyses in Experiment 1 revealed that 180 in the visual reaction task participants were more accurate at answering perceptual questions (M = 181 97.42%; SD = 2.68%) compared to semantic ones (M = 96.33%; SD = 1.99%). This difference was not 182 significant (t 22 = 2.03, P = .055), most likely because accuracy during perception was close to ceiling. 183 Accuracy in the memory task showed that, in line with a reversed processing stream, participants 184 had significantly better accuracy for semantic questions compared with perceptual questions (M = 185 85.83%; SD = 7.57%; vs. 82.63%; SD = 8.79%, respectively; t 20 = 3.12, P = .005). Experiment 2 186 replicated the same accuracy profile, with participants in the visual group showing a significantly 187 higher accuracy for perceptual questions (M = 97.97%; SD = 2.77%) compared to semantic questions 188 (M = 96.41%; SD = 3.07%; t 23 = 2.14, P = .042)). The reverse pattern was present in the memory 189 reaction time task, where an accuracy benefit was found for semantic questions compared to 190 perceptual ones (69.57%; SD = 15.17%; vs. 62.89%; SD = 15.09%, respectively; t 23 = 2.63, P = .015). 191 Accuracy profiles thus generally corroborated our reaction time results, again suggesting that 192 semantic information is more easily accessed during retrieval than perceptual information. 193 The accuracy data from Experiment 1 also allowed us to address an interesting question regarding 194 the dependency of perceptual and conceptual processing stages. Across the retrieval phase of this 195 experiment, both types of questions were asked for each given object, and we were thus able to test 196 to what degree performance on the semantic and perceptual questions was stochastically 197 dependent. Our reasoning was that if the reconstruction of semantic and perceptual aspects from 198 memory was a hierarchical process where access to a later stage depended on having completed the 199 previous stage(s), as predicted by a reversed stream, then the ability to retrieve perceptual details 200 would depend on having accurately retrieved semantic details, but not vice versa. In other words, if 201 the retrieval of semantic information was the first stage in a hierarchical stream, it would not 202 depend much on any other stages. If on the other hand the retrieval of perceptual information is 203 indeed a very late stage in the hierarchy, success at this stage should be considerably influenced by 204 success at earlier (semantic) stages. In line with this reasoning, we found that P(sem/per) -the 205 conditional probability of remembering the correct semantic information given the perceptual 206 question was answered correctly for the same word-picture association (M = 91.61%; SD = 6.98%) -207 was significantly higher (t 20 = 3.08, P = .006) than P(per/sem) -the conditional probability of 208 answering the perceptual question correctly given a correct semantic answer (M = 88.28%; SD = 209 8.34%) (Fig. 2c). For reasons of completeness, we carried out the same conditional probability 210 analysis in the visual task. In this group, the opposite trend was present, with P(per/sem) (M = 211 97.30%; SD = 2.82 %) being numerically higher than P(sem/per) (M = 96.21%; SD = 2.09%). However, 212 this difference was not statistically robust (t 22 = 2.04, P = .054), most likely due to ceiling effects. 213 Altogether, the findings from our two behavioural experiments provide support for our main 214 hypothesis that during retrieval of a complex visual representation, the temporal order in which 215 perceptual and semantic features are processed reverses between perception (feed-forward) and 216 memory retrieval (feed-backward). The results suggest that reaction times can be used as a proxy to 217 probe neural processing speed, as argued in previous studies (Ritchie, Tovar, & Carlson, 2015). In the 218 next sections, we report the findings from an EEG study that more directly taps into the neural 219 processes that we believe are producing the behavioural pattern.

227
The conditional probability of remembering the correct semantic information given the perceptual question was answered 228 correctly for the same object, P(sem/per) was significantly higher than the conditional probability of answering the 229 perceptual question correctly given a correct semantic answer, P(per/sem). Each line represents the trend for one

EEG experiment 234
While it is reasonable to assume that reaction times tap into the neural processing speed for a given 235 feature, based on previous literature (Ritchie et al., 2015), we also wanted to obtain a more direct 236 signature of feature activation from human brain activity. We therefore used multivariate pattern 237 analysis applied to electrophysiological (EEG) recordings, with the goal to pinpoint when in time, on 238 an individual trial, the perceptual and semantic features of an object could be decoded from brain 239 activity. We expected to find the maximum decodability of perceptual information before semantic 240 information when an object was visually presented on the screen, and expected the order of these 241 peaks to reverse when the object was recalled from memory. The design closely followed the 242 behavioural experiments, with the important difference that all factors were manipulated within 243 subjects, such that each participant carried out a visual encoding phase that served to probe visual 244 (forward) processing, and a subsequent recall phase used to probe mnemonic (backward) processing 245 (Fig. 3). The trial timing was optimised for obtaining a clean signal during object presentation and 246 object recall, rather than for measuring reaction times. We therefore presented the perceptual and 247 semantic questions only during the recall phase in order to probe memory accuracy, and questions 248 were presented at the end of each recall trial, such that they would not bias processing towards 249 perceptual or semantic features of the object. 250

Accuracy in the EEG study replicates the response pattern found in the behavioural 251 experiments 252
In the retrieval phase of the EEG experiment, subjects were again cued with a word and asked to 253 retrieve the associated object. On average participants subjectively declared to retrieve the object 254 on 93.6% of the trials (SD = 5.89%), with an average reaction time of 3046ms (SD = 830ms; minimum 255 = 1369ms; maximum = 5124ms) to make this response. We then asked two objective questions at 256 the end of each trial, one perceptual and one semantic, which participants answered with an overall 257 mean accuracy of 86.37% (SD = 6.6). Mirroring our behavioural experiments, hit rates for answering 258 the semantic question were 87.65% (SD = 6.57%), significantly higher (t 20 = 5.16, P = .001) than the 259 accuracy for the perceptual question (M = 85.08%; SD = 6.53%). Note that the EEG task was not 260 designed to measure reaction times, and participants were instructed to prioritize accuracy over 261 speed. 262

Single-trial classifier fidelity suggests a reversal of information processing between 263 perception and memory recall 264
In order to determine the temporal trajectory of feature processing on a single trial level, we carried 265 out a series of time resolved decoding analyses. Linear discriminant analysis (LDA, see Method 266 section) was used to classify perceptual (photograph vs. line drawing) and semantic (animate vs. 267 inanimate) features of an object based on the EEG topography at a given time point, either during 268 object presentation (encoding) or during object retrieval from memory (cued recall). 269 Our first aim was to confirm that there was a forward stream during perceptual object processing. 270 Two separate classifiers were therefore trained and tested during encoding to classify the perceptual 271 category (photograph vs. line drawing) and the semantic category (animate vs. inanimate) of the to-272 be-encoded object, respectively, in each trial and time point per participant (see Fig. 3). For these 273 analyses, decoding was performed in separate time windows starting 100ms before stimulus onset 274 and up until 500ms post-stimulus. Our main interest was to determine the specific moment in each 275 trial at which the two classifiers showed the highest fidelity in determining the correct perceptual 276 and semantic categories ( Fig. 3b and c). For the encoding data, we thus identified the highest d value 277 peak per trial within 500ms of stimulus onset (see Methods section). This approach allowed us to 278 compare, within each encoding trial, whether the classification peak for perceptual features 279 occurred earlier than the classification peak for semantic features.

297
Comparing all single trial d value peaks from encoding ( Fig. 4a), we found a significant difference (z = 298 1.87, P = 0.03) between the timing of perceptual and semantic peaks using a one-tailed Wilcoxon 299 signed rank test, suggesting that confidence peaks for perceptual classification occurred before 300 those for semantic classification. The obtained Z score was compared against a bootstrapped data 301 set (see Methods section) to estimate the likelihood of obtaining a distance between peaks of the 302 same or larger size from a distribution with randomly shuffled category labels, using the same EEG 303 epochs and the same time window. The observed difference score (z = Importantly, following the same procedure, we next analysed the differences between the 309 perceptual and semantic classifier peaks during memory reactivation, to test whether the order 310 reversed during retrieval compared with encoding. The single-trial approach made sure that the 311 relative temporal order of perceptual and semantic peaks within a trial would be preserved even if 312 the retrieval process was set off with a varying delay across trials. To further minimize variance 313 between the retrieval trials, we aligned all trials relative to the button press, i.e. the moment when 314 participants declared that they had retrieved the associated object from memory. The time window 315 used in this analysis covered 3sec prior to participants' response and, based on behavioural reaction 316 times, only trials with a RT ≥ 3 sec were included. Using a one-tailed Wilcoxon signed rank test, a 317 significant difference (z = 2.53, P = .006) was found when we compared d value peak distributions of 318 perceptual with those of semantic classification obtained from all single trials and participants (Fig.  319 4b). The obtained Z score was again higher than the 95 th percentile (z = 1.59) of the random 320 distribution of a bootstrapped data set (see Methods section) using the same EEG signal and time 321 window. Critically, the one-tailed test in this case confirms our central hypothesis that during 322 memory retrieval, semantic information can be classified in brain activity significantly earlier than 323 perceptual information, suggesting a reversal of information flow relative to perception. 324 The last classification analysis was aimed at confirming the results obtained from the previous single-325 trial, fixed-effects analyses using a random-effects approach. We calculated the average d value 326 peak latency for perceptual and semantic classification in each participant, and performed a 2x2 327 ANOVA with stage (encoding vs. retrieval) and type of feature (perceptual vs. semantic) as within-328 subject factors. Confirming our main hypothesis, this analysis revealed a significant interaction (F 1, 20 329 = 4.63, P = .044) between stage and the type of feature. We further found a main effect of type of 330 feature (F 1, 20 = 4.80, P = .04). Post-hoc T-tests showed no significant difference (t 20 = 0.67, P = .253) 331 between the average perceptual and semantic d value peaks during encoding (Fig. 4c). However, 332 during retrieval, we found that the semantic classifier systematically (t 20 = 2.20, P = .020, one-tailed) 333 peaked earlier than the perceptual classifier (Fig. 4d). These findings indicate that even though a 334 single-trial comparison of classifier fidelity is more sensitive to the temporal dynamics of feature 335 processing, the same pattern is also present in the average classification values. 336 Overall, the results again confirm our hypothesis that the information processing hierarchy reverses 337 between perception (encoding) and recall, and that memory recall prioritizes semantic over 338 perceptual information. 339 (blue) and semantic (pink) classes. A significant difference between the two peak distributions was found during object 344 encoding (P = .015), indicating a bias towards earlier occurrence of perceptual (blue) compared with semantic (pink) peaks.

345
During object retrieval, a significant difference between the distributions was found (P = .006) in the opposite direction 346 relative to encoding, with semantic peaks now occurring earlier than perceptual peaks. Box plots represent group peak 347 distribution of d values for perceptual and semantic categories during encoding (c) and retrieval (d) after averaging peaks 348 within participants. A significant interaction (P = .048) was found between task (encoding or retrieval) and type of feature  In a final step, we also sought to corroborate our findings by more conventional event-related 353 potential (ERP) analyses. If the differences in neural activity between perceptual (photograph vs. line 354 drawing) and semantic (animate vs. inanimate) categories, as picked up by the LDA classifier, were 355 produced by a signal that is relatively stable across trials and participants, we expected to see these 356 signal differences in the average ERP time courses across participants. A comparison of the ERP 357 peaks during encoding and retrieval would then reveal the same perception-to-memory reversal as 358 found in our multivariate analyses. 359 Firstly, a series of cluster-based permutation tests (see Methods section) was performed during 360 object presentation to test for ERP differences between perceptual and semantic categories. 361 Contrasting objects from the two different perceptual categories (photographs and line drawings), 362 we obtained a significant positive cluster (P corr = .008) between 136ms and 232ms after stimulus 363 onset, with a maximum difference based on the sum of T values at 188ms, and located over occipital 364 and central electrodes (see Fig. 5a). Contrasting objects from the different semantic categories 365 (animate and inanimate) revealed a later cluster over frontal and occipital electrodes (P corr = .001) 366 from 237ms until 357ms after stimulus presentation, with a maximum difference at 306ms (see Fig.  367 5a). The peak semantic ERP difference for encoding thus occurred ~120ms after the peak perceptual 368 difference, consistent with the existing ERP literature (Fabiani, M., Gratton, G., & Federmeier, 2007) . 369 Similar contrasts between perceptual and semantic categories were then carried out during 370 retrieval, aligning trials to the time of the button press. We found a significant perceptual cluster 371 distinguishing the recall of photographs and line drawings over occipital electrodes (P corr = .046) 372 between 1390ms and 1336ms before participants' responses, with a maximum difference based on 373 the sum of T values at 1360ms prior to response time (see Fig. 5b). Comparing ERPs for the different 374 semantic categories, we found a significant cluster distinguishing the recall of animate from 375 inanimate objects over frontal electrodes (P corr = .032) between 1781ms and 1735ms before object 376 retrieval, with a maximum difference at 1770ms (see Fig. 5b). Therefore, during memory retrieval, 377 the peak semantic ERP difference occurred ~400ms before the peak perceptual difference. Note that 378 the timing of the effects also coincides with the timing of the classifier results in terms of the 379 maximum differences between perceptual and semantic categories (see Fig. 4). The ERP results thus 380 mirror, qualitatively, the results of our previous multivariate analyses in terms of the timing of the 381 maximum signal difference between categories. Again, these results suggest that perceptual aspects 382 are coded in brain activity earlier than semantic aspects during visual processing, but semantic 383 differences dominate the EEG signal earlier than perceptual ones during retrieval. 384 much less is known about the mnemonic processing cascade. Here we demonstrate that the 399 reconstruction of a visual memory does depend on a hierarchical stream too, but this mnemonic 400 stream follows the reverse order relative to visual processing. Across three experiments, we found 401 highly converging evidence in favour of such a reversal from behavioural reaction times and accuracy 402 (Experiments 1 and 2), from multivariate classification analyses, and from univariate ERP analyses 403 (Experiment 3). 404 The behavioural studies demonstrate that participants were significantly faster at detecting low-405 level perceptual differences than abstract, conceptual differences during a visual classification task, 406 i.e. while an object was presented on the screen. Critically, however, when we asked participants to 407 categorize the perceptual or semantic components of objects recalled from memory, the reverse 408 effect was found: subjects required significantly less time to correctly retrieve semantic information 409 about the object compared to perceptual details (see Fig. 2a and Fig. 2b). This reversal was 410 corroborated by a significant interaction between the kind of feature (perceptual or semantic) and 411 the kind of task (visual perception or memory recall task). Based on signal-detection models (Ashby,412 2000; O'Connell, Dockree, & Kelly, 2012), the RT findings suggest that during memory 413 reconstruction, the decision threshold to identify abstract information of a mnemonic 414 representation is reached before a judgment about low-level information can be made. The 415 response latency pattern therefore supports our central hypothesis that the temporal order of 416 feature processing is reversed when retrieving a previously stored representation of an object, 417 relative to its perception. 418 In addition to reaction times, the same reversal pattern was present in accuracy. Here, the accuracy 419 profiles from Experiment 1 also allowed us to conduct a conditional probability analysis. Specifically, 420 we were interested in whether access to semantic features and access to perceptual features are 421 dependent on each other, and whether the direction of this mutual dependency would provide 422 evidence for a processing hierarchy. Conditional probabilities revealed that when participants 423 correctly retrieved perceptual information of a given object, they were highly likely to also make an 424 accurate response about the semantic features of the same object, but not vice versa (see Fig. 2c). In 425 other words, retrieving perceptual features required access to semantic features, but retrieving 426 semantic features did not predict access to perceptual features to the same degree, as would be 427 expected if the processing stream was hierarchically organized. These findings are consistent with an 428 information-processing stream where access to perceptual details of a mnemonic representation 429 depends on having completed the presumably earlier semantic stage, a finding consistent with 430 hierarchical memory system models (Henson & Gagnepain, 2010). 431 The results from our third, EEG experiment fully support the conclusions drawn from the 432 behavioural studies. We used temporally resolved multivariate decoding analyses to observe when 433 in time, during object perception and object retrieval, the perceptual and semantic features of an 434 object would be maximally decodable from a participant's brain activity patterns. These analyses 435 were carried out on a single trial level such that the fidelity peaks of the perceptual and semantic 436 classifiers could be directly compared. When an object was visually presented during encoding, the 437 maximum fidelity (d value) in classifying perceptual information (photograph vs. line drawings) 438 occurred significantly earlier (approximately 100 ms) than the maximum for semantic information 439 (animate vs. inanimate) (see Fig. 4a). This finding is consistent with a predominantly feed-forward 440 processing stream as described previously (Carlson et  reactivate an object's representation from memory, peaks in classifying semantic information were 443 found roughly 300ms before the peaks for perceptual categories (see Fig. 4b). This reversal in 444 classifier fidelity was present on a trial-by-trial level but also when averaging peak latencies per 445 participant (see Fig. 4c and Fig. 4d). Like in the behavioural experiments, a consistent reversal 446 between perception and memory was supported by a significant interaction between the kind of 447 feature (perceptual or semantic) and the type of task (perception vs. retrieval). Finally, we also found 448 the same reversal pattern in the ERP peaks when comparing the maximum ERP difference between 449 perceptual and semantic object classes. During object perception, the largest perceptual ERP cluster 450 occurred ~100ms before the semantic ERP cluster, whereas during retrieval the perceptual cluster 451 followed the semantic one with a lag of about 400ms (see Fig. 5 later allows a partial cue to trigger the reconstruction of these different elements from memory. This 486 memory reconstruction process is thought to depend on back-projections from the hippocampus to 487 neocortical areas, causing the reactivation of memory patterns in at least a subset of the areas that 488 were involved in perceiving the original event. Such reactivation has consistently been reported in 489 higher-order sensory regions related to processing of complex stimulus and task information 490 suggesting that in principle higher-and lower-level information can be reconstructed from memory. 493 Interestingly, however, recent evidence suggests that the semantic structure of complex naturalistic 494 events is represented in brain activity patterns more consistently when participants reproduce the 495 event narratives (movies) from memory, as opposed to watching the movies (Chen et al., 2017). Our 496 work offers a neurobiologically plausible explanation for why higher-order meaningful information 497 might be prioritized during retrieval. Within the medial temporal lobe, regions that are involved in 498 the processing of objects and scenes are also activated when retrieving objects and scenes from 499 memory, but with a delay relative to the actual perception of objects and scenes, consistent with a 500 reversed information flow (Staresina, Cooper, & Henson, 2013). Intracranial EEG recordings have 501 shown that connectivity between the entorhinal cortex and the hippocampus changes directionality 502 between encoding and retrieval (Fell et al., 2016), which could provide the functional basis for 503 cortical reinstatement. Studies in rodents also indicate that the neural codes that represent certain 504 spatial trajectories are often replayed in reverse order when the animal is awake and resting, 505 suggesting a potential role in memory retrieval (Carr & Frank, 2012), and there is very recent work in 506 humans pointing to reverse replay of spatial sequences during offline states (Kurth-Nelson, 507 Economides, Dolan, & Dayan, 2016). Finally, previous work using MEG decoding suggests that it is 508 mainly the later processing stages of the encoding stream that are reactivated during retrieval, 509 consistent with a prioritization of higher-level information during retrieval (Kurth-Nelson et al., 510 2015). Our proposal of a reverse processing hierarchy is thus plausible based on functional anatomy 511 and the existing literature, even though it has never been explicitly proposed or tested so far. 512 We regard our reverse reconstruction hypothesis as complementary to existing models that address 513 the nature and timing of different retrieval processes, including the influential dual process model 514 (for a review see Yonelinas, Aly, Wang, & Koen, 2010). Dual process models focus on recognition 515 rather than recall tasks, and on the cognitive processes and operations required to access a stored 516 memory rather than the reactivated features of a memory themselves. They assume that successful 517 recognition of a previously stored stimulus can be based on a sense of familiarity, or on the 518 additional recollection of contextual information associated with the stimulus during encoding, an 519 influential idea in the memory field since the introspective analyses of William James (James, 1890). 520 While the original model does not explicitly address the time course of these processes, it is now 521 widely accepted, based on the EEG literature, that familiarity signals occur earlier than recollection 522 signals. Familiarity signals can be detected in the EEG as early as 300ms after the onset of a 523 recognition probe, while recollection-related activity typically begins to emerge after 500-600ms 524 ( where successful recall strongly depends on the recollection of associative information. Within this 527 recollection process, we find that the semantic "gist" of a memory is accessed before perceptual 528 details. This hierarchical progression from an early global semantic (i.e., familiarity-like) signal to 529 more fine-grained recollection might thus be a fundamental principle of retrieval that is shared 530 between recall and recognition memory. 531 Beyond specific models of declarative memory, there are also interesting parallels between our 532 findings and visual learning phenomena like the Eureka effect (Ahissar & Hochstein, 1997). The 533 general idea that perception is shaped by stored representations has been proposed over a century 534 ago by von Helmholtz (Helmholtz, 1924). A wealth of findings now support the idea that previous 535 exposures to a stimulus can exert a strong top-down influence on its subsequent perception (for a 536 review; Aggelopoulos, 2015). Reminiscent of our present findings, Ahissar and Hochstein (2004)  537 suggest that such visual learning is a top-down process that progresses from high-level to low-level 538 visual areas with increasing practice. Specifically, they argue that improvements in visual 539 discrimination tasks (e.g. identifying a tilted line among distractors) are guided by high-level 540 information (e.g. "the gist of the scene") during earlier stages of learning, and increasingly more by 541 low-level information (e.g. line orientations or colours) at later stages. Our findings indicate that 542 during the reactivation of an object's stored representation, its high-level features are retrieved 543 more rapidly than its low-level components. Abstract information might thus be reactivated more 544 easily and during earlier stages of visual learning, and thus have a stronger driving influence on 545 performance than more detailed information. Even though speculative at the moment, our reverse 546 reconstruction framework might thus have explanatory value for findings in related fields of learning 547 and memory. 548 How our brain brings back to mind past events, and enriches our mental life with vivid images or 549 sounds or scents beyond the current external stimulation, is still a fascinating and poorly understood 550 phenomenon. Our present results suggest that memories, once they are triggered by a reminder, 551 unfold in a systematic and hierarchical way, and that the mnemonic processing hierarchy is reversed 552 with respect to the major visual processing hierarchy. We hope that these findings can inspire more 553 dynamic frameworks of memory retrieval that explicitly acknowledge the reconstructive nature of 554 the process, rather than simply conceptualizing memories as reactivated snapshots of past events. 555 Such models will help us understand the heuristics and systematic biases that are inherent in our 556 memories and memory-guided behaviours. 557  four of them performed the memory reaction time task and another group of 24 took part in the 567 visual reaction time task. For the electrophysiological experiment we recruited a total of 24 568 volunteers (20 female; mean age 21.91 ± 4.68 years). Since the first 3 subjects we recorded 569 performed a slightly different task during retrieval blocks (i.e., they were not asked to mentally 570 visualise the object for 3 seconds, and they had to answer only one of the perceptual and semantic 571 questions per trial), we did not include these participants in any of the retrieval analyses. 572

Methods
All participants reported being native or highly fluent English speakers, having normal (20/20) or 573 corrected-to-normal vision, normal colour vision, and no history of neurological disorders. We 574 received written informed consent from all participants before the beginning of the experiment. 575 They were naïve as to the goals of the experiments, but were debriefed at the end. Participants were 576 compensated for their time, receiving course credits or £6 per hour for participation in the 577 behavioural task, or a total of £20 for participation in the electrophysiological experiment. The 578 University of Birmingham's Science, Technology, Engineering and Mathematics Ethical Review 579 Committee approved all experiments. 580

Stimuli 581
In total, 128 pictures of unique everyday objects and common animals were used in the main 582 experiment, and a further 16 were used for practice purposes. Out of these, 96 were selected from 583 the BOSS database (Brodeur, Dionne-Dostie, Montreuil, & Lepage, 2010), and the remaining images 584 were obtained from online royalty-free databases. All original images were pictures in colour on a 585 white background. To produce two different semantic object categories, half of the objects were 586 chosen to be animate while the other half was inanimate. Within the category of inanimate objects, 587 we selected the same amount of electronic devices, clothes, fruits and vegetables (16 each). The 588 animate category was composed of an equivalent number of mammals, birds, insects and marine 589 animals (16 each). With the objective of creating two levels of perceptual manipulation, a freehand 590 line drawing of each image was created using the free and open source GNU image manipulation 591 software (www.gimp.org). Hence a total of 128 freehand drawings of the respective 128 pictures of 592 everyday objects were created. Each drawing was composed of a white background and black lines 593 to generate a schematic outline of each stimulus. For each subject, half of the objects were pseudo-594 randomly chose to be presented as photographs, and half of them as drawings, with the restriction 595 that the two perceptual categories were equally distributed across (i.e. orthogonal with respect to) 596 the animate and inanimate object categories. All photographs and line drawings were presented at 597 the centre of the screen with a rescaled size of 500 x 500 pixels. For the memory reaction time task 598 and the EEG experiment, 128 action verbs were selected that served as associative cues. Experiment 599 2 also used colour background scenes of indoor and outdoor spaces (900 x 1600 pixels) that were 600 obtained from online royalty-free databases, which are irrelevant for the present purpose.

Visual reaction time task 605
Before the start of the experiment, participants were given oral instructions and completed a 606 training block of 4 trials to become familiar with the task. The main perceptual task consisted of 4 607 blocks of 32 trials each (Fig.1b). All trials started with a jittered fixation cross (500 to 1500ms) that 608 was followed by a question screen. On each trial, the question could either be a perceptual question 609 asking the participant to decide as quickly as possible whether the upcoming object is shown as a 610 colour photograph or as a line drawing; or a semantic question asking whether the upcoming object 611 represents an animate or inanimate object. Two possible response options were displayed at the 612 two opposite sides of the screen (right or left). The options for "animate" and "photograph" were 613 always located on the right side to keep the response mapping easy. The question screen was 614 displayed for 3 seconds, and an object was then added at the centre of the screen. In Experiment 2, 615 this object was overlaid onto a background that filled large parts of the screen. Participants were 616 asked to categorize the object in line with the question as fast as they could as soon as the object 617 appeared on the screen, by pressing the left or right arrow on the keyboard. Reaction times (RTs) 618 were measured to test if participants were faster at making perceptual compared to semantic 619 decisions. 620 All pictures were presented until the participant made a response but for a maximum of 10 sec, after 621 which the next trial started. Feedback about participants' performance was presented at the end of 622 each experimental block. There were 256 trials overall, with each object being presented twice 623 across the experiment, once together with a perceptual and once with a semantic question. 624 Repetitions of the same object were separated by a minimum distance of 2 intervening trials. In each 625 block, we asked the semantic question first for half of the objects, and the perceptual question first 626 for the other half. 627 The final reaction time analyses only included trials with correct responses, and excluded all trials 628 with an RT that exceeded the average over subjects by +-2.5 standard deviations (SDs). 629

Memory reaction time tasks 630
The memory version was kept very similar to the visual reaction time task, but we now measured 631 RTs for objects that were reconstructed from memory rather than being presented on the screen, 632 and we thus had to introduce a learning phase first. At the beginning of the session, all participants 633 received instructions and performed two short practice blocks. Each of the overall 16 experimental 634 blocks consisted of an associative learning phase (8 word-object associations) and a retrieval phase 635 (16 trials, testing each object twice, once with a perceptual and once with a semantic question). The 636 associative learning and the retrieval test were separated by a distractor task. During the learning 637 phase (Fig. 1c), each trial started with a jittered fixation cross (between 500 and 1500ms) that was 638 followed by a unique action verb displayed on the screen (1500ms). After presentation of another 639 fixation cross (between 500 and 1500ms), a picture of an object was presented on the centre of the 640 screen for a minimum of 2 and a maximum of 10 seconds. Participants were asked to come up with a 641 vivid mental image that involved the object and the action verb presented in the current trial. They  642 were instructed to press a key (up arrow on the keyboard) as soon as they had a clear association in 643 mind; this button press initiated the onset of the next trial. Participants were made aware during the 644 initial practice that they would later be asked about the object's perceptual properties as well as its 645 meaning, and should thus pay attention to details including colour and shape. Within a participant, 646 each semantic category and sub-category (electronic devices, clothes, fruits, vegetables, mammals, 647 birds, insects, and marine animals) was presented equally often at each type of perceptual level (i.e. 648 as a photograph or as a line drawing). The assignment of action verbs to objects for associative 649 learning was random, and the occurrence of the semantic and perceptual object categories was 650 equally distributed over the first and the second half of the experiment in order to avoid random 651 sequences with overly strong clustering. 652 After each learning phase, participants performed a distractor task where they were asked to classify 653 a random number (between 1 and 99) on the screen as odd or even. The task was self-paced and 654 they were instructed to accomplish as many trials as they could in 45 seconds. At the end of the 655 distractor task, they received feedback about their accuracy (i.e., how many trials they performed 656 correctly in this block). 657 The retrieval phase (Fig. 1c) started following the distractor task. Each trial began with a jittered 658 fixation cross (between 500 and 1500ms), followed by a question screen asking either about the 659 semantic (animate vs. inanimate) or perceptual (photograph vs. line drawing) features for the 660 upcoming trial, just like in the visual perception version of the task. The question screen was 661 displayed for 3 seconds by itself, and then one of the verbs presented in the directly preceding 662 learning phase appeared above the two responses. We asked participants to bring back to mind the 663 object that had been associated with this word and to answer the question as fast as possible by 664 selecting the correct response alternative (left or right keyboard press). If they were unable to 665 retrieve the object, participants were asked to press the down arrow. The next trial began as soon as 666 an answer was selected. At the end of each retrieval block, a feedback screen showing the 667 percentage of accurate responses was displayed. 668 Throughout the retrieval test, we probed memory for all word-object associations learned in the 669 immediately preceding encoding phase in pseudorandom order. Each word-object association was 670 tested twice, once together with a semantic and once with a perceptual question, with a minimum 671 distance of 2 intervening trials. In addition, we controlled that the first question for half of the 672 associations was semantic, and perceptual for the other half. Like in the visual RT task, the response 673 options for "animate" and "photograph" responses were always located on the right side of the 674 screen. In total, including instructions, a practice block and the 16 learning-distractor-retrieval 675 blocks, the experiment took approximately 60 minutes. 676 For RT analyses we only used correct trials, and excluded all trials with an RT that exceeded the 677 average over subjects by +-2.5 SDs. 678

Experiment 2 679
Experiment 2 was very similar in design and procedures to Experiment 1, and we therefore only 680 describe the differences between the two experiments in the following. 681

Visual reaction time task 682
The second experiment started with a familiarisation phase where all objects were presented 683 sequentially. In each trial of this phase, a jittered fixation cross (between 500 and 1500 ms) was 684 followed by one screen that showed the photograph and line drawing version of one object 685 simultaneously, next to each other. During the presentation of this screen (2.5 sec) participants were 686 asked to overtly name the object. After a jittered fixation cross (between 500 and 1500 ms), the 687 name of the object was presented. 688 After this familiarisation phase, the experiment followed the same procedures as the visual reaction 689 time task in Experiment 1 except for the following changes. Objects were overlaid onto a coloured 690 background scene (1600 x 900 pixels). Also, each object (286 x 286 pixels) was probed only once, 691 either together with a perceptual question, a semantic question (like above), or a contextual 692 question asking whether the background scene was indoor or outdoor. For the current purpose we 693 only describe the RTs to object-related questions in the Results section. Another minor difference to 694 Experiment 1 was that in this version of the task, the question screen was displayed for 4sec, and the 695 two options to answer during stimulus presentation were removed from the screen as soon as the 696 object/reminder appeared. 697

Memory reaction time task 698
The memory reaction time task in Experiment 2 also included, during the associative learning phase, 699 a background scene (1600 x 900 pixels) that was shown on the screen behind each object (286 x 286 700 pixels), and participants were asked to remember the word-background-object combination. In this 701 version of the task, each word-object association was tested only once, together with either a 702 perceptual question about the object, a semantic question about the object, or a contextual 703 question regarding the background scene (indoor or outdoor). Therefore, one third of the objects 704 were tested with a semantic question, one third with a perceptual question, and one third with a 705 contextual question. Again, context was not further taken into account in the present analyses. 706

EEG experiment (Experiment 3) 707
Following the EEG set-up, instructions were given to participants and two blocks of practice were 708 completed. The task procedure of the EEG experiment was similar to the memory task in 709 Experiments 1 and 2 except for the retrieval phase (Fig. 3a). Each block started with a learning phase 710 where participants created associations between overall 8 action verbs and objects. After a 40 sec 711 distractor task, participants' memory for these associations was tested in a cued recall test. In total, 712 the experiment was composed of 16 blocks of 8 associations each. 713 Each trial of the retrieval test started with a jittered fixation cross (500-1500ms), followed by the 714 presentation of one of the action verbs presented during the learning phase as a reminder. 715 Participants were asked to visualize the object associated with this action verb as vividly and in as 716 much detail as possible while the cue was on the screen. To capture the moment of retrieval, 717 participants were asked to press the up-arrow key as soon as they had the object back in mind; or 718 the down-arrow if they could not remember the object. This reminder was presented on the screen 719 for a minimum of 2 sec and until a response was made (maximum 7 sec). Immediately afterwards, a 720 blank square with the same size as the original image was displayed for 3 sec. During this time, 721 participants were asked to "mentally visualize the originally associated object on the blank square 722 space". After a short interval where only the fixation cross was present (500-1500ms), a question 723 screen was displayed for 10 seconds or until participant response asking about perceptual 724 (photograph vs. line drawing) or semantic (animate vs. inanimate) features of the retrieved 725 representation, like in the behavioural tasks. However, in this case both types of questions were 726 always asked on the same trial, and they were asked at the end of the trial rather than before the 727 appearance of the reminder. The first question was semantic in half of the trials, and perceptual in 728 the other half. Therefore, each retrieval phase consisted of 8 trials where we tested all verb-object 729 associations learned in the same block in random order. 730

Data Collection (behavioural and EEG) 731
Behavioural response recording and stimulus presentation were performed using Psychophysics 732 Toolbox Version 3 (Brainard, 1997)  was epoched into trials starting 500ms before stimulus onset and lasting until 1500ms after stimulus 741 offset. The resulting signal was baseline corrected based on pre-stimulus signal (-500ms to onset). 742 Retrieval epochs contained segments from 4000ms before until 500ms post-response. Since the 743 post-response signal during retrieval will likely still contain task-relevant (i.e., object specific) 744 information, we baseline-corrected the signal based on the whole trial. Both datasets were filtered 745 using a low-pass filter at 100 Hz and a high-pass filter at 0.1 Hz. To reduce line noise at 50 Hz we 746 band-stop filtered the signal between 48 and 52 Hz. The signal was then visually inspected and all 747 epochs that contained coarse artefacts were removed. As a result, a minimum of 92 and a maximum 748 of 124 trials remained per participant for the encoding phase, and a range between 80 and 120 trials 749 per subject remained for retrieval. Independent component analysis was then used to remove eye-750 blink and horizontal eye movement artefacts; this was followed by an interpolation of noisy 751 channels. Finally, all data was referenced to a common-average-reference (CAR). 752

Time resolved multivariate decoding 753
First, to further increase the signal to noise ratio for multivariate decoding, we smoothed our pre-754 processed EEG time courses using a Gaussian kernel with a full-width at half-maximum of 24ms. 755 Time resolved decoding via linear discriminant analysis (LDA) using shrinkage regularization (Lemm,756 Blankertz, Dickhaus, & Müller, 2011) was then carried out using custom-written code in MATLAB 757 2014b (MathWorks). Two independent classifiers were applied to each given time window and each 758 trial (see Fig. 3b): one to classify the perceptual category (photograph or line drawing) and one to 759 classify the semantic category (animate or inanimate). In both decoding analyses, we used 760 undersampling after artefact rejection (i.e. for the category with more trials we randomly selected 761 the same number of trials as available in the smallest category). The pre-processed raw amplitudes 762 on the 128 EEG channels, at a given time point, were used as features for the classifier. LDA 763 classification was performed separately for each participant and time point using a leave-one-out 764 cross-validation approach. This procedure resulted in a decision value (d value) for each trial and 765 time point, where the sign indicates in which category the observation had been classified (e.g., -for 766 photographs and + for line drawings in the perceptual classifier), and the value of d indicates the 767 distance to the hyper-plane that divided the two categories (with the hyper-plane being 0). This 768 distance to the hyper-plane provided us with a single trial time-resolved value that indicates how 769 confident the classifier was at assigning a given object to a given category. In order to use the 770 resulting d values for further analysis, the sign of the d values in in one category was inverted, 771 resulting in d-values that always reflected correct classification if they had a positive value, and  772 increasingly confident classification with increasingly higher values. 773 Our main intention was to identify the specific moment within a given trial at which each of the two 774 classifiers showed the highest fidelity, and to then compare the temporal order of the perceptual 775 and semantic peaks. We thus found the maximum positive d value in each trial and separately for 776 the semantic and perceptual classifiers, with the important restriction that we only used peaks with 777 a value exceeding the 95 th percentile of the classifier chance distribution (see section on 778 bootstrapping below), such as to minimize the risk of including meaningless noise peaks. The 779 resulting output from this approach allowed us to track and compare the temporal "emergence" of 780 perceptual and semantic classification within each single-trial. In addition to this single-trial analysis, 781 we also calculated the average d value peak latency for perceptual and semantic classification in 782 each participant to compare the two average temporal distributions. Note, however, that many 783 factors could obscure differences between semantic and perceptual peaks when using this average 784 approach, including variance in processing speed across trials, e.g. for more or less difficult recalls. 785 We therefore believe that the single trial values are more sensitive to differences in timing between 786 the reactivated features. 787

Generating an empirical null distribution for the classifier 788
Previous work has shown that the true level of chance performance of a classifier can differ 789 substantially from its theoretical chance level that is usually assumed to be 1/number of categories 790 ( threshold for considering only those d value peaks as significant whose values are higher than the 793 95 th percentile of this null distribution. We generated such an empirical null distribution of d values 794 by repeating our classifier analysis with randomly shuffled labels a number of times, and combined 795 this with a bootstrapping approach, as detailed in the following. 796 As a first step, we generated a set of d-value outputs that were derived from carrying out the same 797 decoding procedure as for the real data (including the leave-one-out cross-validation), but using 798 category labels that were randomly shuffled at each repetition. This procedure was carried out 799 independently per participant. On each repetition, before starting the time-resolved LDA, all trials 800 were randomly divided into two categories with the constraint that each group contained a similar 801 number of photographs and line drawings, and approximately the same amount of animate and 802 inanimate objects (the difference in trial numbers was smaller than 8%). The output of one such 803 repetition per participant was one d-value per trial and time-point, just as in the real analysis. This 804 procedure was conducted 50 times per participant for object perception (encoding) and retrieval, 805 respectively, with a new random trial split and random label assignment on each repetition. For each 806 participant we thus had a total of 51 classification outputs, one using the real labels, and 50 using 807 the randomly shuffled labels. 808 Second, we also used the shuffled label outputs in order to generate an empirical Z-score 809 distribution for our single-trial analyses. Our main statistic of interest with respect to the EEG data 810 was a Wilcoxon signed rank test comparing the order of the perceptual and semantic classifier peaks 811 on each single trial. This analysis was based on all available single trials accumulated across 812 participants, and thus resulted in a high number of degrees of freedom, with a possibly exaggerated 813 likelihood of finding a significant Z-score. We therefore tested our real data against an empirical Z-814 score distribution obtained from a series of bootstrapping analyses that were based on the same 815 data and simulated the same number of degrees of freedom. For each participant' trial, we took the 816 outputs from two different classifiers randomly selected from a sample of 52 classifiers (i.e., 50 with 817 shuffled labels, one real perceptual, and one real semantic). That is, we created two arbitrary 818 conditions per trial to make a pairwise comparison (emulating our perceptual vs. semantic 819 conditions). There was a 50:1 chance that the "pseudo-semantic" classifier contained the output of 820 the real semantic classifier, and likewise a 50:1 chance that the "pseudo-perceptual" classifier 821 contained the d-values from the real perceptual classifier. Next, we choose for each type of 822 condition the highest d value per trial in the accurate direction and in a given time window, using the 823 same constraints as for the real classifier outputs. This provided us with one peak per condition 824 (two) for every trial. To equate the number of degrees of freedom with our contrast of interest, we 825 randomly selected the same number of pairs as available in the real analysis. Finally, a Wilcoxon 826 signed rank test was used to compare the temporal distance of the d value peaks between the two 827 conditions, and the corresponding Z-value was registered, again mirroring the analysis carried out on 828 the real data. This approach was repeated with replacement for a total of 10000 times, generating 829 an empirical distribution of Z-values under the null hypothesis that there is no meaningful 830 information about an object's category in the EEG data. 831 Thirdly, to estimate our classification chance distribution for the random-effects (i.e., trial-averaged) 832 peak analyses, we used the 51 classification outputs from all participants in a bootstrapping 833 procedure (Stelzer, Chen, & Turner, 2013). On each of the bootstrapped repetitions, we randomly 834 selected one of the 51 classification outputs (50 from shuffled labels classifiers and one from a real 835 labels classifier) per participant, and calculated the d value group average based on this random 836 selection for each given time point. This procedure was repeated with replacement 10000 times. To 837 generate different distributions for the perceptual and semantic classifiers, we run this 838 bootstrapping approach two times: once where the real labels output from each subject came from 839 the semantic classifier, and once where the real d-values came from the perceptual classifier. 840

Univariate event-related potential (ERP) analysis 841
A series of cluster-based permutation tests (Monte Carlo, 2000 repetitions, clusters with a minimum 842 of 2 neighbouring channels within the FieldTrip software) was carried out in order to test for 843 differences in ERPs between the two perceptual (photograph vs. line drawing) and the two semantic 844 (animate vs. inanimate) categories, controlling for multiple comparisons across time and electrodes. 845 First, we contrasted ERPs during object presentation in the encoding phase in the time interval from 846 stimulus onset until 500ms post-stimulus. We then carried out the same type of perceptual and 847 semantic ERP contrasts during retrieval, in this case aligning all trials to the time of the button press. 848 We used the full time window from 3000ms before until 100ms after the button press, but we 849 further subdivided this time window into smaller epochs of 300ms to run a series of T-tests, again 850 using cluster statistics to correct for multiple comparisons across time and electrodes. We were 851 mainly interested in the temporal order of the ERP peaks that differentiated between perceptual 852 and semantic classes during encoding and retrieval. These peaks are based on statistically 853 meaningful clusters as described above, but we conducted no further statistical comparisons 854 between the average perceptual and semantic ERP peaks. 855