{"title": "Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity", "book": "Advances in Neural Information Processing Systems", "page_first": 15501, "page_last": 15511, "abstract": "Convolutional neural networks (CNNs) trained for object classification have been widely used to account for visually-driven neural responses in both human and primate brains. However, because of the generality and complexity of object classification, despite the effectiveness of CNNs in predicting brain activity, it is difficult to draw specific inferences about neural information processing using CNN-derived representations. To address this problem, we used learned representations drawn from 21 computer vision tasks to construct encoding models for predicting brain responses from BOLD5000---a large-scale dataset comprised of fMRI scans collected while observers viewed over 5000 naturalistic scene and object images. Encoding models based on task features predict activity in different regions across the whole brain. Features from 3D tasks such as keypoint/edge detection explain greater variance compared to 2D tasks---a pattern observed across the whole brain. Using results across all 21 task representations, we constructed a ``task graph\u2019\u2019 based on the spatial layout of well-predicted brain areas from each task. A comparison of this brain-derived task structure to the task structure derived from transfer learning accuracy demonstrate that tasks with higher transferability make similar predictions for brain responses from different regions. These results---arising out of state-of-the-art computer vision methods---help reveal the task-specific architecture of the human visual system.", "full_text": "Neural Taskonomy: Inferring the Similarity of\n\nTask-Derived Representations from Brain Activity\n\nAria Y. Wang\n\nCarnegie Mellon University\n\nariawang@cmu.edu\n\nMichael J. Tarr\n\nCarnegie Mellon University\nmichaeltarr@cmu.edu\n\nAbstract\n\nLeila Wehbe\n\nCarnegie Mellon University\n\nlwehbe@cmu.edu\n\nConvolutional neural networks (CNNs) trained for object classi\ufb01cation have been\nwidely used to account for visually-driven neural responses in both human and\nprimate brains. However, because of the generality and complexity of object\nclassi\ufb01cation, despite the effectiveness of CNNs in predicting brain activity, it is\ndif\ufb01cult to draw speci\ufb01c inferences about neural information processing using CNN-\nderived representations. To address this problem, we used learned representations\ndrawn from 21 computer vision tasks to construct encoding models for predicting\nbrain responses from BOLD5000\u2014a large-scale dataset comprised of fMRI scans\ncollected while observers viewed over 5000 naturalistic scene and object images.\nEncoding models based on task features predict activity in different regions across\nthe whole brain. Features from 3D tasks such as keypoint/edge detection explain\ngreater variance compared to 2D tasks\u2014a pattern observed across the whole brain.\nUsing results across all 21 task representations, we constructed a \u201ctask graph\u201d based\non the spatial layout of well-predicted brain areas from each task. A comparison of\nthis brain-derived task structure to the task structure derived from transfer learning\naccuracy demonstrate that tasks with higher transferability make similar predictions\nfor brain responses from different regions. These results\u2014arising out of state-of-\nthe-art computer vision methods\u2014help reveal the task-speci\ufb01c architecture of the\nhuman visual system.\n\n1\n\nIntroduction\n\nScene understanding requires the integration of space perception, visual object recognition, and the\nextraction of semantic meaning. The human brain\u2019s solution to this challenge has been elucidated\nin recent years by the identi\ufb01cation of scene-selective brain areas via comparisons between images\nof places and common objects [1]. This basic contrast has been extended across a wide variety of\nimage manipulations that have provided evidence for the neural coding of scene-relevant properties\nsuch as relative openness [2\u20134], the distance of scenes to the viewer [2,3,5], 3D spatial layout [6\u20138]\nand navigational affordances [9]. Recently, to help explain such \ufb01ndings, Lescroart and Gallant [10]\ndeveloped an encoding model using a feature space that parametrizes 3D scene structures along the\ndistance and orientation dimensions and provides a computational framework to account for human\nscene processing. Intriguingly, Lescroart and Gallant [10] were able to identify distance and openness\nwithin scenes as the dimensions that best account for neural responses in scene-selective brain areas.\nAt a higher, semantic, level, Stansbury et al. [11] found that neural responses in scene-selective brain\nareas can be predicted using scene categories that were learned from object co-occurrence statistics.\nSuch \ufb01ndings demonstrate that human scene-selective areas represent both visual and semantic scene\nfeatures. At the same time, there is still no robust model of how these different kinds of information\nare integrated both within and across brain regions.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fEncoding models are widely used in understanding feedforward information processing in human\nperception, including scene perception. Encoding models are predictive models of brain activity\nthat are able to generalize and predict brain responses to novel stimuli [12]. Researchers have also\nused encoding models to infer which dimensions are critical for prediction by comparing the weights\nlearned by the model [10,13]. One of the successes of encoding models lies in predicting low- to\nmid-level visual cortex responses in humans and primates using features that were learned via a\nconvolutional neural network trained on object recognition [14\u201317]. Most interestingly, these studies\ndemonstrate a correspondence between human neural representation and learned representations\nwithin CNN models along the perceptual hierarchy: early layers tend to predict early visual processing\nregions, whereas later layers tend to predict later visual processing regions. Similarly, researchers\nhave found that network representations from other task-driven networks, including networks trained\non speech or music related tasks, are able to explain neural responses in human auditory pathways [18].\nSuch successes are not mere coincidences but rather indications of how fundamental task-driven\nrepresentations are to both task training and to information processing in the brain.\nDespite these advances, CNN features themselves are notoriously dif\ufb01cult to interpret. First, acti-\nvations from the convolutional layers lie in extremely high-dimensional spaces and it is dif\ufb01cult to\ninterpret what each feature dimension signi\ufb01es. Second, features from a CNN tailored for a particular\nvisual task can represent any image information that is relevant to that task. As a consequence of these\ntwo issues, the feature representations learned by the network are not necessarily informative with\nrespect to the nature of visual processing in the brain despite their good performance in predicting\nbrain activity.\nTo better understand the speci\ufb01city of the information represented in the human visual processing\npathways, we adopted a different approach. Instead of choosing a generic object-classi\ufb01cation CNN\nas a source of visual features, we built encoding models with individual feature spaces obtained from\ndifferent task-speci\ufb01c networks. These tasks included mid-level features such as surface normal\nestimation, edge detection, scene classi\ufb01cation, etc. In any task-driven network, the feature space\nlearned to accomplish the task at hand should only represent information from input images that is\ntask-relevant. Therefore we can use the predictive regions from each of the models to identify the brain\nregions where speci\ufb01c task-relevant information is localized. Independently, Dwivedi and Roig [19]\nhave shown that representation similarity analysis (RSA) performed between task representations and\nbrain representations can differentiate scene-selective regions of interest (ROIs) by their preferred\ntask. For example, representations in scene-selective occipital place area (OPA) are more highly\ncorrelated with representations from a network trained to predict navigational affordances. However,\nthis study was limited to pre-de\ufb01ned regions of interest, while the task representations we identify\nspan the entire brain. Consequently, the brain regions predicted by each model provide an atlas of\nneural representation of visual tasks and allow us to further study the representational relationships\namong tasks.\nIndependently of the brain, visual tasks have relationships among them. Task representations that\nare learned speci\ufb01cally for one task can be transferred to other tasks. Computer vision researchers\ncommonly use transfer learning between tasks to save supervision and computational resources.\nIn this vein, Zamir et al. [20] recently showed that by standardizing model structure and measuring\nperformance in transfer learning, one can generate a taxonomic map for task transfer learning\n(\u201cTaskonomy\u201d). This map provides an account of how much information is shared across different\nvision tasks. Given this global task structure, we can infer clusters of information de\ufb01ned by\nsegregation of tasks, and then ask: does the brain represent visual information in the same task-\nrelevant manner?\nWe compared the relationships between tasks using both brain representations and task learning.\nThese comparisons reveal clustering of 2D tasks, 3D tasks, and semantic tasks. Compared to general\nencoding models, building individual encoding models and exploiting existing relationship among\nmodels has the potential to provide more in-depth understanding of the neural representation of visual\ninformation.\n\n2\n\n\f2 Methods1\n\n2.1 Encoding Model\n\nTo explore how and where visual features are represented in human scene processing, we extracted\ndifferent features spaces describing each of the stimulus images and used them in an encoding model\nto predict brain responses. Our reasoning is as follows. If a feature is a good predictor of a speci\ufb01c\nbrain region, information about that feature is likely encoded in that region. In this study, we \ufb01rst\nparameterized each image in the training set into values along different feature dimensions in a feature\nspace. For example, if the feature space of interest is an intermediate layer in a task-driven network,\nwe simply fed the image into the network and extracted its layer activation. These values are used as\nregressors in a ridge regression model (implemented in PyTorch; see [21]) to predict brain responses to\nthat image. Performance from the validation data is used to choose the regularization parameter in\nthe ridge regression model. We chose to use a ridge regression model instead of more complicated\nmodels in order to retain the interpretability of model weights, which may provide insights into the\nunderlying dimensions of the brain responses. For each subject, each voxel\u2019s regularization parameter\nwas chosen independently via 7-fold cross-validation based on the prediction performance of the\nvalidation data. Model performance was evaluated on the test data using both Pearson\u2019s correlation\nand coef\ufb01cient of determination (R2). To determine the signi\ufb01cance of the predictions, we ran\npermutation tests where we shuf\ufb02ed responses 5000 times, computed the correlation scores, and\nobtained FDR corrected p-values for both ROI and whole brain results.\n\n2.2 Feature Spaces\n\nTo simultaneously test representations from multiple 2D, and 3D vision tasks, we used the latent\nspace features from each of the 21 tasks in Taskonomy [20] model bank: autoencoding, colorization,\ncurvature estimation, denoising, depth estimation, edge detection (2D), edge detection (3D) or\nocclusion edges detection, keypoint detection (2D), keypoint detection (3D), depth, reshading, room\nlayout estimation, segmentation (2D), segmentation (2.5D), surface normal estimation, vanishing\npoint estimation, semantic segmentation, jigsaw puzzle, inpainting, object classi\ufb01cation and scene\nclassi\ufb01cation. In the Taskonomy training scheme, an intermediate latent space with \ufb01xed dimension\n(16 \u00d7 16 \u00d7 8) was enforced for each of these networks. We obtained these latent space activations by\nfeeding our images into each pre-trained task-speci\ufb01c network in the task bank provided with the\nTaskonomy paper. Four of the 25 tasks were excluded from this analysis because these tasks take\nmultiple images as input, while the brain responses we have are only to single images. Examples\nof these excluded tasks include camera pose estimation and egomotion estimation. We then built\nindividual ridge regression models with the extracted latent features to predict brain responses and\nmeasured the correlation between the prediction and the true response in the held-out dataset.\n\n2.3 Neural Data\n\nThe images used in this paper are from a publicly available large-scale fMRI dataset, BOLD5000 [22].\nIn the BOLD5000 study, participants\u2019 brains were scanned while they \ufb01xated at real-world images\nand judged how much they liked the image using a button press. Images in the BOLD5000 dataset\nwere chosen from standard computer vision datasets (ImageNet [23], COCO [24] and SUN [25]). The\nexperiment was run in a slow-event setting where trials are separated by 10 seconds. From BOLD5000,\nwe used data from three participants viewing 4916 unique images. These 4916 image trials are\nseparated into random training, validation, and testing sets during model \ufb01tting. Average of TR 3 and\n4 of each slow-event trial is used for model \ufb01tting and testing. Region of interest (ROI) boundaries\nthat identify category-selective brain regions in the whole-brain map presented in our results were\ngenerated directly from the ROI masks provided with the BOLD5000 dataset.\n\n2.4 Task Similarity Computation\nFor each task, we took prediction performance scores across all voxels (n \u2248 55, 000). We set the\nscore of a voxel to zero if the p-value of the correlation obtained from permutation test is above\nsigni\ufb01cance threshold (p> 0.05, FDR corrected). This gave us a performance matrix of meaningful\n\n1All code is available on https://github.com/ariaaay/NeuralTaskonomy\n\n3\n\n\fcorrelations of size m \u00d7 v, where m is the number of tasks of interest and v is the number of voxels.\nTo analyze the relationship between tasks based on neural representations, we computed pairwise\nsimilarity across tasks in the performance matrix using cosine similarity. These pairwise similarities\nwere then used to construct graphs and similarity trees among tasks. Other distance or similarity\nfunctions such as euclidean distance did not show substantial differences.\n\n3 Results\n\n3.1 Model Prediction on ROIs\n\nIn Figure 1 we show the prediction accuracy measured using the Pearson correlation coef\ufb01cient.\nThis was done for the 21 task-related feature spaces that were used to predict brain responses in\nprede\ufb01ned ROIs. Each bar shown in the \ufb01gure represents the average correlation score across all\nvoxels in that ROI. Overall, the predictions using these feature spaces\u2014which come from mid-level\ncomputer vision tasks\u2014show signi\ufb01cant correlations with brain responses, except for the feature\nspace from the curvature task. Among scene-selective regions, such as parahippocampal place area\n(PPA), retrosplenial complex (RSC), occipital place area (OPA), and lateral occipital complex (LOC),\nmodels with 3D features (e.g. keypoints, edges) show far better predictions than models with 2D\nfeatures. This \ufb01nding is consistent with the results of Lescroart and Gallant [10]. In contrast, within\nearly visual areas, the prediction results between 2D and 3D features are not differentiable. Across all\nROIs, features from object and scene classi\ufb01cation tasks provide the best predictions. For more scene\nspeci\ufb01c tasks or semantic tasks such as 3D keypoints/edges, 2.5D/semantic segmentation, depth,\ndistance, reshading, surface normal, room layout, vanishing points estimation, and object/scene\nclassi\ufb01cation, scene-selective regions are better predicted as compared to early visual areas. These\npatterns are consistent across all three participants. To quantify the consistency of results across\nsubjects, we computed correlations of prediction accuracy for each pair of subjects: 0.7957 (S1 vs.\nS2), 0.9034 (S1 vs. S3) and 0.9345 (S2 vs. S3). These results provide evidence that scene-selective\nareas show selectivity for scene-speci\ufb01c task representations.\n\n3.2 Model Prediction Across the Whole Brain\n\nPrediction performance in pre-de\ufb01ned ROIs may omit relevant information arising in other brain\nregions. In Figure 2 and 3 we show prediction performance across the entire brain in a \ufb02attened\nview (generated using Pycortex [26]). Figure 2 shows the raw prediction performance as correlation\ncoef\ufb01cients for each task feature space. Figure 3 shows a contrast in prediction between 3D and 2D\nkeypoints as well as edges. In this \ufb01gure, red-colored voxels are better predicted by 3D features\nthan 2D features, and vice-versa for blue-colored voxels; white-colored voxels are well predicted by\nboth features. We \ufb01nd that 3D features make better predictions for scene-selective regions\u2014those\ndelimited by ROI borders, while 3D and 2D features seem to predict early visual areas equally well.\nFigure 2 shows that prediction results are consistent across three participants despite anatomical\ndifferences in their brain structures and 3 shows that the results are consistent across tasks.\nModel performance using feature spaces from other tasks are shown in Figure 4. Here we plot\n6 of the 21 tasks, and the remaining \ufb01gures for this sample subject (subject 1) are provided\nin the appendix. Voxels with insigni\ufb01cant predictions (p \u2265 0.05, FDR corrected) are masked\nin these \ufb01gures. Prediction performances of all tasks and all three subjects can be viewed at\nhttps://cs.cmu.edu/\u223cneural-taskonomy.\nTo provide a better estimate of the variance ceiling, we ran ridge regression to predict responses\nof one subject from another. In Figure 5 we show the prediction correlation for each subject from\nthe remaining two subjects. The average correlation between predictions and true responses across\nvoxels for each subject are: 0.0931, 0.0932, 0.112, as shown by the black lines on each plot. The\nhistogram includes low signal to noise ratio (SNR) voxels that are not engaged by the task. The\nhistograms distribution indicates that the accuracy we obtained on signi\ufb01cant voxels across the whole\nbrain using features from various task is close to the ceiling. Cross subject prediction results from\neach pair of subjects is provided in the appendix. Note that we are predicting single-trial fMRI data\nwith no repetitions, which leads to a lower signal to noise ratio (and therefore lower variance ceiling)\nthan other fMRI studies that average repetitions.\n\n4\n\n\fFigure 1: Pearson correlation coef\ufb01cient between predicted and true responses across tasks.\nEach sub-\ufb01gure corresponds to a particular participant. Colors in the legend are arranged by columns.\nFeatures from 3D tasks, compared to those from 2D tasks, predict better in OPA, PPA, RSC, and\nLOC.\n\nFigure 2: Whole brain prediction correlation using task representation of scene classi\ufb01cation\nnetwork. The \ufb02at maps are cropped from the occipital regions of the brain. The upper zoom-out view\nshows the relative locations of the \ufb02at maps. Lower colored \ufb01gures are the prediction performance\nacross 3 participants. Prediction results are consistent across subjects.\n\n5\n\n0.00.10.2Correlations (r)Subject = 10.00.10.2Correlations (r)Subject = 2OPAPPALOCEarlyVisRSCROI0.00.10.2Correlations (r)Subject = 3TasksAutoencodingDenoisingColorCurvature2D EdgesOcculusion Edges2D Keypoint3D Keypoint2D Segm.2.5D Segm.Semantic Segm.DepthDistanceReshadingNormalsLayoutVanishing Pts.Object ClassScene ClassJigsawInpaintingCorrelation\fFigure 3: Contrast of prediction performance (measured with Pearson correlation coef\ufb01cients)\nbetween 2D and 3D features in one sample subject (subject 1). The \ufb02at maps are cropped similar\nto as in Figure 2. The color map indicates the difference in correlation coef\ufb01cients: red: 3D > 2D;\nblue: 2D > 3D. 3D task features predict better in scene selective regions and in more anterior parts of\nthe brain.\n\n(a) 2D Segmentation\n\n(b) Semantic segmentation\n\n(c) Distance estimation\n\n(d) Reshading\n\n(e) Object Classi\ufb01cation\n\n(f) Scene Classi\ufb01cation\n\nFigure 4: Predictive voxels using tasks features from Taskonomy [20] in one sample subject\n(subject 1). Predictive regions of different tasks differ from each other across tasks.\n\n3.3 Evaluation of Neural Representation Similarity\n\nTo this point we have shown that the neural prediction maps across tasks differ from one another;\nat the same time, there are many overlapping voxels across the predicted regions. Importantly, this\npattern of voxels as predicted by the tasks can be exploited and used to infer task relationships in\nthe brain. We computed task similarity averaged across 3 subjects using the methods discussed\n\n6\n\n3D vs. 2D Keypoints3D vs. 2D Edges3D2DCorrelation (r)0.050.2\fFigure 5: Noise ceiling derived from cross subjection prediction. Each sub\ufb01gure is a histogram\nof correlation scores across voxels. The black lines on each sub\ufb01gure indicate the average correlation\nvalues.\n\nin 2.4 (Figure 6). The individual patterns of task similarity are almost identical across 3 subjects.\nMore speci\ufb01cally, correlations (Pearson\u2019s r) between similarity matrices for each pair of subjects are:\n0.9610 (S1 vs. S2), 0.9477 (S1 vs. S3) and 0.9407 (S2 vs. S3). In this comparison across the whole\nbrain, tasks such as 2.5D segmentation, room layout estimation, surface normal estimation, scene\nclassi\ufb01cation etc. have similar predictions patterns.\n\nFigure 6: Prediction similarity matrix across 21 tasks, averaged across 3 subjects. A large\nsimilarity value between task X and Y indicates encoding models with features representation from\ntask X and Y have similar predictions of brain responses.\n\n3.4 Task Similarity Tree\n\nTo further explore the relationship between tasks as represented in the brain, we ran hierarchical\nclustering on the prediction correlation results and visualized the clustering results as dendrograms.\nFigure 7 compares the task similarity tree based on transferring-out patterns in the original Taskonomy\npaper [20], with the task similarity tree generated based on similarity in voxel prediction performance.\nTrees independently generated for each subject show great similarity. In the Taskonomy result, tasks\nare clustered into 3D (indicated in green), 2D (blue), low-dimensional geometric (red) and semantic\n(purple) tasks. Interestingly, the tree derived from brain representation also shows a similar structure:\nsemantic, 2D and 3D tasks are clustered together. The differences between the two similarity trees\n\n7\n\n0.10.00.10.20.30.40.5S10.10.00.10.20.30.40.5S20.10.00.10.20.30.40.5S3Object Class2D Segm.Semantic Segm.Scene ClassDenoising2D EdgesOcclusion EdgesJigsawAutoencodingColor3D KeypointReshadingDistanceDepthNormalsLayout2.5D Segm.2D KeypointInpaintingCurvatureVanishing Pts.Object Class2D Segm.Semantic Segm.Scene ClassDenoising2D EdgesOcclusion EdgesJigsawAutoencodingColor3D KeypointReshadingDistanceDepthNormalsLayout2.5D Segm.2D KeypointInpaintingCurvatureVanishing Pts.0.20.40.60.81.0\fFigure 7: Task Trees from (a) Taskonomy [20] and (b-d) brain representation of tasks from 3\ndiffernet subjects. Tasks in (b-d) are colored according to colors in (a). Similar clusters of 2D (in\nblue), 3D (green) and semantic (purple) tasks are found among neural taskonomy trees. Clustering\nresults are highly consistent across three subjects.\n\nmay be due to low absolute performance of the encoding model. For example, the model with features\nfrom curvature estimation task has less than 10 signi\ufb01cant voxels in some subjects which may lead to\nbias in the representation of the task tree. Overall the similarity between two task trees shows that,\nat a coarse level, neural representation of task information is similar to that found through transfer\nlearning. The clustering and dendrogram structures are stable across subjects and across different\nlinkage criteria. Aside from using \u201caverage\u201d linkage for clustering, as shown here, we also used\n\u201cward\u201d linkage criterion (shown in the appendix) and obtained similar structures.\n\n4 Discussion\n\nThe architecture of the primate visual system re\ufb02ects a series of computational mechanisms that\nenable high performance for accomplishing evolutionarily adaptive tasks [27]. However, the precise\n\n8\n\n2DSegm.2DKeypointsColor2DEdgesVanishingPts.DenoisingJigsawInpaintingAutoencodingCurvatureSemanticSegm.OcclusionEdgesReshadingDistanceLayoutDepth3DKeypoint2.5DSegm.NormalsObjectClass.SceneClass.(c)NeuralTaskonomy(subject2)(b)NeuralTaskonomy(subject1)2DSegm.2DKeypointColorJigsawVanishingPts.AutoencodingCurvature2DEdgesDenoisingInpaintingDistance3DKeypointReshadingSemanticSegm.LayoutOcclusionEdgesNormals2.5DSegm.DepthObjectClass.SceneClass.2DSegm.2DEdgesColorJigsawCurvatureVanishingPts.InpaintingAutoencoding2DKeypointDenoisingDistanceNormals2.5DSegm.3DKeypointSemanticSegm.LayoutDepthReshadingOcclusionEdgesObjectClass.SceneClass.(d)NeuralTaskonomy(subject3)(a)Taskonomyfromtransferlearning\fnature of these tasks remains unknown because of the limitations of neuroscience data collection\nmethods and the lack of interpretability of intermediate visual representations. To address these\nissues we leveraged the space of vision tasks learned through transfer learning in Taskonomy [20]\nand the recent availability of a larger-scale human functional imaging dataset, BOLD5000 [22]. One\nchallenge we faced was the substantial difference between the image distributions of BOLD5000\n(which contains general objects and scenes) and the Taskonomy dataset (which includes indoor\nscenes exclusively). As such, when we applied the pre-trained Taskonomy models to BOLD5000\nimages, we found that these models didn\u2019t perform as well as on the Taskonomy dataset, especially\nfor the outdoor images used in BOLD5000. Such inconsistency in image distribution is unavoidably\nre\ufb02ected in the encoding model performance and hinders us from making more speci\ufb01c claims about\ntask spaces in the brain. One solution to this issue would be to use a more general computational\nmodel of visual tasks, as well as a larger brain dataset based on more images, both of which are\noutside of the scope of this paper.\nIn the future we would also like to investigate the unique and shared variance explained by each task.\nAt present we are still unclear as to what transferability between tasks within Taskonomy predicts for\nsimilarity in task representations within the brain.\nFinally, although our whole brain prediction maps do seem to suggest the involvement of additional\nfunctional brain areas beyond the pre-de\ufb01ned ROIs, we strongly feel that making claims about new\nfunctionally-de\ufb01ned brain areas would be premature given our current data and analysis. We believe\nthat to make robust claims about new \"functional territories\", we would \ufb01rst need to run additional\nvalidation experiments in which speci\ufb01c manipulations are used to establish that speci\ufb01c brain regions\nare sensitive to the tasks in question.\n\n5 Conclusion\n\nOur results reveal that task-speci\ufb01c representations in neural networks are useful in predicting brain\nresponses and localizing task-related information in the brain. One of the main \ufb01ndings is that\nfeatures from 3D tasks, compared to those from 2D tasks, predict a distinct part of visual cortex. In\nthe future we will incorporate features from other tasks to obtain a more comprehensive picture of\ntask representation in the brain.\nFor years neuroscientists have focused on recovering which parts of the brain represent a given type\nof information. However, what are the computational principles behind the encoding of information\nin the brain? We observe feedforward hierarchies in the visual pathways, but what are the stages of\ninformation processing? To date, we have few satisfying answers. The ultimate goal in studying task\nrepresentation in the brain is to answer some of these questions. We exploited the task relationship\nfound in transfer learning and used it as a ground truth of visual information space to study the neural\nrepresentation of visual and semantic information. In sum, our paper provides an initial attempt in\nusing task relationships to answer broader questions of neural information processing.\n\nAcknowledgments\n\nWe thank the BOLD5000 team for providing the public accessible data. We would also like to thank\nJayanth Koushik for the help with model \ufb01tting, paper edits and result visualizations, as well as\nNadine Chang for useful discussions regarding the project.\n\n9\n\n\fReferences\n[1] Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment.\n\nNature, 392(6676):598, 1998.\n\n[2] Dwight J Kravitz, Cynthia S Peng, and Chris I Baker. Real-world scene representations in\nhigh-level visual cortex: it\u2019s the spaces more than the places. Journal of Neuroscience, 31(20):\n7322\u20137333, 2011.\n\n[3] Soojin Park, Talia Konkle, and Aude Oliva. Parametric coding of the size and clutter of natural\n\nscenes in the human brain. Cerebral cortex, 25(7):1792\u20131805, 2014.\n\n[4] Assaf Harel, Dwight J Kravitz, and Chris I Baker. Deconstructing visual scenes in cortex:\n\ngradients of object and spatial layout information. Cerebral Cortex, 23(4):947\u2013957, 2012.\n\n[5] Mark D Lescroart, Dustin E Stansbury, and Jack L Gallant. Fourier power, subjective distance,\nand object categories all provide plausible models of BOLD responses in scene-selective visual\nareas. Frontiers in computational neuroscience, 9:135, 2015.\n\n[6] Katrina Ferrara and Soojin Park. Neural representation of scene boundaries. Neuropsychologia,\n\n89:180\u2013190, 2016.\n\n[7] Frederik S Kamps, Joshua B Julian, Jonas Kubilius, Nancy Kanwisher, and Daniel D Dilks. The\noccipital place area represents the local elements of scenes. Neuroimage, 132:417\u2013424, 2016.\n\n[8] Simon Kornblith, Xueqi Cheng, Shay Ohayon, and Doris Y Tsao. A network for scene\n\nprocessing in the macaque temporal lobe. Neuron, 79(4):766\u2013781, 2013.\n\n[9] Michael F Bonner and Russell A Epstein. Coding of navigational affordances in the human\nvisual system. Proceedings of the National Academy of Sciences, 114(18):4793\u20134798, 2017.\n\n[10] Mark D Lescroart and Jack L Gallant. Human scene-selective areas represent 3d con\ufb01gurations\n\nof surfaces. Neuron, 101(1):178\u2013192, 2019.\n\n[11] Dustin E Stansbury, Thomas Naselaris, and Jack L Gallant. Natural scene statistics account for\nthe representation of scene categories in human visual cortex. Neuron, 79(5):1025\u20131034, 2013.\n\n[12] Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and\n\ndecoding in fmri. Neuroimage, 56(2):400\u2013410, 2011.\n\n[13] Alexander G Huth, Wendy A De Heer, Thomas L Grif\ufb01ths, Fr\u00e9d\u00e9ric E Theunissen, and Jack L\nGallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532\n(7600):453, 2016.\n\n[14] Pulkit Agrawal, Dustin Stansbury, Jitendra Malik, and Jack L Gallant. Pixels to voxels:\n\nModeling visual representation in the human brain. arXiv preprint arXiv:1407.5104, 2014.\n\n[15] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J\nDiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual\ncortex. Proceedings of the National Academy of Sciences, 111(23):8619\u20138624, 2014.\n\n[16] Umut G\u00fc\u00e7l\u00fc and Marcel AJ van Gerven. Deep neural networks reveal a gradient in the\ncomplexity of neural representations across the ventral stream. Journal of Neuroscience, 35\n(27):10005\u201310014, 2015.\n\n[17] Michael Eickenberg, Alexandre Gramfort, Ga\u00ebl Varoquaux, and Bertrand Thirion. Seeing it all:\nConvolutional network layers map the function of the human visual system. NeuroImage, 152:\n184\u2013194, 2017.\n\n[18] Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haignere, and Josh H\nMcDermott. A task-optimized neural network replicates human auditory behavior, predicts\nbrain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630\u2013644, 2018.\n\n[19] Kshitij Dwivedi and Gemma Roig. Task-speci\ufb01c vision models explain task-speci\ufb01c areas of\n\nvisual cortex. BioRxiv, page 402735, 2018.\n\n10\n\n\f[20] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio\nIn Proceedings of the IEEE\n\nSavarese. Taskonomy: Disentangling task transfer learning.\nConference on Computer Vision and Pattern Recognition, pages 3712\u20133722, 2018.\n\n[21] Jayanth Koushik. torch-gel. https://github.com/jayanthkoushik/torch-gel, 2017.\n\n[22] Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael J Tarr, and Elissa M\nAminoff. BOLD5000, a public fMRI dataset while viewing 5000 visual images. Scienti\ufb01c data,\n6(1):49, 2019.\n\n[23] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng\nHuang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.\nImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision\n(IJCV), 115(3):211\u2013252, 2015. doi: 10.1007/s11263-015-0816-y.\n\n[24] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays,\nPietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll\u00e1r. Microsoft coco: Common\nobjects in context, 2014.\n\n[25] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database:\nLarge-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference\non Computer Vision and Pattern Recognition, pages 3485\u20133492. IEEE, 2010.\n\n[26] James S Gao, Alexander G Huth, Mark D Lescroart, and Jack L Gallant. Pycortex: an interactive\n\nsurface visualizer for fmri. Frontiers in neuroinformatics, 9:23, 2015.\n\n[27] Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand\n\nsensory cortex. Nature neuroscience, 19(3):356, 2016.\n\n11\n\n\f", "award": [], "sourceid": 8988, "authors": [{"given_name": "Aria", "family_name": "Wang", "institution": "Carnegie Mellon University"}, {"given_name": "Michael", "family_name": "Tarr", "institution": "Carnegie Mellon University"}, {"given_name": "Leila", "family_name": "Wehbe", "institution": "Carnegie Mellon University"}]}