Research

Representational Structure and Dimensionality in Visual Cortex

1. Scale-invariant representations of natural images in visual cortex
   • Led by: Raj Magesh Gauthaman, PhD student
   • Summary: We examine the covariance spectra of neural responses to natural images and reveal unexpectedly high-dimensional structure in a large-scale human fMRI dataset. This suggests that cortical representations are highly expressive even in “high-level” visual regions that have previously been thought to encode low-dimensional representations. Interestingly, we observe similar power-law spectra when computing cross-covariances across individuals, demonstrating that these high-dimensional cortical representations are shared to a surprising extent across people.
2. High-dimensional latent manifolds as predictors of individual differences
   • Led by: Kelsey Han, PhD student
   • Summary: While a prominent theory proposes that the visual system transforms high-dimensional sensory inputs into simpler, low-dimensional representations, recent theoretical and empirical work suggests that the dimensionality of visual cortical representations may be more extensive than previously thought. Using cross-decomposition, we investigate the possibility that even low-variance dimensions in cortical population activity are critical to human vision and that reliable individual differences in visual experience are captured by these high-dimensional codes.
3. Universal dimensions of vision
   • Led by: Ray Chen, PhD student
   • Summary: We observe that diverse designs of neural networks all yield similarly good models of the human visual cortex. This suggests that the subset of model representations that align with cortical activities are general-purpose and can be learned independent of the task or architectural constraints on these models. By comparing the representations of widely varied models, we provide evidence that the universality of a feature among many neural networks is a strong indicator of its presence in human cortical representation.
4. Hierarchical organization of social action features along the lateral visual pathway
   • Led by: Emalie McMahon, PhD student (with Leyla Isik)
   • Summary: Recent theoretical work suggests that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. We investigate the hierarchical organization of naturalistic social visual content in this lateral stream using a condition-rich fMRI experiment and within-subject encoding models. Our results show that low-level visual features are represented in early visual cortex (EVC) and middle temporal (MT) area, mid-level visual social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Notably, communicative interactions explain unique variance in STS regions after accounting for all other labeled features. These findings support the representation of increasingly abstract social visual content along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of this pathway.
5. Neural manifold geometries underlying emergent model-to-brain similarity
   • Led by: Colin Conwell, PostDoc
   • Summary: What do deep neural network models of biological vision tell us about the computational structure of the brain? A now common finding in visual neuroscience is that many different kinds of deep neural network models – with different architectures, tasks, and training diets – are all comparably good predictors of image-evoked brain activity in the ventral visual cortex. This relative parity of highly diverse models may at first seem to undermine the common intuition that we can use these models to infer the key computational principles that govern the visual brain. In this line of work, we use metrics derived from statistical physics and high-dimensional geometry to derive more proximate, structural intuitions for what makes one model more brain-like than another. We find these metrics can in many cases be highly informative of the underlying differences between models, even when they initially appear to be almost equally similar to the brain, and provide computationally-principled explanations of what makes a brain-like representation brain-like in the first place.

Learning Dynamics and Objectives in Visual Cortex Models

1. Studying the dynamics of learning in brain representational alignment
   • Led by: Yash Mehta, PhD student
   • Summary: Training artificial deep neural networks for object recognition tasks enhances their alignment with the visual cortex. During training, a recent study made an intriguing observation: only the first few eigenvalues of the weight matrices change significantly, while the rest remain close to their initial values. This phenomenon is not yet well characterized or understood. Could this be a key factor in achieving a high brain-alignment score? How does this process evolve over the course of training? This project aims to explore the evolution of representations in deep networks during training and their relationship to visual system alignment.
2. Computational models of visual cortex without supervised learning
   • Led by: Atlas Kazemian, Lab Manager
   • Summary: Current deep learning models of the visual cortex make assumptions about the architecture and learning objectives of biological visual systems. We explore an alternative approach, in which we study how brain-relevant dimensions emerge in untrained models solely from the statistics of natural scenes, without relying on supervised learning.
3. Local unsupervised learning algorithm for building a visual hierarchy
   • Led by: Ananya Passi, PhD student
   • Summary: Deep neural networks (DNNs) are the leading computational models of visual cortex but are trained using biologically implausible backpropagation. We are developing an algorithm for building a hierarchy of visual features using only local unsupervised learning, without backpropagation. Our work identifies a new approach for learning a visual hierarchy consistent with principles of learning in biology, requires no labels or tasks, and may sufficiently account for a large fraction of visual cortex representations.