Visual perception of materials and their properties

- Humans are remarkably good at visually perceiving materials.
- Material perception involves some unique computational challenges.
- Inferring true physical parameters (e.g., gloss, viscosity) is often impossible.
- We suggest the brain instead uses generative models to capture typical appearance.
- Such models might explain the curious pattern of errors seen in many studies.

Misidentifying materials-such as mistaking soap for pâté, or vice versa-could lead to some pretty messy mishaps. Fortunately, we rarely suffer such indignities, thanks largely to our outstanding ability to recognize materials-and identify their properties-by sight. In everyday life, we encounter an enormous variety of materials, which we usually distinguish effortlessly and without error. However, despite its subjective ease, material perception poses the visual system with some unique and significant challenges, because a given material can take on many different appearances depending on the lighting, viewpoint and shape. Here, I use observations from recent research on material perception to outline a general theory of material perception, in which I suggest that the visual system does not actually estimate physical parameters of materials and objects. Instead-I argue-the brain is remarkably adept at building 'statistical generative models' that capture the natural degrees of variation in appearance between samples. For example, when determining perceived glossiness, the brain does not estimate parameters of the BRDF. Instead, it uses a constellation of low- and mid-level image measurements to characterize the extent to which the surface manifests specular reflections. I argue that these 'statistical appearance models' are both more expressive and easier to compute than physical parameters, and therefore represent a powerful middle way between a 'bag of tricks' and 'inverse optics'.