A large body of research supports the hypothesis that the human visual system does not process a face as a collection of separable facial features but as an integrated perceptual whole. One common assumption is that we quickly build holistic representations to extract useful second-order information provided by the variation between the faces of different individuals. An alternative account suggests holistic processing is a fast, early grouping process that first serves to distinguish faces from other competing objects. From this perspective, holistic processing is a quick initial response to the first-order information present in every face. To test this hypothesis we developed a novel paradigm for measuring the face inversion effect, a standard marker of holistic face processing, that measures the minimum exposure time required to discriminate between two stimuli. These new data demonstrate that holistic processing operates on whole upright faces, regardless of whether subjects are required to extract first- or second-level information. In light of this, we argue that holistic processing is a general mechanism that may occur at an earlier stage of face perception than individual discrimination to support the rapid detection of face stimuli in everyday visual scenes.