Decades ago, the first category-selective region in the human brain was described: The fusiform face area. Afterwards, neuroimaging studies have uncovered a complex landscape of focal and distributed selectivity in lateral occipital and ventral occipitotemporal cortex, with some apparent discrepancies in terms of what the primary organizational principles are. I will discuss how we can explain the organization of this large brain region in terms of the interplay of multiple factors such as visual statistics, computational constraints, and behavioral goals. Together, these factors once again point to visual categories as an organizational principle in the human brain. Empirical studies that dissociate category membership from other relevant dimensions support this primary role of category selectivity.