Do semantic parts emerge in Convolutional Neural Networks?

Semantic object parts can be useful for several visual recognition tasks. Lately, these tasks have been addressed using ConvolutionalNeural Networks (CNN), achieving outstanding results. In this talk I will introduce our latest work, in which we study whether CNNs learn semantic parts in their internal representation. We investigate the responses of convolutional filters and try to associate their stimuli with semantic parts. While previous efforts studied this matter by visual inspection, we perform an extensive quantitative analysis based on ground-truth part bounding-boxes, exploring different layers, network depths, and supervision levels. Even after assisting the filters with several mechanisms to favor this association, we find that only about 25% of the semantic parts in PASCAL-Part dataset  emerge in the popular AlexNet network finetuned for object detection. Interestingly, both the supervision level and the network depth do not seem to significantly affect the emergence of parts. Finally, we investigate if filters are responding to recurrent discriminative patches as opposed to semantic parts. We discover that the discriminative power of the network can be attributed to a few discriminative filters specialized to each object class. Moreover, about 60% of them can be associated with semantic parts. The overlap between discriminative and semantic filters might be the reason why previous studies suggested a stronger emergence of semantic parts, based on visual inspection only.