Outstanding supervised classification performances obtained by CNNs indicate they have the ability to create relevant invariants for classification. We show that this can be achieved through progressive invariance incorporation and as well via perfectly invertible architectures. Illustrations are given through Hybrid Scattering Networks, based on a geometric representation, and $i$-RevNets, a class of invertible CNNs. We explicit several empirical properties, like progressive linear separability, in order to shed light on the inner mechanisms implemented by CNNs.