The Cosmic Microwave Background (CMB) has provided us an incredible picture of the early Universe. After 25 years of precision CMB measurements, we have reached a stage at which we have to conclude that the primordial Universe can in principle be explained by a very simple model containing only 2 degrees of freedom. The challenge for the next decade(s) is to find deviations form this very simple picture, since these would give us profound insight into the workings of the cosmos. In this talk I will discuss the search for such deviations in the form of the breaking of scale invariance (i.e. features), primordial tensor modes and non-Gaussianities. I will argue that each of these would independently provide clues of new physics beyond the 2-parameter paradigm and demonstrate that significant information on primordial physics remains to be harvested from CMB anisotropies in the foreseeable future.