We investigate the origin of the evolution of the population-averaged size of quenched galaxies ( QGs ) through a spectroscopic analysis of their stellar ages . The two most favoured scenarios for this evolution are either the size growth of individual galaxies through a sequence of dry minor merger events , or the addition of larger , newly quenched galaxies to the pre-existing population ( i.e. , a progenitor bias effect ) . We use the 20k zCOSMOS-bright spectroscopic survey to select bona fide quiescent galaxies at 0.2 < z < 0.8 . We stack their spectra in bins of redshift , stellar mass and size to compute stellar population parameters in these bins through fits to the rest-frame optical spectra and through Lick spectral indices . We confirm a change of behaviour in the size-age relation below and above the \sim 10 ^ { 11 } \mathrm { M } _ { \odot } stellar mass scale : In our 10.5 < \log \mathrm { M _ { * } / M _ { \odot } } < 11 mass bin , over the entire redshift window , the stellar populations of the largest galaxies are systematically younger than those of the smaller counterparts , pointing at progenitor bias as the main driver of the observed average size evolution at sub-10 ^ { 11 } \mathrm { M } _ { \odot } masses . In contrast , at higher masses , there is no clear trend in age as a function of galaxy size , supporting a substantial role of dry mergers in increasing the sizes of these most massive QGs with cosmic time . Within the errors , the [ \alpha /Fe ] abundance ratios of QGs are ( i ) above-solar over the entire redshift range of our analysis , hinting at universally short timescales for the buildup of the stellar populations of QGs , and ( ii ) similar at all masses and sizes , suggesting similar ( short ) timescales for the whole QG population and strengthening the role of mergers in the buildup of the most massive QGs in the Universe .