At fixed stellar mass , the size of low redshift early-type galaxies is found to be a factor of two larger than their counterparts at z \sim 1 , a result with important implications to galaxy formation models . In this paper , we have explored the buildup of the local mass-size relation of elliptical galaxies using two visually classified samples . At low redshift we compiled a subsample of 2,656 elliptical galaxies from the Sloan Digital Sky Survey ( SDSS ) , whereas at higher redshift ( up to z \sim 1 ) we extracted a sample of 228 objects from the HST/ACS images of the Great Observatories Origins Deep Survey ( GOODS ) . All the galaxies in our study have spectroscopic data , allowing us to determine the age and mass of the stellar component . Contrary to previous claims in the literature , using the fossil record information contained in the stellar populations of our local sample , we do not find any evidence for an age segregation at a given stellar mass depending on the size of the galaxies . At a fixed dynamical mass there is only a \buildrel < \over { \sim } 9 % size difference in the two extreme age quartiles of our sample . Consequently , the local evidence does not support a scenario whereby the present-day mass-size relation has been progressively established via a bottom-up sequence , where older galaxies occupy the lower part this relation , remaining in place since their formation . We do not find any age segregation difference in our high-z sample either . Therefore , we find a trend in size that is insensitive to the age of the stellar populations , at least since z \sim 1 . This result supports the idea that the stellar mass-size relation is formed at z \sim 1 , with all galaxies populating a region which roughly corresponds to 1/2 of the present size distribution . We have explored two possible scenarios for size growth : puffing up or minor merging . The fact that the evolution in size is independent of stellar age , together with the absence of an increase in the scatter of the relationship with redshift does not support the puffing up mechanism . The observational evidence , however , can not reject at this stage the minor merging hypothesis . We have made an estimation of the number of minor merger events necessary to bring the high-z galaxies into the local relation compatible with the observed size evolution . Since z=0.8 , if the merger mass ratio is 1:3 we estimate \sim 3 \pm 1 minor mergers and if the ratio is 1:10 we obtain \sim 8 \pm 2 events .