Using a sample of \sim 28,000 sources selected at 3.6–4.5 microns with Spitzer observations of the HDF-N , the CDF-S , and the Lockman Hole ( surveyed area : \sim 664 arcmin ^ { 2 } ) , we study the evolution of the stellar mass content of the Universe at 0 < z < 4 . We calculate stellar masses and photometric redshifts , based on \sim 2,000 templates built with stellar population and dust emission models fitting the UV-to-MIR spectral energy distributions of galaxies with spectroscopic redshifts . We estimate stellar mass functions for different redshift intervals . We find that 50 % of the local stellar mass density was assembled at 0 < z < 1 ( average SFR : 0.048 \mathcal { M } _ { \sun } yr ^ { -1 } Mpc ^ { -3 } ) , and at least another 40 % at 1 < z < 4 ( average SFR : 0.074 \mathcal { M } _ { \sun } yr ^ { -1 } Mpc ^ { -3 } ) . Our results confirm and quantify the “ downsizing ” scenario of galaxy formation . The most massive galaxies ( \mathcal { M } > 10 ^ { 12.0 } \mathcal { M } _ { \sun } ) assembled the bulk of their stellar content rapidly ( in 1-2 Gyr ) beyond z \sim 3 in very intense star formation events ( producing high specific SFRs ) . Galaxies with 10 ^ { 11.5 } < \mathcal { M } < 10 ^ { 12.0 } \mathcal { M } _ { \sun } assembled half of their stellar mass before z \sim 1.5 , and more than 90 % of their mass was already in place at z \sim 0.6 . Galaxies with \mathcal { M } < 10 ^ { 11.5 } \mathcal { M } _ { \sun } evolved more slowly ( presenting smaller specific SFRs ) , assembling half of their stellar mass below z \sim 1 . About 40 % of the local stellar mass density of 10 ^ { 9.0 } < \mathcal { M } < 10 ^ { 11.0 } \mathcal { M } _ { \sun } galaxies was assembled below z \sim 0.4 , most probably through accretion of small satellites producing little star formation . The cosmic stellar mass density at z > 2.5 is dominated by optically faint ( R \gtrsim 25 ) red galaxies ( Distant Red Galaxies or BzK sources ) which account for \sim 30 % of the global population of galaxies , but contribute at least 60 % to the cosmic stellar mass density . Bluer galaxies ( e.g. , Lyman Break Galaxies ) are more numerous but less massive , contributing less than 50 % to the global stellar mass density at high redshift .