We use a deep K _ { AB } \leq 25 galaxy sample in the Hubble Deep Field South to trace the evolution of the cosmological stellar mass density from z \simeq 0.5 to z \simeq 3 . We find clear evidence for a decrease of the average stellar mass density at high redshift , 2 \leq z \leq 3.2 , that is 15 ^ { +25 } _ { -5 } \% of the local value , two times higher than what observed in the Hubble Deep Field North . To take into account for the selection effects , we define a homogeneous subsample of galaxies with 10 ^ { 10 } M _ { \odot } \leq M _ { * } \leq 10 ^ { 11 } M _ { \odot } : in this sample , the mass density at z > 2 is 20 ^ { +20 } _ { -5 } \% of the local value . In the mass–limited subsample at z > 2 , the fraction of passively fading galaxies is at most 25 % , although they can contribute up to about 40 % of the stellar mass density . On the other hand , star–forming galaxies at z > 2 form stars with an average specific rate at least < \dot { M } / M _ { * } > \simeq 4 \times 10 ^ { -10 } yr ^ { -1 } , 3 times higher than the z \leq 1 value . This implies that UV bright star–forming galaxies are substancial contributors to the rise of the stellar mass density with cosmic time . Although these results are globally consistent with \Lambda –CDM scenarios , the present rendition of semi analytic models fails to match the stellar mass density produced by more massive galaxies present at z > 2 .