We present an analysis of the stellar mass growth over the last 10 Gyrs ( z \leq 2 ) using a unique large sample of galaxies selected at 3.6 \mu m . We have assembled accurate photometric and spectroscopic redshifts for \sim 21200 and 1500 galaxies , respectively , with F ( 3.6 \mu m ) \geq 9.0 \mu Jy by combining data from Spitzer-SWIRE IRAC , the VIMOS VLT Deep Survey ( VVDS ) , UKIDSS and very deep optical CFHTLS photometry . We split our sample into quiescent ( red ) and active ( blue ) galaxies on the basis of an SED fitting procedure that we have compared with the strong rest-frame color bimodality ( NUV - r ^ { \prime } ) _ { ABS } . The present sample contains \sim 4400 quiescent galaxies . Our measurements of the K-rest frame luminosity function and luminosity density evolution support the idea that a large fraction of galaxies is already assembled at z \sim 1.2 , with almost 80 % and 50 % of the active and quiescent populations already in place , respectively . Based on the analysis of the evolution of the stellar mass-to-light ratio ( in K-band ) for the spectroscopic sub-sample , we derive the stellar mass density for the entire sample . We find that the global evolution of the stellar mass density is well reproduced by the star formation rate derived from UV based measurements when an appropriate dust correction is applied , which supports the idea of an initial mass function that is on average universal . Over the last 8Gyrs ( z \leq 1.2 ) we observe that the stellar mass density of the active population shows a modest mass growth rate ( \dot { \rho } \sim 0.005 ( \pm 0.005 ) M _ { \odot } / Mpc ^ { 3 } / yr ) , consistent with a constant stellar mass density , \rho _ { \star } ^ { active } \sim 3.1 10 ^ { 8 } M _ { \odot } / Mpc ^ { 3 } . In contrast , an increase by a factor of \sim 2 for the quiescent population over the same timescale is observed . As a consequence , the growth of the stellar mass in the quiescent population must be due to the shutoff of star formation in active galaxies that migrate into the quiescent population . We estimate this stellar mass flux to be \dot { \rho } _ { A \rightarrow Q } \sim 0.017 ( \pm 0.004 ) M _ { \odot } / Mpc ^ { 3 } / yr , which balances the major fraction of new stars born according to our best SFR estimate ( \dot { \rho } = 0.025 ( \pm 0.003 ) M _ { \odot } / Mpc ^ { 3 } / yr ) . From z = 2 to z = 1.2 , we observe a major build-up of the quiescent population with an increase by a factor of \sim 10 in stellar mass ( a mass growth rate of \sim 0.063 M _ { \odot } / Mpc ^ { 3 } / yr ) . This rapid evolution suggests that we are observing the epoch when , for the first time in the history of the universe , an increasing fraction of galaxies end their star formation activity and start to build up the red sequence .