We estimate the galaxy stellar mass function and stellar mass density for star-forming and quiescent galaxies with 0.2 < z < 4 . We construct a large , deep ( K _ { s } < 24 ) sample of 220,000 galaxies selected using the new UltraVISTA DR1 data release . Our analysis is based on precise 30-band photometric redshifts . By comparing these photometric redshifts with 10,800 spectroscopic redshifts from the zCOSMOS bright and faint surveys , we find a precision of \sigma _ { \Delta z / ( 1 + z ) } = 0.008 at i ^ { + } < 22.5 and \sigma _ { \Delta z / ( 1 + z ) } = 0.03 at 1.5 < z < 4 . We derive the stellar mass function and correct for the Eddington bias . We find a mass-dependent evolution of the global and star-forming populations , with the low-mass end of the mass functions evolving more rapidly than the high-mass end . This mass-dependent evolution is a direct consequence of the star formation being “ quenched ” in galaxies more massive than { \cal M } \gtrsim 10 ^ { 10.7 - 10.9 } { \cal M } _ { \sun } . For the mass function of the quiescent galaxies , we do not find any significant evolution of the high-mass end at z < 1 ; however we observe a clear flattening of the faint-end slope . From z \sim 3 to z \sim 1 , the density of quiescent galaxies increases over the entire mass range . Their comoving stellar mass density increases by 1.6 dex between z \sim 3 and z \sim 1 and by less than 0.2 dex at z < 1 . We infer the star formation history from the mass density evolution . This inferred star formation history is in excellent agreement with instantaneous star formation rate measurements at z < 1.5 , while we find differences of 0.2 dex at z > 1.5 consistent with the expected uncertainties . We also develop a new method to infer the specific star formation rate from the mass function of star-forming galaxies . We find that the specific star formation rate of 10 ^ { 10 - 10.5 } { \cal M } _ { \sun } galaxies increases continuously in the redshift range 1 < z < 4 . Finally , we compare our results with a semi-analytical model and find that these models overestimate the density of low mass quiescent galaxies by an order of magnitude , while the density of low-mass star-forming galaxies is successfully reproduced .