We present the evolution of the rest-frame optical luminosity density , j ^ { rest } _ { \lambda } , of the integrated rest-frame optical color , and of the stellar mass density , \rho _ { * } , for a sample of K _ { s } -band selected galaxies in the HDF-S. We derived j ^ { rest } _ { \lambda } in the rest-frame U , B , and V -bands and found that j ^ { rest } _ { \lambda } increases by a factor of 1.9 \pm 0.4 , 2.9 \pm 0.6 , and 4.9 \pm 1.0 in the V , B , and U rest-frame bands respectively between a redshift of 0.1 and 3.2 . We derived the luminosity weighted mean cosmic ( U~ { } - ~ { } B ) _ { rest } and ( B~ { } - ~ { } V ) _ { rest } colors as a function of redshift . The colors bluen almost monotonically with increasing redshift ; at z = 0.1 , the ( U~ { } - ~ { } B ) _ { rest } and ( B~ { } - ~ { } V ) _ { rest } colors are 0.16 and 0.75 respectively , while at z = 2.8 they are -0.39 and 0.29 respectively . We derived the luminosity weighted mean { M } / { L } ^ { * } _ { V } using the correlation between ( U~ { } - ~ { } V ) _ { rest } and log _ { 10 } { M } / { L } ^ { * } _ { V } which exists for a range in smooth SFHs and moderate extinctions . We have shown that the mean of individual { M } / { L } ^ { * } _ { V } estimates can overpredict the true value by \sim 70 \% while our method overpredicts the true values by only \sim 35 \% . We find that the universe at z \sim 3 had \sim 10 times lower stellar mass density than it does today in galaxies with { L } _ { V } ^ { rest } > 1.4 \times 10 ^ { 10 } { ~ { } h _ { 70 } ^ { -2 } ~ { } L _ { \odot } } . 50 % of the stellar mass of the universe was formed by z \sim 1 - 1.5 . The rate of increase in \rho _ { * } with decreasing redshift is similar to but above that for independent estimates from the HDF-N , but is slightly less than that predicted by the integral of the SFR ( z ) curve .