We present multi-band optical and near-infrared observations of 19 short \gamma -ray burst ( GRB ) host galaxies , aimed at measuring their stellar masses and population ages . The goals of this study are to evaluate whether short GRBs track the stellar mass distribution of galaxies , to investigate the progenitor delay time distribution , and to explore any connection between long and short GRB progenitors . Using single stellar population models we infer masses of { log } ( M _ { * } / M _ { \odot } ) \approx 8.8 - 11.6 , with a median of \langle { log } ( M _ { * } / M _ { \odot } ) \rangle \approx 10.1 , and population ages of \tau _ { * } \approx 0.03 - 4.4 Gyr with a median of \langle \tau _ { * } \rangle \approx 0.3 Gyr . We further infer maximal masses of { log } ( M _ { * } / M _ { \odot } ) \approx 9.7 - 11.9 by assuming stellar population ages equal to the age of the universe at each host ’ s redshift . Comparing the distribution of stellar masses to the general galaxy mass function we find that short GRBs track the cosmic stellar mass distribution only if the late-type hosts generally have maximal masses . However , there is an apparent dearth of early-type hosts compared to the equal contribution of early- and late-type galaxies to the cosmic stellar mass budget . These results suggest that stellar mass may not be the sole parameter controlling the short GRB rate , and raise the possibility of a two-component model with both mass and star formation playing a role ( reminiscent of the case for Type Ia supernovae ) . If short GRBs in late-type galaxies indeed track the star formation activity , the resulting typical delay time is \sim 0.2 Gyr , while those in early-type hosts have a typical delay of \sim 3 Gyr . Using the same stellar population models we fit the broad-band photometry for 22 long GRB host galaxies in a similar redshift range and find that they have significantly lower masses and younger population ages , with \langle { log } ( M _ { * } / M _ { \odot } ) \rangle \approx 9.1 and \langle \tau _ { * } \rangle \approx 0.06 Gyr , respectively ; their maximal masses are similarly lower , \langle { log } ( M _ { * } / M _ { \odot } ) \rangle \approx 9.6 , and as expected do not track the galaxy mass function . Most importantly , the two GRB host populations remain distinct even if we consider only the star-forming hosts of short GRBs , supporting our previous findings ( based on star formation rates and metallicities ) that the progenitors of long GRBs and short GRBs in late-type galaxies are distinct . Given the much younger stellar populations of long GRB hosts ( and hence of long GRB progenitors ) , and the substantial differences in host properties , we caution against the use of Type I and II designations for GRBs since this may erroneously imply that all GRBs which track star formation activity share the same massive star progenitors .