We study the observational constraints on the growth of massive black holes ( BHs ) in galactic nuclei . We use the velocity dispersions of early-type galaxies obtained by the Sloan Digital Sky Survey and the relation between BH mass and velocity dispersion to estimate the local BH mass density to be \rho _ { \bullet } ( z = 0 ) \simeq ( 2.5 \pm 0.4 ) \times 10 ^ { 5 } h _ { 0.65 } ^ { 2 } { M _ { \odot } } % { Mpc } ^ { -3 } . We also use the QSO luminosity function from the 2dF Redshift Survey to estimate the BH mass density accreted during optically bright QSO phases . The local BH mass density is consistent with the density accreted during optically bright QSO phases if QSOs have a mass-to-energy conversion efficiency \epsilon \simeq 0.1 . By studying the continuity equation for the BH mass distribution , including the effect of BH mergers , we find relations between the local BH mass function and the QSO luminosity function . If the BH mass is assumed to be conserved during BH mergers , comparison of the predicted relations with the observations suggests that luminous QSOs ( L _ { bol } \ga 10 ^ { 46 } { erg s ^ { -1 } } ) have a high efficiency ( e.g . \epsilon \sim 0.2 , which is possible for thin-disk accretion onto a Kerr BH ) and the growth of high-mass BHs ( \ga 10 ^ { 8 } { M _ { \odot } } ) comes mainly from accretion during optically bright QSO phases , or that luminous QSOs have a super-Eddington luminosity . If luminous QSOs are not accreting with super-Eddington luminosities and the growth of low-mass BHs also occurs mainly during optically bright QSO phases , less luminous QSOs must accrete with a low efficiency < 0.1 ; alternatively , they may accrete with high efficiency , but a significant fraction should be obscured . We estimate that the mean lifetime of luminous QSOs ( L _ { bol } \ga 10 ^ { 46 } { erg s ^ { -1 } } ) is ( 3–13 ) \times 10 ^ { 7 } { yr } , which is comparable to the Salpeter time . We also investigate the case in which total BH mass decreases during BH mergers due to gravitational radiation ; in the extreme case in which total BH entropy is conserved , the observations again suggest that BHs in most luminous QSOs are Kerr BHs accreting with an efficiency \ga 0.1 .