The model of holographic dark energy ( HDE ) with massive neutrinos and/or dark radiation is investigated in detail . The background and perturbation evolutions in the HDE model are calculated . We employ the PPF approach to overcome the gravity instability difficulty ( perturbation divergence of dark energy ) led by the equation-of-state parameter w evolving across the phantom divide w = -1 in the HDE model with c < 1 . We thus derive the evolutions of density perturbations of various components and metric fluctuations in the HDE model . The impacts of massive neutrino and dark radiation on the CMB anisotropy power spectrum and the matter power spectrum in the HDE scenario are discussed . Furthermore , we constrain the models of HDE with massive neutrinos and/or dark radiation by using the latest measurements of expansion history and growth of structure , including the Planck CMB temperature data , the baryon acoustic oscillation data , the JLA supernova data , the Hubble constant direct measurement , the cosmic shear data of weak lensing , the Planck CMB lensing data , and the redshift space distortions data . We find that \sum m _ { \nu } < 0.186 eV ( 95 % CL ) and N _ { eff } = 3.75 ^ { +0.28 } _ { -0.32 } in the HDE model from the constraints of these data .