We use N-body/photometric galaxy simulations to examine the impact of sample variance of spectroscopic redshift samples on the accuracy of photometric redshift ( photo-z ) determination and calibration of photo-z errors . We estimate the biases in the cosmological parameter constraints from weak lensing and derive requirements on the spectroscopic follow-up for three different photo-z algorithms chosen to broadly span the range of algorithms available . We find that sample variance is much more relevant for the photo-z error calibration than for photo-z training , implying that follow-up requirements are similar for different algorithms . We demonstrate that the spectroscopic sample can be used for training of photo-zs and error calibration without incurring additional bias in the cosmological parameters . We provide a guide for observing proposals for the spectroscopic follow-up to ensure that redshift calibration biases do not dominate the cosmological parameter error budget . For example , assuming optimistically ( pessimistically ) that the weak lensing shear measurements from the Dark Energy Survey could obtain 1 - \sigma constraints on the dark energy equation of state w of 0.035 ( 0.055 ) , implies a follow-up requirement of 150 ( 40 ) patches of sky with a telescope such as Magellan , assuming a 1 / 8 \textrm { deg } ^ { 2 } effective field of view and 400 galaxies per patch . Assuming ( optimistically ) a VVDS-like spectroscopic completeness with purely random failures , this could be accomplished with about 75 ( 20 ) nights of observation . For more realistic assumptions regarding spectroscopic completeness , or with the presence of other sources of systematics not considered here , further degradations to dark energy constraints are possible . We test several approaches for making the requirements less stringent . For example , if the redshift distribution of the overall sample can be estimated by some other technique , e.g . cross-correlation , then follow-up requirements could be reduced by an order of magnitude .