We present an analysis of the optical colors of 413 Virgo cluster early-type dwarf galaxies ( dEs ) , based on Sloan Digital Sky Survey imaging data . Our study comprises ( 1 ) a comparison of the color-magnitude relation ( CMR ) of the different dE subclasses that we identified in Paper III of this series , ( 2 ) a comparison of the shape of the CMR in low and high-density regions , ( 3 ) an analysis of the scatter of the CMR , and ( 4 ) an interpretation of the observed colors with ages and metallicities from population synthesis models . We find that the CMRs of nucleated ( dE ( N ) ) and non-nucleated dEs ( dE ( nN ) ) are significantly different from each other , with similar colors at fainter magnitudes ( m _ { \mathrm { r } } \gtrsim 17 mag ) , but increasingly redder colors of the dE ( N ) s at brighter magnitudes . We interpret this with older ages and/or higher metallicities of the brighter dE ( N ) s. The dEs with disk features have similar colors as the dE ( N ) s and seem to be only slightly younger and/or less metal-rich on average . Furthermore , we find a small but significant dependence of the CMR on local projected galaxy number density , consistently seen in all of u - r , g - r , and g - i , and weakly i - z . We deduce that a significant intrinsic color scatter of the CMR is present , even when allowing for a distance spread of our galaxies . No increase of the CMR scatter at fainter magnitudes is observed down to m _ { \mathrm { r } } \approx 17 mag ( M _ { \mathrm { r } } \approx - 14 mag ) . The color residuals , i.e. , the offsets of the data points from the linear fit to the CMR , are clearly correlated with each other in all colors for the dE ( N ) s and for the full dE sample , implying that , at a given magnitude , a galaxy with an older stellar population than average typically also exhibits a higher metallicity than average . Given the observational data for Virgo dEs presented here and in the previous papers of this series , we conclude that there must be at least two different formation channels for early-type dwarfs in order to explain the heterogeneity of this class of galaxy .