Optical spectroscopy has enabled us to identify the optical counterparts to over 200 faint X-ray sources to a flux limit of S _ { ( 0.5 - 2 keV ) } = 4 \times 10 ^ { -15 } erg s ^ { -1 } cm ^ { -2 } on 5 deep \em ROSAT fields . Here we present a spectral analysis of all the X-ray sources to investigate claims that the average source spectra harden at faint X-ray flux . From a hardness ratio analysis we confirm that the average spectra from 0.5 - 2 keV harden from an equivalent photon index of \Gamma = 2.2 at S _ { ( 0.5 - 2 keV ) } = 1 \times 10 ^ { -13 } erg s ^ { -1 } cm ^ { -2 } to \Gamma \simeq 1.7 below 1 \times 10 ^ { -14 } erg s ^ { -1 } cm ^ { -2 } . These spectral changes are due to the emergence of an unidentified source population rather than the class of X-ray QSOs already identified . The 128 QSOs detected so far show no evidence for spectral hardening over this energy range and retain a mean photon index of \Gamma = 2.2 . Recent work suggests that many of the remaining unidentified sources are X-ray luminous galaxies . Taking a subset identified as the most likely galaxy candidates we find that these show significantly harder spectra than QSOs . The emission line galaxies in particular show spectra more consistent with the residual X-ray background , with \Gamma = 1.51 \pm 0.1 from 0.1 - 2 keV . Individually the galaxies appear to be a mixture of absorbed and unabsorbed X-ray sources . Combined with recent cross-correlation results and work on the source number count distribution , these results suggest that we may be uncovering the missing hard component of the cosmic X-ray background .