We constrain the evolution of the galaxy stellar mass function from 2 < z < 5 for galaxies with stellar masses as low as 10 ^ { 5 } M _ { \odot } by combining star formation histories of Milky Way satellite galaxies derived from deep Hubble Space Telescope observations with merger trees from the ELVIS suite of N-body simulations . This approach extends our understanding more than two orders of magnitude lower in stellar mass than is currently possible by direct imaging . We find the faint end slopes of the mass functions to be \alpha = − 1.42 ^ { +0.07 } _ { -0.05 } at z = 2 and \alpha = − 1.57 ^ { +0.06 } _ { -0.06 } at z = 5 , and show the slope only weakly evolves from z = 5 to z = 0 . Our findings are in stark contrast to a number of direct detection studies that suggest slopes as steep as \alpha = -1.9 at these epochs . Such a steep slope would result in an order of magnitude too many luminous Milky Way satellites in a mass regime that is observationally complete ( M _ { \star } > 2 \times 10 ^ { 5 } \mbox { M$ { } _ { \odot } $ } at z = 0 ) . The most recent studies from ZFOURGE and CANDELS also suggest flatter faint end slopes that are consistent with our results , but with a lower degree of precision . This work illustrates the strong connections between low and high- z observations when viewed through the lens of \Lambda CDM numerical simulations .