We investigate the stellar populations of a sample of 162 Ly \alpha emitting galaxies ( LAEs ) at z = 3.1 in the Extended Chandra Deep Field South , using deep Spitzer IRAC data available from the GOODS and SIMPLE surveys to derive reliable stellar population estimates . We divide the LAEs according to their rest-frame near-IR luminosities into IRAC-detected and IRAC-undetected samples . About 70 % of the LAEs are undetected in 3.6 µm down to m _ { 3.6 } = 25.2 AB . Stacking analysis reveals that the average stellar population of the IRAC-undetected sample has an age of \sim 200 Myr and a mass of \sim 3 \times 10 ^ { 8 } { M } _ { \sun } , consistent with the expectation that LAEs are mostly young and low-mass galaxies . On the other hand , the IRAC-detected LAEs are on average significantly older and more massive , with an average age \gtrsim 1 Gyr and mass \sim 10 ^ { 10 } { M } _ { \sun } . Comparing the IRAC colors and magnitudes of the LAEs to z \sim 3 Lyman break galaxies ( LBGs ) shows that the IRAC-detected LAEs lie at the faint blue end of the LBG color-magnitude distribution , suggesting that IRAC-detected LAEs may be the low mass extension of the LBG population . We also present tentative evidence for a small fraction ( \sim 5 \% ) of obscured AGN within the LAE sample . Our results suggest that LAEs posses a wide range of ages and masses . Additionally , the presence of evolved stellar populations inside LAEs suggests that the Ly \alpha luminous phase of galaxies may either be a long-lasting or recurring phenomenon .