We study the evolution of the size - stellar mass relation for a large spectroscopic sample of galaxies in the GOODs North field up to z \sim 3.5 . The sizes of the galaxies are measured from \textit { K } _ { s } -band images ( corresponding to rest-frame optical/NIR ) from the Subaru 8m telescope . We reproduce earlier results based on photometric redshifts that the sizes of galaxies at a given mass evolve with redshift . Specifically , we compare sizes of UV-bright galaxies at a range of redshifts : Lyman break galaxies ( LBGs ) selected through the U-drop technique ( z \sim 2.5 - 3.5 ) , BM/BX galaxies at z \sim 1.5 - 2.5 , and GALEX LBGs at low redshift ( z \sim 0.6 - 1.5 ) . The median sizes of these UV-bright galaxies evolve as ( 1 + z ) ^ { -1.11 \pm 0.13 } between z \sim 0.5 - 3.5 . The UV-bright galaxies are significantly larger than quiescent galaxies at the same mass and redshift by 0.45 \pm 0.09 dex . We also verify the correlation between color and stellar mass density of galaxies to high redshifts . The sizes of sub-mm galaxies in the same field are measured and compared with BM/BX galaxies . We find that median half-light radii of SMGs is 2.90 \pm 0.45 kpc and there is little difference in their size distribution to the UV-bright star forming galaxies .