We study the structural evolution of massive galaxies by linking progenitors and descendants at a constant cumulative number density of n _ { c } = 1.4 \times 10 ^ { -4 } Mpc ^ { -3 } to z \sim 3 . Structural parameters were measured by fitting Sérsic profiles to high-resolution CANDELS HST WFC3 J _ { 125 } and H _ { 160 } imaging in the UKIDSS-UDS at 1 < z < 3 and ACS I _ { 814 } imaging in COSMOS at 0.25 < z < 1 . At a given redshift , we selected the HST band that most closely samples a common rest-frame wavelength so as to minimize systematics from color gradients in galaxies . At fixed n _ { c } , galaxies grow in stellar mass by a factor of \sim 3 from z \sim 3 to z \sim 0 . The size evolution is complex : galaxies appear roughly constant in size from z \sim 3 to z \sim 2 and then grow rapidly to lower redshifts . The evolution in the surface mass density profiles indicates that most of the mass at r < 2 kpc was in place by z \sim 2 , and that most of the new mass growth occurred at larger radii . This inside-out mass growth is therefore responsible for the larger sizes and higher Sérsic indices of the descendants toward low redshift . At z < 2 , the effective radius evolves with the stellar mass as r _ { e } \propto M ^ { 2.0 } , consistent with scenarios that find dissipationless minor mergers to be a key driver of size evolution . The progenitors at z \sim 3 were likely star forming disks with r _ { e } \sim 2 kpc , based on their low Sérsic index of n \sim 1 , low median axis ratio of b / a \sim 0.52 , and typical location in the star-forming region of the U - V versus V - J diagram . By z \sim 1.5 , many of these star-forming disks disappeared , giving rise to compact quiescent galaxies . Toward lower redshifts , these galaxies continued to assemble mass at larger radii and became the local ellipticals that dominate the high-mass end of the mass function at the present epoch .