We report on our search for the optical/infrared counterparts to the central compact objects in four young supernova remnants : Pup A , PKS 1209 - 52 , RCW 103 , and Cas A . The X-ray point sources in these supernova remnants are excellent targets for probing the existence of supernova fallback disks , since irradiation of a disk by a central X-ray source should lead to an infrared excess . We used ground-based optical and near-infrared imaging and Spitzer Space Telescope mid-infrared imaging to search for optical/infrared counterparts at the X-ray point source positions measured by the Chandra X-Ray Observatory . We did not detect any counterparts , and hence find no evidence for fallback disks around any of these sources . In PKS 1209 - 52 , we are able to exclude a nearby optical/infrared candidate counterpart . In RCW 103 , a blend of 3 faint stars at the X-ray source position prevents us from deriving useful limits . For the other targets , the upper limits on the infrared/X-ray flux ratio are as deep as ( 1.0–1.7 ) \times 10 ^ { -4 } . Comparing these limits to the ratio of \approx 6 \times 10 ^ { -5 } measured for 4U 0142+61 ( a young pulsar recently found with an X-ray irradiated dust disk ) , we conclude that the non-detection of any disks around young neutron stars studied here are consistent with their relatively low X-ray luminosities , although we note that a similar dust disk around the neutron star in Pup A should be detectable by deeper infrared observations .