Active Galactic Nuclei ( AGN ) produce a dominant fraction ( \sim 80 \% ) of the Soft X-ray background ( SXB ) at photon energies 0.5 < E < 2 keV . If dust pervaded throughout the intergalactic medium , its scattering opacity would have produced diffuse X-ray halos around AGN . Taking account of known galaxies and galaxy clusters , only a fraction F _ { halo } ~ { } \hbox to 0.0 pt { $ < $ } { \lower 4.3 pt \hbox { $ \sim$ } } 10 \% of the SXB can be in the form of diffuse X-ray halos around AGN . We therefore limit the intergalactic opacity to optical/infrared photons from large dust grains ( with radii in the range a = 0.2 - 2.0 \mu m ) to a level \tau _ { GD } ~ { } \hbox to 0.0 pt { $ < $ } { \lower 4.3 pt \hbox { $ \sim$ } } 0.15 ( F _ { halo% } / 10 \% ) to a redshift z \sim 1 . Our results are only weakly dependent on the grain size distribution or the redshift evolution of the intergalactic dust . Stacking X-ray images of AGN can be used to improve our constraints and diminish the importance of dust as a source of systematic uncertainty for future supernova surveys which aim to improve the precision on measuring the redshift evolution of the dark energy equation-of-state .