We report a 4.8 \sigma measurement of the cross-correlation signal between the cosmic microwave background ( CMB ) lensing convergence reconstructed from measurements of the CMB polarization made by the Polarbear experiment and the infrared-selected galaxies of the Herschel -ATLAS survey . This is the first measurement of its kind . We infer a best-fit galaxy bias of b = 5.76 \pm 1.25 , corresponding to a host halo mass of \log _ { 10 } ( M _ { h } / M _ { \odot } ) = 13.5 ^ { +0.2 } _ { -0.3 } at an effective redshift of z \sim 2 from the cross-correlation power spectrum . Residual uncertainties in the redshift distribution of the sub-mm galaxies are subdominant with respect to the statistical precision . We perform a suite of systematic tests , finding that instrumental and astrophysical contaminations are small compared to the statistical error . This cross-correlation measurement only relies on CMB polarization information that , differently from CMB temperature maps , is less contaminated by galactic and extragalactic foregrounds , providing a clearer view of the projected matter distribution . This result demonstrates the feasibility and robustness of this approach for future high-sensitivity CMB polarization experiments .