Chandra data in the COSMOS , AEGIS-XD and 4 Ms Chandra Deep Field South are combined with multiwavelength photometry available in those fields to determine the rest-frame U - V vs V - J colours of X-ray AGN hosts in the redshift intervals 0.1 < z < 0.6 ( mean \overline { z } = 0.40 ) and 0.6 < z < 1.2 ( mean \overline { z } = 0.85 ) . This combination of colours provides an effective and least model-dependent means of separating quiescent from star-forming , including dust reddened , galaxies . Morphological information emphasises differences between AGN populations split by their U - V vs V - J colours . AGN in quiescent galaxies consist almost exclusively of bulges , while star-forming hosts are equally split between early and late-type hosts . The position of AGN hosts on the U - V vs V - J diagram is then used to set limits on the accretion density of the Universe associated with evolved and star-forming systems independent of dust induced biases . It is found that most of the black hole growth at z \approx 0.40 and 0.85 is associated with star-forming hosts . Nevertheless , a non-negligible fraction of the X-ray luminosity density , about 15-20 % , at both \overline { z } = 0.40 and 0.85 , is taking place in galaxies in the quiescent region of the U - V vs V - J diagram . For the low-redshift subsample , 0.1 < z < 0.6 , we also find tentative evidence , significant at the 2 \sigma level , that AGN split by their U - V and V - J colours have different Eddington ratio distributions . AGN in blue star-forming hosts dominate at relatively high Eddington ratios . In contrast , AGN in red quiescent hosts become increasingly important as a fraction of the total population toward low Eddington ratios . At higher redshift , z > 0.6 , such differences are significant at the 2 \sigma level only for sources with Eddington ratios \ga 10 ^ { -3 } . These findings are consistent with scenarios in which diverse accretion modes are responsible for the build-up of supermassive black holes at the centres of galaxies . We compare these results with the predictions of the galform semi-analytic model for the cosmological evolution of AGN and galaxies . This model postulates two black hole fuelling modes , the first is linked to star-formation events and the second takes place in passive galaxies . galform predicts that a substantial fraction of the black hole growth at z < 1 is associated with quiescent galaxies , in apparent conflict with the observations . Relaxing the strong assumption of the model that passive AGN hosts have zero star-formation rate could bring those predictions in better agreement with the data .