While the quiet Sun magnetic field shows only little variation with the solar cycle , long-term variations can not be completely ruled out from first principles . We investigate the potential effect of quiet Sun magnetism on spectral solar irradiance through a series of small-scale dynamo simulations with zero vertical flux imbalance ( \langle B _ { z } \rangle = 0 ) and varying levels of small-scale magnetic field strength , and one weak network case with an additional flux imbalance corresponding to a flux density of \langle B _ { z } \rangle = 100 G. From these setups we compute the dependence of the outgoing radiative energy flux on the mean vertical magnetic field strength in the photosphere at continuum optical depth \tau = 1 ( \langle|B _ { z } | \rangle _ { \tau = 1 } ) . We find that a quiet Sun setup with a mean vertical field strength of \langle|B _ { z } | \rangle _ { \tau = 1 } = 69 G is about 0.6 ~ { } \% brighter than a non-magnetic reference case . We find a linear dependence of the outgoing radiative energy flux on the mean field strength \langle|B _ { z } | \rangle _ { \tau = 1 } with a relative slope of 1.4 \cdot 10 ^ { -4 } G ^ { -1 } . With this sensitivity , only a moderate change of the quiet Sun field strength by 10 \% would lead to a total solar irradiance variation comparable to the observed solar cycle variation . While this does provide strong indirect constraints on possible quiet Sun variations during a regular solar cycle , it also emphasizes that potential variability over longer time scales could make a significant contribution to longer-term solar irradiance variations .