We studied the optical counterpart of the intermediate-mass black hole candidate HLX-1 in ESO 243-49 . We used a set of Very Large Telescope imaging observations from 2010 November , integrated by Swift X-ray data from the same epoch . We measured standard Vega brightnesses U = 23.89 \pm 0.18 mag , B = 25.19 \pm 0.30 mag , V = 24.79 \pm 0.34 mag and R = 24.71 \pm 0.40 mag . Therefore , the source was \approx 1 mag fainter in each band than in a set of Hubble Space Telescope images taken a couple of months earlier , when the X-ray flux was a factor of 2 higher . We conclude that during the 2010 September observations , the optical counterpart was dominated by emission from an irradiated disk ( which responds to the varying X-ray luminosity ) , rather than by a star cluster around the black hole ( which would not change ) . We modelled the Comptonized , irradiated X-ray spectrum of the disk , and found that the optical luminosity and colours in the 2010 November data are still consistent with emission from the irradiated disk , with a characteristic outer radius r _ { out } \approx 2800 r _ { in } \sim 10 ^ { 13 } cm and a reprocessing fraction \approx 2 \times 10 ^ { -3 } . The optical colours are also consistent with a stellar population with age \la 6 Myr ( at solar metallicity ) and mass \approx 10 ^ { 4 } M _ { \odot } ; this is only an upper limit to the mass , if there is also a significant contribution from an irradiated disk . We strongly rule out the presence of a young super-star-cluster , which would be too bright . An old globular cluster might be associated with HLX-1 , as long as its mass \la 2 \times 10 ^ { 6 } M _ { \odot } for an age of 10 Gyr , but it can not significantly contribute to the observed very blue and variable optical/UV emission .