Context : Aims : We study the spectral energy distribution ( SED ) of the Crab Pulsar and its nearby knot in the optical and in the infrared ( IR ) regime . We want to investigate how the contribution from the knot affects the pulsar SED in that regime , and examine the evidence for synchrotron self-absorption in the IR . We also draw the attention to the predicted secular decrease in luminosity of the Crab Pulsar , and attempt to investigate this with CCD observations . Methods : We present high-quality UBVRIz , as well as adaptive optics JHK _ { s } L ^ { \prime } photometry , achieved under excellent conditions with the FORS1 and NAOS/CONICA instruments at the VLT . We combine these data with re-analyzed archival Spitzer Space Telescope data to construct a SED for the pulsar , and quantify the contamination from the knot . We have also gathered optical imaging data from 1988 to 2008 from several telescopes in order to examine the predicted secular decrease in luminosity . Results : For the Crab Pulsar SED we find a spectral slope of \alpha _ { \nu } = 0.27 \pm 0.03 in the optical/near-IR regime , when we exclude the contribution from the knot . For the knot itself , we find a much redder slope of \alpha _ { \nu } = -1.3 \pm 0.1 . Our best estimate of the average decrease in luminosity for the pulsar is 2.9 \pm 1.6 mmag per year . Conclusions : We have demonstrated the importance of the nearby knot in precision measurements of the Crab Pulsar SED , in particular in the near-IR . We have scrutinized the evidence for the traditional view of a synchrotron self-absorption roll-over in the infrared , and find that these claims are unfounded . We also find evidence for a secular decrease in the optical light for the Crab Pulsar , in agreement with current pulsar spin-down models . However , although our measurements of the decrease significantly improve on previous investigations , the detection is still tentative . We finally point to future observations that can improve the situation significantly .