Starburst galaxies and star-forming active galactic nuclei ( AGN ) are among the candidate sources thought to contribute appreciably to the extragalactic gamma-ray and neutrino backgrounds . NGC 1068 is the brightest of the star-forming galaxies found to emit gamma rays from 0.1 to 50 GeV . Precise measurements of the high-energy spectrum are crucial to study the particle accelerators and probe the dominant emission mechanisms . We have carried out 125 hours of observations of NGC 1068 with the MAGIC telescopes in order to search for gamma-ray emission in the very high energy band . We did not detect significant gamma-ray emission , and set upper limits at 95 % confidence level to the gamma-ray flux above 200 GeV f < 5.1 \times 10 ^ { -13 } cm ^ { -2 } s ^ { -1 } . This limit improves previous constraints by about an order of magnitude and allows us to put tight constraints on the theoretical models for the gamma-ray emission . By combining the MAGIC observations with the Fermi -LAT spectrum we limit the parameter space ( spectral slope , maximum energy ) of the cosmic ray protons predicted by hadronuclear models for the gamma-ray emission , while we find that a model postulating leptonic emission from a semi-relativistic jet is fully consistent with the limits . We provide predictions for IceCube detection of the neutrino signal foreseen in the hadronic scenario . We predict a maximal IceCube neutrino event rate of 0.07 yr ^ { -1 } .