Context : Rapidly rotating neutron stars are an ideal laboratory to test models of matter at high densities . In particular the maximum rotation frequency of a neutron star is equation of state dependent , and can be used to test models of the interior . However observations of the spin distribution of rapidly rotating neutron stars show evidence for a lack of stars spinning at frequencies larger than f \approx 700 Hz , well below the predictions of theoretical equations of state . This has generally been taken as evidence of an additional spin-down torque operating in these systems and it has been suggested that gravitational wave torques may be operating and be linked to a potentially observable signal . Aims : In this paper we aim to determine whether additional spin-down torques ( possibly due to gravitational wave emission ) are necessary , or whether the observed limit of f \approx 700 Hz could correspond to the Keplerian ( mass-shedding ) break-up frequency for the observed systems and is simply a consequence of the , currently unknown , state of matter at high densities . Methods : Given our ignorance with regard to the true equation of state of matter above nuclear saturation densities , we make minimal physical assumption and only demand causality , i.e . that the speed of sound in the interior of the neutron star should be less or equal to the speed of light c . We then connect our causally-limited equation of state to a realistic microphysical crustal equation of state for densities below nuclear saturation density . This produces a limiting model that will give the lowest possible maximum frequency , which we compare to observational constraints on neutron star masses and frequencies . We also compare our findings with the constraints on the tidal deformability obtained in the observations of the GW170817 event . Results : We find that the lack of pulsars spinning faster than f \approx 700 Hz is not compatible with our causal limited ‘ minimal ’ equation of state , for which the breakup frequency can not be lower than f _ { max } \approx 1200 Hz . A low frequency cutoff , around f \approx 800 Hz could only be possible if we assume that these systems do not contain neutron stars with masses above M \approx 2 M _ { \odot } . This would have to be due either to selection effects , or possibly to a phase transition in the interior of the neutron star , that leads to softening at high densities and a collapse either to a black hole or a hybrid star above M \approx 2 M _ { \odot } . Such a scenario would , however , require a somewhat unrealistically stiff equation of state for hadronic matter . Conclusions :