We present a timing solution for the 598.89 Hz accreting millisecond pulsar , IGR J00291+5934 , using Rossi X-ray Timing Explorer data taken during the two outbursts exhibited by the source on 2008 August and September . We estimate the neutron star spin frequency and we refine the system orbital solution . To achieve the highest possible accuracy in the measurement of the spin frequency variation experienced by the source in-between the 2008 August outburst and the last outburst exhibited in 2004 , we re-analysed the latter considering the whole data set available . We find that the source spins down during quiescence at an average rate of \dot { \nu } _ { sd } = ( -4.1 \pm 1.2 ) \times 10 ^ { -15 } Hz s ^ { -1 } . We discuss possible scenarios that can account for the long-term neutron star spin-down in terms of either magneto-dipole emission , emission of gravitational waves , and a propeller effect . If interpreted in terms of magneto-dipole emission , the measured spin down translates into an upper limit to the neutron star magnetic field , B \lower 2.15 pt \hbox { $ \buildrel < \over { \sim } $ } 3 \times 10 ^ { 8 } G , while an upper limit to the average neutron star mass quadrupole moment of Q \lower 2.15 pt \hbox { $ \buildrel < \over { \sim } $ } 2 \times 10 ^ { 36 } g cm ^ { 2 } is set if the spin down is interpreted in terms of the emission of gravitational waves .