In the first of these two papers we demonstrated that assuming streams delineate orbits can lead to order one errors in potential parameters for realistic Galactic potentials . Motivated by the need for an improvement on orbit-fitting , we now present an algorithm for constraining the Galactic potential using tidal streams without assuming that streams delineate orbits . This approach is independent of the progenitor mass so is valid for all observed tidal streams . The method makes heavy use of angle-action variables and seeks the potential which recovers the expected correlations in angle space . We demonstrate that the method can correctly recover the parameters of a simple two-parameter logarithmic potential by analysing an N-body simulation of a stream . We investigate the magnitude of the errors in observational data for which the method can still recover the correct potential and compare this to current and future errors in data . The errors in the observables of individual stars for current and near future data are shown to be too large for the direct use of this method , but when the data are averaged in bins on the sky , the resulting averaged data are accurate enough to constrain correctly the potential parameters for achievable observational errors . From pseudo-data with errors comparable to those that will be furnished in the era of Gaia ( 20 \text { per cent } distance errors , 1.2 { mas yr ^ { -1 } } proper motion errors , and 10 { km s ^ { -1 } } line-of-sight velocity errors ) we recover the circular velocity , V _ { c } = 220 { km s ^ { -1 } } , and the flattening of the potential , q = 0.9 , to be V _ { c } = 223 \pm 10 { km s ^ { -1 } } and q = 0.91 \pm 0.09 .