We present a new numerical study of the equilibrium and stability properties of close binary systems .
We use the smoothed-particle hydrodynamics ( SPH ) technique both to construct accurate equilibrium configurations in three dimensions and to follow their hydrodynamic evolution .
We adopt a simple polytropic equation of state p = K \rho ^ { \Gamma } with \Gamma = 5 / 3 and K = constant within each star , applicable to low-mass degenerate dwarfs as well as low-mass main-sequence stars .
For degenerate configurations , we set K = K ^ { \prime } independent of the mass ratio .
For main-sequence stars , we adjust K and K ^ { \prime } so as to obtain a simple mass-radius relation of the form R / R ^ { \prime } = M / M ^ { \prime } .
Along a sequence of binary equilibrium configurations for two identical stars , we demonstrate the existence of both secular and dynamical instabilities , confirming directly the results of recent analytic work .
We use the SPH method to calculate the nonlinear development of the dynamical instability and to determine the final fate of the system .
We find that the two stars merge together into a single , rapidly rotating object in just a few orbital periods .
Equilibrium sequences are also constructed for systems containing two nonidentical stars .
These sequences terminate at a Roche limit , which we can determine very accurately using SPH .
For two low-mass main-sequence stars with mass ratio q \lesssim 0.4 we find that the ( synchronized ) Roche limit configuration is secularly unstable .
For q \lesssim 0.25 , a dynamical instability is encountered before the Roche limit .
Degenerate binary configurations remain hydrodynamically stable all the way to the Roche limit for all mass ratios q \neq 1 .
However , unstable mass transfer can occur beyond the Roche limit , and this is indeed observed in our numerical simulations .
Dynamically unstable mass transfer also leads to the rapid coalescence of the binary system , although the details of the hydrodynamic evolution are quite different .
We discuss the implications of our results for the evolution of double white-dwarf systems and W Ursae Majoris binaries .