This paper presents numerical simulations of test particle Fermi acceleration at relativistic shocks of Lorentz factor \Gamma _ { sh } = 2 - 60 , using a realistic downstream magnetic structure obtained from the shock jump conditions . The upstream magnetic field is described as pure Kolmogorov turbulence ; the corresponding downstream magnetic field lies predominantly in the plane tangential to the shock surface and the coherence length is smaller along the shock normal than in the tangential plane . Acceleration is nonetheless efficient and leads to powerlaw spectra with index \simeq 2.6 - 2.7 at large shock Lorentz factor \Gamma _ { sh } \gg 1 , markedly steeper than for isotropic scattering downstream . The acceleration timescale t _ { acc } in the upstream rest frame becomes a fraction of Larmor time t _ { L } in the ultra-relativistic limit , t _ { acc } \approx 10 t _ { L } / \Gamma _ { sh } . Astrophysical applications are discussed , in particular the acceleration in \gamma - ray bursts internal and external shocks .