Spiral antiferromagnets beyond the spin-wave approximation: Frustrated X y and Heisenberg models on the honeycomb lattice

We examine the stability of classical states with a generic incommensurate spiral order against quantum fluctuations. Specifically, we focus on the frustrated spin-1/2 XY and Heisenberg models on the honeycomb lattice with nearest-neighbor J1 and next-nearest-neighbor J2 antiferromagnetic couplings. Our variational approach is based on the Jastrow wave functions, which include quantum correlations on top of classical spin waves. We perform a systematic optimization of wave vectors and Jastrow pseudopotentials within this class of variational states and find that quantum fluctuations favor collinear states over generic coplanar spirals. The Néel state with Q=(0,0) extends its stability well beyond the classical value J2/J1=1/6. Most importantly, the collinear states with Q=(0,2π/3) (and the two symmetry-related states) are found to be stable in a large regime with intermediate frustration, while at the classical level they are limited to the point J2/J1=0.5. For large frustration, the 120â̂̃ state is stabilized for finite values of J2/J1 in both models.