The edge of the M 87 halo and the kinematics of the diffuse light in the Virgo cluster core

Aims. We study the kinematics and dynamics of the extreme outer halo of M 87, the central galaxy in the Virgo cluster; and its transition to the intracluster light (ICL). Methods. We present high resolution FLAMES/VLT spectroscopy of intracluster planetary nebula (PN) candidates, targeting three new fields in the Virgo cluster core with surface brightness down to μ_B = 28.5. Based on the projected phase space information (sky positions and line-of-sight velocities) we separate galaxy and cluster components in the confirmed PN sample. We then use the spherical Jeans equation and the total gravitational potential as traced by the X-ray emission to derive the orbital distribution in the outer stellar halo of M 87. We determine the luminosity-specific PN number for the M 87 halo and the ICL from the photometric PN catalogs and sampled luminosities, and discuss the origin of the ICL in Virgo based on its measured PN velocities. Results. We confirm a further 12 PNs in Virgo, five of which are bound to the halo of M 87, and the remainder are true intracluster planetary nebulas (ICPNs). The M 87 PNs are confined to the extended stellar envelope of M 87, within a projected radius of ∼160 kpc, while the ICPNs are scattered across the whole surveyed region between M 87 and M 86, supporting a truncation of M 87's luminous outer halo at a 2σ level. The line-of-sight velocity distribution of the M 87 PNs at projected radii of 60 kpc and 144 kpc shows (i) no evidence for rotation of the halo along the photometric major axis; and (ii) that the velocity dispersion decreases in the outer halo, down to σ_last = 78 ± 25 km s^-1 at 144 kpc. The Jeans model for the M 87 halo stars fits the observed line-of-sight velocity dispersion profile only if the stellar orbits are strongly radially anisotropic (β ≃ 0.4 at r ≃ 10 kpc increasing to 0.8 at the outer edge), and if additionally the stellar halo is truncated at ≃ kpc average elliptical radius. The α-parameters for the M 87 halo and the ICL are in the range of values observed for old (>10 Gyr) stellar populations. Conclusions. Both the spatial segregation of the PNs at the systemic velocity of M 87 and the dynamical model support that the stellar halo of M 87 ends at ∼150 kpc. We discuss several possible explanations for the origin of this truncation but are unable to discriminate between them: tidal truncation following an earlier encounter of M 87 with another mass concentration in the Virgo core, possibly around M 84, early AGN feedback effects, and adiabatic contraction due to the cluster dark matter collapsing onto M 87. From the spatial and velocity distribution of the ICPNs we infer that M 87 and M 86 are falling towards each other and that we may be observing them just before the first close pass. The new PN data support the view that the core of the Virgo cluster is not yet virialized but is in an ongoing state of assembly, and that massive elliptical galaxies are important contributors to the ICL in the Virgo cluster.