Human authentication is critical to protect personal and property security. Existing contactless authentication methods face some drawbacks, such as requiring large data size and low accuracy in cross-domain recognition, which hinders widespread popularization in practical applications. In this paper, we design and implement WiLCA, a WiFi-based lightweight contactless authentication system. First, we devise a Channel State Information (CSI) stream selection scheme to extract human movement features and reduce the sample size in the recognition process. Then, an AGO model is proposed, in which a Siamese Neural Network (SNN) framework with a cross-entropy module is used to guarantee accurate human authentication with limited data, and a lightweight GhostNet accelerates authentication with cheap operations. At last, extensive experiments are conducted to demonstrate the advantages of WiLCA, revealing that compared with state-of-the-art methods, WiLCA can reduce the data size by at least 2.5 x and achieve accurate authentication with an accuracy of over 98%.