TY - GEN
T1 - Be Real in Scale
T2 - 22nd IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2023
AU - Yu, Rui
AU - Wang, Jian
AU - Ma, Sizhuo
AU - Huang, Sharon X.
AU - Krishnan, Gurunandan
AU - Wu, Yicheng
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - Many mobile AR apps that use the front-facing camera can benefit significantly from knowing the metric scale of the user's face. However, the true scale of the face is hard to measure because monocular vision suffers from a fundamental ambiguity in scale. The methods based on prior knowledge about the scene either have a large error or are not easily accessible. In this paper, we propose a new method to measure the face scale by a simple user interaction: the user only needs to swing the phone to capture two selfies while using the recently popular Dual Camera mode. This mode allows simultaneous streaming of the front camera and the rear cameras and has become a key feature in many social apps. A computer vision method is applied to first estimate the absolute motion of the phone from the images captured by two rear cameras, and then calculate the point cloud of the face by triangulation. We develop a prototype mobile app to validate the proposed method. Our user study shows that the proposed method is favored compared to existing methods because of its high accuracy and ease of use. Our method can be built into Dual Camera mode and can enable a wide range of applications (e.g., virtual try-on for online shopping, true-scale 3D face modeling, gaze tracking, and face anti-spoofing) by introducing true scale to smartphone-based XR. The code is available at https://github.com/ruiyu0/Swing-for-True-Scale.
AB - Many mobile AR apps that use the front-facing camera can benefit significantly from knowing the metric scale of the user's face. However, the true scale of the face is hard to measure because monocular vision suffers from a fundamental ambiguity in scale. The methods based on prior knowledge about the scene either have a large error or are not easily accessible. In this paper, we propose a new method to measure the face scale by a simple user interaction: the user only needs to swing the phone to capture two selfies while using the recently popular Dual Camera mode. This mode allows simultaneous streaming of the front camera and the rear cameras and has become a key feature in many social apps. A computer vision method is applied to first estimate the absolute motion of the phone from the images captured by two rear cameras, and then calculate the point cloud of the face by triangulation. We develop a prototype mobile app to validate the proposed method. Our user study shows that the proposed method is favored compared to existing methods because of its high accuracy and ease of use. Our method can be built into Dual Camera mode and can enable a wide range of applications (e.g., virtual try-on for online shopping, true-scale 3D face modeling, gaze tracking, and face anti-spoofing) by introducing true scale to smartphone-based XR. The code is available at https://github.com/ruiyu0/Swing-for-True-Scale.
UR - http://www.scopus.com/inward/record.url?scp=85180374681&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85180374681&partnerID=8YFLogxK
U2 - 10.1109/ISMAR59233.2023.00140
DO - 10.1109/ISMAR59233.2023.00140
M3 - Conference contribution
AN - SCOPUS:85180374681
T3 - Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2023
SP - 1231
EP - 1239
BT - Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2023
A2 - Bruder, Gerd
A2 - Olivier, Anne-Helene
A2 - Cunningham, Andrew
A2 - Peng, Evan Yifan
A2 - Grubert, Jens
A2 - Williams, Ian
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 16 October 2023 through 20 October 2023
ER -