Recognizing and classifying users' routine behavior patterns from sensor data has been a hot topic in pervasive computing. Its objective is to automatically discover recurrent routine patterns in a user's daily life by leveraging the multimodal data generated from wearable sensors such as mobile phones. This kind of knowledge can be utilized in many ways such as identifying similar users in terms of their behaviors, providing behavior contexts to enable advanced human-centered applications, etc. While numerous works have been done in this area, most of them rely on densely sampled mobile data collected from specially-programmed sensors that can "follow" people throughout the day. In this paper, we study how to achieve the same objective when the mobile data presented is much sparser, such as traditional mobile phone data where a user's location is reported only when he makes a call. Although a single user's mobile data is far from sufficient to reveal his characteristic behavior, we show that when exploiting a large number of users' mobile data in a principled collaborative way which facilitate similar users' data to complement each other, representative routine patterns can be revealed and each user can be characterized properly. Experiments on synthetic and real mobile phone data set demonstrate the effectiveness of our methods, and also show our model's ability in predicting human activity using the patterns learned.