The third-party enterprises, such as sociologists and commercial companies, are mining data published from online social network (OSN) websites (e.g., Facebook, Twitter) to serve their diverse purposes. This process leads to critical user concerns over their privacy, especially sensitive relationship with others on OSNs. Existing anonymization techniques in publishing online social data are focused on user identities, as users'relationship privacy will be automatically protected in general, if their identities are hidden. However, in reality, some users can still be identified from an identity-anonymized OSN by an attacker, as an individual user may publish his personal information to the public, through blog for example, which can be exploited by the attacker to re-identify the user from the published data. Therefore, we intend to preserve relationship privacy between two users one of whom can even be identified in the released OSN data. We define the l-diversity anonymization model to preserve users'relationship privacy. Additionally, we devise two algorithms to achieve the l-diversity anonymization - one only removes edges while the other only inserts vertices/edges for maintaining as many topological properties of the original social networks as possible, thus retaining the utility of the published data for the third-parties. Extensive experiments are conducted on both synthetic and real-world social network data sets to demonstrate that except from the achievement of privacy preservation, the utility loss caused by our proposed graphmanipulation based techniques is acceptable. Besides, we analyze the influence of social network topology (e.g., average degree, network scalability) on the performance of our algorithms.