TY - JOUR
T1 - Optimal transport, mean partition, and uncertainty assessment in cluster analysis
AU - Li, Jia
AU - Seo, Beomseok
AU - Lin, Lin
N1 - Publisher Copyright:
© 2019 Wiley Periodicals, Inc.
PY - 2019/10/1
Y1 - 2019/10/1
N2 - In scientific data analysis, clusters identified computationally often substantiate existing hypotheses or motivate new ones. Yet the combinatorial nature of the clustering result, which is a partition rather than a set of parameters or a function, blurs notions of mean, and variance. This intrinsic difficulty hinders the development of methods to improve clustering by aggregation or to assess the uncertainty of clusters generated. We overcome that barrier by aligning clusters via optimal transport. Equipped with this technique, we propose a new algorithm to enhance clustering by any baseline method using bootstrap samples. Cluster alignment enables us to quantify variation in the clustering result at the levels of both overall partitions and individual clusters. Set relationships between clusters such as one-to-one match, split, and merge can be revealed. A covering point set for each cluster, a concept kin to the confidence interval, is proposed. The tools we have developed here will help address the crucial question of whether any cluster is an intrinsic or spurious pattern. Experimental results on both simulated and real data sets are provided. The corresponding R package OTclust is available on CRAN.
AB - In scientific data analysis, clusters identified computationally often substantiate existing hypotheses or motivate new ones. Yet the combinatorial nature of the clustering result, which is a partition rather than a set of parameters or a function, blurs notions of mean, and variance. This intrinsic difficulty hinders the development of methods to improve clustering by aggregation or to assess the uncertainty of clusters generated. We overcome that barrier by aligning clusters via optimal transport. Equipped with this technique, we propose a new algorithm to enhance clustering by any baseline method using bootstrap samples. Cluster alignment enables us to quantify variation in the clustering result at the levels of both overall partitions and individual clusters. Set relationships between clusters such as one-to-one match, split, and merge can be revealed. A covering point set for each cluster, a concept kin to the confidence interval, is proposed. The tools we have developed here will help address the crucial question of whether any cluster is an intrinsic or spurious pattern. Experimental results on both simulated and real data sets are provided. The corresponding R package OTclust is available on CRAN.
UR - http://www.scopus.com/inward/record.url?scp=85065815225&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85065815225&partnerID=8YFLogxK
U2 - 10.1002/sam.11418
DO - 10.1002/sam.11418
M3 - Article
AN - SCOPUS:85065815225
SN - 1932-1864
VL - 12
SP - 359
EP - 377
JO - Statistical Analysis and Data Mining
JF - Statistical Analysis and Data Mining
IS - 5
ER -