File systems unfit as distributed storage backends: Lessons from 10 years of Ceph evolution

Abutalib Aghayev, Sage Weil, Michael Kuchnik, Mark Nelson, Gregory R. Ganger, George Amvrosiadis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

65 Scopus citations

Abstract

For a decade, the Ceph distributed file system followed the conventional wisdom of building its storage backend on top of local file systems. This is a preferred choice for most distributed file systems today because it allows them to benefit from the convenience and maturity of battle-tested code. Ceph’s experience, however, shows that this comes at a high price. First, developing a zero-overhead transaction mechanism is challenging. Second, metadata performance at the local level can significantly affect performance at the distributed level. Third, supporting emerging storage hardware is painstakingly slow. Ceph addressed these issues with BlueStore, a new backend designed to run directly on raw storage devices. In only two years since its inception, BlueStore outperformed previous established backends and is adopted by 70% of users in production. By running in user space and fully controlling the I/O stack, it has enabled space-efficient metadata and data checksums, fast overwrites of erasure-coded data, inline compression, decreased performance variability, and avoided a series of performance pitfalls of local file systems. Finally, it makes the adoption of backwards-incompatible storage hardware possible, an important trait in a changing storage landscape that is learning to embrace hardware diversity.

Original languageEnglish (US)
Title of host publicationSOSP 2019 - Proceedings of the 27th ACM Symposium on Operating Systems Principles
PublisherAssociation for Computing Machinery, Inc
Pages353-369
Number of pages17
ISBN (Electronic)9781450368735
DOIs
StatePublished - Oct 27 2019
Event27th ACM Symposium on Operating Systems Principles, SOSP 2019 - Huntsville, Canada
Duration: Oct 27 2019Oct 30 2019

Publication series

NameSOSP 2019 - Proceedings of the 27th ACM Symposium on Operating Systems Principles

Conference

Conference27th ACM Symposium on Operating Systems Principles, SOSP 2019
Country/TerritoryCanada
CityHuntsville
Period10/27/1910/30/19

All Science Journal Classification (ASJC) codes

  • Software
  • Computational Theory and Mathematics
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'File systems unfit as distributed storage backends: Lessons from 10 years of Ceph evolution'. Together they form a unique fingerprint.

Cite this