Ceph is an open-source, software-defined distributed storage system that turns a group of ordinary servers and their disks into a single, self-managing cluster that delivers block, object and file storage from the same pool of capacity. In server storage, Ceph is the typical choice when capacity must grow from tens of terabytes to many petabytes without a central controller, a proprietary array or a single point of failure.
Instead of relying on RAID controllers or a dual-controller SAN, Ceph spreads data and its redundancy across many independent nodes, calculates data placement with an algorithm rather than a lookup table, and repairs itself automatically when a disk or a whole server fails. It runs on commodity hardware and Linux, and it is widely used under virtualization platforms, private clouds and Kubernetes.
How Ceph stores and places data
Ceph is built on RADOS, the Reliable Autonomic Distributed Object Store. Every piece of data, whether it came from a virtual disk, an S3 upload or a file, is ultimately stored as objects in RADOS, usually of 4 MiB for block images. Objects are grouped into placement groups (PGs), and each placement group is mapped to a set of disks.
The mapping is computed by CRUSH (Controlled Replication Under Scalable Hashing). CRUSH takes the cluster map and a rule describing the failure domains, such as host, rack or room, and deterministically decides which disks hold each placement group. Because every client can run the same calculation, clients talk directly to the storage daemons without asking a metadata server where the data lives, which removes a central bottleneck.
A Ceph cluster consists of several daemon types, each with a distinct role. The following table summarizes them.
| Daemon | Role | Typical count |
|---|---|---|
| OSD (object storage daemon) | Stores data on one disk, replicates and recovers it | One per disk, from 3 to thousands |
| MON (monitor) | Maintains the cluster map and quorum | 3 or 5, always an odd number |
| MGR (manager) | Metrics, dashboard, orchestration modules | 2 (active and standby) |
| MDS (metadata server) | Directory tree and file metadata for CephFS | 1 or more active, plus standbys |
| RGW (RADOS Gateway) | S3- and Swift-compatible object API | 2 or more behind a load balancer |
Modern releases store data with BlueStore, which writes directly to raw block devices instead of going through a local file system. BlueStore keeps metadata in an embedded RocksDB database, computes checksums on every read, and supports inline compression. Placing the RocksDB and write-ahead log on a fast NVMe device in front of HDDs is a common way to improve small-write latency.
Block, object and file interfaces
The defining feature of Ceph is that one cluster serves three storage models. Each interface is a separate layer on top of RADOS, so capacity, replication and recovery are shared, while access semantics differ.
| Interface | Storage model | Typical consumers |
|---|---|---|
| RBD (RADOS Block Device) | Block storage, thin-provisioned images | Virtual machine disks, Kubernetes volumes |
| RGW (RADOS Gateway) | Object storage over HTTP | Backups, media, data lakes, applications using S3 |
| CephFS | POSIX-compliant shared file system | Home directories, shared application data |
| librados | Native object API | Custom applications needing direct access |
RBD images are thin-provisioned by default, support snapshots and clones, and can be mirrored to a second cluster. RGW adds multi-site replication, bucket policies and lifecycle rules. CephFS adds the MDS layer, which makes it the most demanding interface to operate; many organizations start with RBD and RGW and add CephFS only when a shared file system is truly required.
Replication, erasure coding and durability
Each Ceph pool chooses how data is protected. Replicated pools keep full copies, most often size 3 with min_size 2, meaning three copies are kept and writes continue as long as two are available. Erasure-coded pools split each object into k data chunks and m coding chunks, so a 4+2 profile tolerates two lost chunks while using only 1.5 times the raw space.
| Scheme | Raw overhead | Failures tolerated | Best suited for |
|---|---|---|---|
| Replication, size 3 | 3.0x | 2 copies | VM disks, databases, metadata |
| Replication, size 2 | 2.0x | 1 copy | Rarely advisable, high risk |
| Erasure coding 4+2 | 1.5x | 2 chunks | Object storage, backups, archives |
| Erasure coding 8+3 | about 1.4x | 3 chunks | Large clusters with many hosts |
Erasure coding saves capacity but costs CPU, latency and network traffic, and it needs at least k+m failure domains; a 4+2 profile with host as the failure domain requires six or more hosts. Every OSD also runs scrubbing, a background comparison of copies or chunks, and deep scrubbing that verifies checksums, which detects silent corruption before it spreads.
How Ceph compares with RAID, ZFS and a SAN
Ceph solves a different problem from local redundancy. RAID and ZFS protect data inside one server, while Ceph protects data across servers, so it survives the loss of a whole node. The trade-off is that a single Ceph write travels over the network to several machines, so latency per operation is higher than on a local NVMe mirror.
| Approach | Scope of redundancy | Scaling model | Main limitation |
|---|---|---|---|
| Ceph | Across hosts, racks or rooms | Scale-out, add nodes | Operational complexity, network-bound latency |
| ZFS | Disks within one server | Scale-up, add disks | Server is a single point of failure |
| Hardware RAID | Disks behind one controller | Scale-up | Controller and rebuild risk |
| SAN array | Dual controllers in one chassis | Scale-up, then new array | Cost and vendor lock-in |
| NAS appliance | Within the appliance | Scale-up or limited clustering | File access only |
Ceph exposes block devices natively through RBD, but it can also be presented to legacy hosts through iSCSI or NVMe-oF gateways, and CephFS can be re-exported over NFS or SMB. These gateways add a hop and are best treated as compatibility layers rather than the primary access path.
Designing and sizing a Ceph cluster
A production cluster normally starts at three nodes for a minimum and works far better with five or more, because recovery after a failure needs somewhere to place the missing copies. Key planning decisions include:
- Network: at least 10 GbE, commonly 25 GbE or faster, often with a separate cluster network for replication and recovery traffic.
- Memory: BlueStore targets about 4 GiB of RAM per OSD by default, plus headroom for the operating system.
- Media: all-flash for IOPS-heavy block workloads; HDD with NVMe for the database and log for capacity-oriented object storage.
- Capacity headroom: keep usage well below the 85 percent near-full threshold so the cluster can heal after losing a node.
- Placement groups: let the PG autoscaler manage counts rather than setting them by hand.
Most new deployments use cephadm, which runs daemons in containers and handles upgrades, or Rook, which operates Ceph inside Kubernetes. Proxmox VE also integrates Ceph directly for hyperconverged virtualization clusters.
Pitfalls and operational risks
Ceph is reliable when it is designed correctly, but it punishes undersized clusters. The most frequent problems come from the mismatch between expectations of a simple disk array and the reality of a distributed system.
| Mistake | Consequence | Better practice |
|---|---|---|
| Running with size 2, min_size 1 | Data loss during overlapping failures | Use size 3, min_size 2 |
| Filling the cluster above 80 percent | Recovery stalls, writes block at full ratio | Plan capacity for one node lost |
| Slow or shared network | High latency, slow recovery | Dedicated 25 GbE or faster links |
| Consumer SSDs without power-loss protection | Very poor sync-write performance | Enterprise SSDs with PLP |
| Even number of monitors | No gain in fault tolerance | Run 3 or 5 monitors |
Recovery traffic after a failure also competes with client I/O, so operators tune recovery and backfill limits to balance healing speed against application performance. Finally, Ceph is not a backup: replication faithfully copies deletions and corruption caused by software, so separate backups remain necessary.
Stripped to the essentials
Imagine a library that keeps three copies of every book in three different buildings. A computer program knows exactly which buildings hold each book without checking a catalog, so readers go straight to the right shelf. If one building closes, the library quietly makes new copies elsewhere until every book again exists three times.
Ceph does the same with data. It takes many ordinary servers, treats all of their disks as one large storage pool, and keeps data safe even when a disk or an entire server breaks. The key idea is simple: more servers mean more space, more speed and more safety, with no single machine in charge.
Where it is applied
A regional hosting provider runs about 400 virtual machines on a five-node Proxmox VE cluster. Each node holds six enterprise NVMe drives, and the nodes are connected by redundant 25 GbE links. Ceph runs on the same servers, giving hyperconverged storage with an RBD pool at size 3, min_size 2 for virtual machine disks and an erasure-coded 2+2 RGW pool for customer backups.
When one node suffers a motherboard failure, the virtual machines restart on the remaining four nodes, and their disks remain available because every block still has two healthy copies. Ceph starts rebuilding the missing copies across the surviving NVMe drives, which completes in under an hour. Because the team kept utilization near 60 percent, the cluster had enough free space to heal fully, and customers saw only a brief restart rather than any data loss.