Understanding Ceph Placement Groups
Placement groups are the fundamental unit of data distribution in a Ceph cluster. When you write an object, Ceph hashes the object name and maps it to a placement group. That placement group is then mapped to a set of OSDs determined by the CRUSH map. This two-level indirection — objects to PGs, PGs to OSDs — is what allows Ceph to redistribute data efficiently when OSDs are added or removed without rebalancing individual objects.
Every pool in a Ceph cluster has a fixed pg_num that determines how many placement groups exist for that pool, and a pgp_num that controls how many of those PGs are actually used for placement. In most cases pg_num and pgp_num should be equal. Setting pgp_num lower than pg_num temporarily pauses data migration while you prepare for the change — a useful technique when expanding a cluster in stages.
Why PG count matters
Too few PGs
Data distributes unevenly across OSDs. Some OSDs become hotspots while others sit underutilized. Recovery after an OSD failure concentrates load on fewer OSDs, slowing recovery time. Below 30 PGs per OSD, performance degrades noticeably under write load.
Too many PGs
Each PG carries memory overhead on every OSD that hosts it — approximately 10MB per PG per OSD in older releases, improving in Quincy and later. A cluster with 50 OSDs and 10,000 PGs consumes hundreds of megabytes of RAM just for PG metadata. Recovery time also increases with high PG counts.
The target range
The Ceph documentation recommends 100–200 PGs per OSD as a starting target, with the total across all pools divided appropriately. This calculator targets approximately 100 PGs per OSD and then snaps up to the next power of two, which is required for the CRUSH algorithm to produce even distribution.
Power of two requirement
Ceph requires pg_num to be a power of two (32, 64, 128, 256...) for the CRUSH algorithm to map PGs to OSDs with mathematical evenness. Non-power-of-two values cause uneven distribution that worsens as the cluster scales. This calculator always rounds up to the next power of two.
Replication vs Erasure Coding
Ceph supports two fundamentally different data protection strategies, each with different PG implications, performance characteristics, and storage efficiency.
| Factor | Replicated Pool | Erasure Coded Pool |
|---|
| How it works |
Full copies of every object written to multiple OSDs. With size=3, three identical copies exist simultaneously. |
Object split into data chunks and parity chunks using erasure code math. A 4+2 profile stores 4 data and 2 parity chunks across 6 OSDs. |
| Storage efficiency |
Poor — 3x replication uses 3x raw storage for every byte stored. A 100TB usable pool requires 300TB raw. |
Good — a 4+2 profile uses 1.5x raw storage. A 100TB usable pool requires only 150TB raw. |
| Performance |
Excellent for random I/O. Reads can be served from any replica — reads scale linearly with replica count. Low latency. |
Write performance lower than replication due to encoding overhead. Reads require reconstruction when chunks are missing. Best for sequential workloads. |
| Best use cases |
RBD block storage, CephFS metadata pools, any workload requiring low-latency random I/O. OpenStack Cinder volumes. |
RGW object storage, cold data archives, backup targets, large sequential-access datasets where raw capacity efficiency matters more than latency. |
| OSD requirement |
Minimum OSDs equal to replication factor — 3 OSDs for size=3. |
Minimum OSDs equal to k+m — a 4+2 profile requires at least 6 OSDs, preferably spread across 6 failure domains. |
Pool Design for Common OpenStack Deployments
An OpenStack environment backed by Ceph typically uses several pools, each with different performance and durability requirements. The PG allocation across these pools significantly impacts cluster performance.
Standard pool layout
| Pool Name | Purpose | Recommended Type | PG Weight |
|---|
| volumes |
Cinder block storage volumes for Nova instances |
Replicated, size=3 |
High — typically your largest pool |
| images |
Glance image storage — VM templates and ISOs |
Replicated, size=3 |
Medium — images are large but fewer in number |
| vms |
Nova ephemeral disk storage for running instances |
Replicated, size=3 |
High — active I/O from all running instances |
| backups |
Cinder volume backups via RBD |
Erasure coded 4+2 |
Medium-low — sequential writes, rarely read |
| .rgw.buckets.data |
Swift/RGW object store data |
Erasure coded 4+2 |
Variable — depends on object storage usage |
Frequently Asked Questions
What happens if I set pg_num too low on an existing pool?
Data will distribute unevenly and some OSDs will become hotter than others. You can increase pg_num on an existing pool with ceph osd pool set <pool> pg_num <new-value> — Ceph will automatically begin migrating data to rebalance across the new PG count. This migration causes cluster-wide backfill activity that impacts performance. Always increase pg_num gradually and monitor ceph -s until the cluster returns to HEALTH_OK before increasing further.
Can I decrease pg_num on an existing pool?
PG merging (decreasing pg_num) was introduced in Ceph Nautilus (14.x) and is supported on clusters running Nautilus or later. Use ceph osd pool set <pool> pg_num <lower-value> followed by ceph osd pool set <pool> pgp_num <lower-value>. PG merging is slower than splitting and causes a different pattern of data movement. Always take a snapshot or backup before reducing PG counts on production pools.
What is the difference between pg_num and pgp_num?
pg_num controls how many placement groups exist for a pool. pgp_num controls how many of those PGs are used for placement decisions by CRUSH. When you increase pg_num, Ceph creates new PGs but does not move any data until pgp_num is also increased. This two-step process lets you stage large migrations by increasing pg_num first, then gradually increasing pgp_num to control the rate of data movement. In steady state operation these two values should always be equal.
How do I check the current PG distribution health of my cluster?
Run ceph osd pool stats to see read/write activity per pool and spot hotspots. Use ceph pg dump | awk '{print $14}' | sort | uniq -c | sort -rn | head -20 to see which OSDs host the most PGs. The ceph balancer status command shows whether the automatic balancer module is active and what improvements it could make. A healthy cluster should show fairly even PG distribution across all OSDs of similar size.
Should I enable the Ceph balancer module?
Yes, on Ceph Luminous and later the balancer module provides automatic PG redistribution that achieves better OSD utilization than the basic CRUSH algorithm alone. Enable it with ceph mgr module enable balancer and set it to upmap mode with ceph balancer mode upmap. The upmap balancer makes fine-grained adjustments to PG mappings without requiring a CRUSH map change. On large clusters it routinely improves OSD utilization variance from 20-30% down to under 5%.