Proxmox Ceph Network Design: Public, Cluster, and 10GbE

proxmox ceph network - custom-cephnet-featured.png

After hardware (Ceph requirements), the top cause of slow ceph proxmox clusters is **shared 1GbE** for replication and VM traffic. Split networks early.

Storage setup: Proxmox Ceph storage.

Public vs cluster network

Client I/O (VM disks) vs OSD replication/backfill — keep on separate subnets when possible.

bashceph config idea
ceph config dump | grep -i network | head
public
cluster

10GbE and bonding

Use LACP bonds with switch-side config documented; verify `iperf3` between nodes before OSD traffic.

bashiperf
iperf3 -s &
# on peer: iperf3 -c node2 -P 4
10GbE
bond

MTU / jumbo frames

Enable end-to-end or stay at 1500 — half-enabled jumbo causes silent pain.

WarningChange MTU on Ceph and switches together; document rollback.
MTU
9000

Symptoms of bad networking

Slow recovery, high latency in `ceph -w`, VMs stutter during rebalance.

bashwatch
ceph -s
ceph osd perf
slow ops
watch

Quick Reference

  • Separate cluster replication VLAN
  • Test with iperf3 before OSDs
  • Pair with Proxmox SDN for tenant nets

Related tutorials

Diagrams are original illustrations by Gnome IT Solutions. Tutorial text © Gnome IT Solutions.