ceph osd getcrushmap -o /tmp/crushmap crushtool -d /tmp/crushmap -o crush_map crushtool -c crush_map -o /tmp/crushmap ceph osd setcrushmap -i /tmp/crushmap
[client] rbd_cache = true
If the automatic device class detection gets something wrong (e.g., because the device driver is not properly exposing information about the device via /sys/block), you can also adjust device classes from the command line:
$ ceph osd crush rm-device-class osd.2 osd.3 done removing class of osd(s): 2,3 $ ceph osd crush set-device-class ssd osd.2 osd.3 set osd(s) 2,3 to class 'ssd'
# types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 region type 10 root
ceph osd crush set 0 0.5 pool=default host=proxmox01 ceph osd crush set 1 0.5 pool=default host=proxmox02 ceph osd crush set 2 0.5 pool=default host=proxmox03
rados -p ceph bench 60 write --no-cleanup
Default object size is 4 MB, and the default number of simulated threads (parallel writes) is 16.
write / seq / read
rados -p ceph df
ceph osd pool set data size 3 ceph osd pool set data min_size 2
For n = 4 nodes each with 1 osd and 1 mon and settings of replica min_size 1 and size 4 three osd can fail, only one mon can fail (the monitor quorum means more than half will survive). 4 + 1 number of monitors is required for two failed monitors (at least one should be external without osd). For 8 monitors (four external monitors) three mon can fail, so even three nodes each with 1 osd and 1 mon can fail. I am not sure that setting of 8 monitors is possible.
For three nodes each with one monitor and osd the only reasonable settings are replica min_size 2 and size 3 or 2. Only one node can fail. If you have an external monitors, if you set min_size to 1 (this is very dangerous) and size to 2 or 1 the 2 nodes can be down. But with one replica (no copy, only original data) you can loose your job very soon.