参考资料:
ceph-common
3.10.0-957
以上)
pg_num
计算器:https://ceph.com/pgcalc/
计划对象存储,rbd, fs共用一个集群,开发环境和生成环境区分pool:kube.rbd, kube.fs, kube.dev.rbd, kube.dev.fs
相关命令
# ceph osd pool get <poolname> pg_num # ceph osd pool get <poolname> pgp_num # ceph osd pool set <poolname> pg_num # ceph osd pool set <poolname> pgp_num # ceph osd pool rename <pool> <newpool> # ceph pg dump pgs |grep ^<pool number> |awk '{print $1,$2,$17}'
ceph auth get-or-create client.k8s mon 'allow r' osd 'allow rwx pool=k8s,allow rwx pool=rbd' -o ceph.client.k8s.keyring
# grep key ceph.client.k8s.keyring | awk '{printf "%s", $NF}' | base64 VBGFaeN3OWJYdUZPSHhBQTNrU2E2QlUyaEF5UUV0SnNPRHdXeRT8PQ==
创建类型为 kubernetes.io/rbd
的 Secret
apiVersion: v1 kind: Secret metadata: name: ceph-k8s-secret type: "kubernetes.io/rbd" data: key: VBGFaeN3OWJYdUZPSHhBQTNrU2E2QlUyaEF5UUV0SnNPRHdXeRT8PQ== --- apiVersion: v1 kind: Secret metadata: name: ceph-admin-secret namespace: kube-system type: "kubernetes.io/rbd" data: key: VBGFaeN3OWJYdUZPSHhBQTNrU2E2QlUyaEF5UUV0SnNPRHdXeRT8PQ==
kind: StorageClass # 允许扩容 allowVolumeExpansion: true reclaimPolicy: Retain metadata: name: rbd provisioner: kubernetes.io/rbd parameters: monitors: 10.32.24.11:6789,10.32.24.12:6789,10.32.24.13:6789 adminId: k8s adminSecretName: ceph-k8s-secret adminSecretNamespace: kube-system pool: k8s userId: k8s userSecretName: ceph-k8s-secret
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: test-pvc spec: accessModes: - ReadWriteOnce storageClassName: rbd resources: requests: storage: 10Gi
apiVersion: apps/v1 kind: Deployment metadata: labels: app: ceph-demo name: ceph-demo namespace: dev spec: replicas: 1 template: spec: containers: - env: - name: ONLINE_DATE value: 2019-09-02 image: benchmark:latest name: nginx volumeMounts: - mountPath: "/data" name: test volumes: - name: test persistentVolumeClaim: claimName: test-pvc
操作 | 效果 | 备注 |
---|---|---|
删除未挂载的pvc | 回收策略未Delete时,pv和ceph image都被删除 | 比较危险,应人工执行 |
删除挂载中的pvc | 回收策略未Delete时,Pod销毁后pv和ceph image都被删除 | 比较危险,应人工执行 |
参考: https://kubernetes.io/zh/docs/tasks/administer-cluster/change-pv-reclaim-policy/
kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
无法patch,似乎可以直接删除在重建
https://github.com/kubernetes/kubernetes/issues/72393
升级 kube-controller-manager
到1.12.10版已解决(未完全解决,见 https://github.com/kubesphere/kubesphere/issues/1156#issuecomment-546223518)
不需要在 k8s node 上部署 /etc/ceph/ceph.conf
和 /etc/ceph/client.<user>.keyring
。报错是因为 secret 中存储的 key 有错误。参考:https://github.com/kubernetes/kubernetes/issues/66757
auth: unable to find a keyring on /etc/ceph/ceph.client.k8s-production.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
检查以下项目
echo -n <keyring> |base64