磁盘空间

磁盘空间超85%，引发pod回收

I1014 19:25:50.887467   22790 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 85% which is over the high threshold (85%). Trying to free 9488877977 bytes
W1014 19:27:13.081862   22790 eviction_manager.go:329] eviction manager: attempting to reclaim ephemeral-storage
I1014 19:27:13.081935   22790 container_gc.go:85] attempting to delete unused containers
I1014 19:27:13.089080   22790 image_gc_manager.go:317] attempting to delete unused images
I1014 19:27:13.101351   22790 eviction_manager.go:340] eviction manager: must evict pod(s) to reclaim ephemeral-storage
I1014 19:27:13.101411   22790 eviction_manager.go:358] eviction manager: pods ranked for eviction: calico-node-ck4lv_kube-system(41e59785-cf9a-11e8-9fea-fa168f866a38), intra-router-cg495_intra(64b19736-aa01-11e8-80db-fa168f005614)

ingress-controller pod也被回收，导致有几分钟流量无法进来

查看监控，磁盘空间在19:27左右确实超过了85%，eviction执行之后，磁盘空间被释放

查找占用磁盘空间的容器，发现是ingress-controller占用了大量磁盘空间。

88G	672e99e643cf26c22a471ffa36ba2c924c9c3d9a437ff471967100f60702b3c7-json.log
4.0K	checkpoints
12K	config.v2.json
4.0K	hostconfig.json
4.0K	mounts

文件内容是容器日志

解决方案

配置logging driver

• loggin configure
• Docker 如何支持多种日志方案？
• Kubernetes Pod日志太大导致空间问题

/var/空间导致的evict

Docker Root Dir 没有设置为/var/，依然出现DiskPressure并触发了pod驱逐，所有pod均被删除。

调查发现kubelet的root dir也会作为磁盘空间的指标，参考eviction-policy。而该参数的默认值为/var/lib/kubelet。修改为大磁盘分区即可

--root-dir string                                                                                           Directory path for managing kubelet files (volume mounts,etc). (default "/var/lib/kubelet")