Annhe's Wiki

用户工具

站点工具

您在这里: Annhe's Wiki » 工程实践 » Kubernetes » 故障记录 » kube-proxy 1.12.5 重启断网
02-工程实践:kubernetes:issue:kube-proxy_1.12.5_network
书籍创建器
 将本页添加到您的书籍中
书籍创建器
 从您的书籍中删除本页  

 显示/修改书籍(0 页面)
 帮助

目录

kube-proxy 1.12.5 重启断网

解决方案

https://github.com/kubernetes/kubernetes/issues/73154

可能受cleanup-ipvs参数的影响 

# ./kube-proxy --help |grep clean
      --cleanup                                      If true cleanup iptables and ipvs rules and exit.
      --cleanup-ipvs                                 If true make kube-proxy cleanup ipvs rules before running.  Default is true (default true)

描述

为解决1.12.2 ipvs rs不更新 问题，升级到1.12最新的1.12.5版本，升级过程中似乎有断网问题，应用会受到影响，似乎每重启一次kube-proxy，该node上容器网络就中断一会

分析

以上图中业务使用169网段的Cluster IP来访问另一个业务，当kube-proxy重启时，连不上Clsuter IP，导致响应慢

Url监控程序在kube-proxy重启时，由于连不上coredns，导致大面积误报，此故障也可以佐证以上图中的情况。

日志

# grep "169.169.0.2" kube-proxy.INFO 
I0120 16:41:24.997360   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/TCP/172.20.65.159:53
I0120 16:41:24.997390   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/TCP/172.20.65.159:53: 0 ActiveConn, 12 InactiveConn
I0120 16:41:24.997410   11442 graceful_termination.go:154] Adding an element to graceful delete rsList: 169.169.0.2:53/TCP/172.20.65.159:53
I0120 16:41:24.997423   11442 graceful_termination.go:66] Adding rs 169.169.0.2:53/TCP/172.20.65.159:53 to graceful delete rsList
I0120 16:41:24.997432   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/TCP/172.20.38.193:53
I0120 16:41:24.997455   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/TCP/172.20.38.193:53: 0 ActiveConn, 12 InactiveConn
I0120 16:41:24.997474   11442 graceful_termination.go:154] Adding an element to graceful delete rsList: 169.169.0.2:53/TCP/172.20.38.193:53
I0120 16:41:24.997483   11442 graceful_termination.go:66] Adding rs 169.169.0.2:53/TCP/172.20.38.193:53 to graceful delete rsList
I0120 16:41:24.997490   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/TCP/172.20.41.162:53
I0120 16:41:24.997516   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/TCP/172.20.41.162:53: 0 ActiveConn, 11 InactiveConn
I0120 16:41:24.997530   11442 graceful_termination.go:154] Adding an element to graceful delete rsList: 169.169.0.2:53/TCP/172.20.41.162:53
I0120 16:41:24.997537   11442 graceful_termination.go:66] Adding rs 169.169.0.2:53/TCP/172.20.41.162:53 to graceful delete rsList
I0120 16:41:24.997753   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/UDP/172.20.65.159:53
I0120 16:41:24.997778   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/UDP/172.20.65.159:53: 0 ActiveConn, 13855 InactiveConn
I0120 16:41:24.997795   11442 graceful_termination.go:154] Adding an element to graceful delete rsList: 169.169.0.2:53/UDP/172.20.65.159:53
I0120 16:41:24.997802   11442 graceful_termination.go:66] Adding rs 169.169.0.2:53/UDP/172.20.65.159:53 to graceful delete rsList
I0120 16:41:24.997815   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/UDP/172.20.38.193:53
I0120 16:41:24.997867   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/UDP/172.20.38.193:53: 0 ActiveConn, 13877 InactiveConn
I0120 16:41:24.997885   11442 graceful_termination.go:154] Adding an element to graceful delete rsList: 169.169.0.2:53/UDP/172.20.38.193:53
I0120 16:41:24.997893   11442 graceful_termination.go:66] Adding rs 169.169.0.2:53/UDP/172.20.38.193:53 to graceful delete rsList
I0120 16:41:24.997899   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/UDP/172.20.41.162:53
I0120 16:41:24.997925   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/UDP/172.20.41.162:53: 0 ActiveConn, 13920 InactiveConn
I0120 16:41:24.997945   11442 graceful_termination.go:154] Adding an element to graceful delete rsList: 169.169.0.2:53/UDP/172.20.41.162:53
I0120 16:41:24.997952   11442 graceful_termination.go:66] Adding rs 169.169.0.2:53/UDP/172.20.41.162:53 to graceful delete rsList
I0120 16:41:25.008265   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/UDP/172.20.38.193:53
I0120 16:41:25.008424   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/UDP/172.20.38.193:53: 0 ActiveConn, 13877 InactiveConn
I0120 16:41:25.008569   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/TCP/172.20.65.159:53
I0120 16:41:25.008617   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/TCP/172.20.65.159:53: 0 ActiveConn, 12 InactiveConn
I0120 16:41:25.008630   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/TCP/172.20.38.193:53
I0120 16:41:25.008696   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/TCP/172.20.38.193:53: 0 ActiveConn, 12 InactiveConn
I0120 16:41:25.008719   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/TCP/172.20.41.162:53
I0120 16:41:25.008766   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/TCP/172.20.41.162:53: 0 ActiveConn, 11 InactiveConn
I0120 16:41:25.008779   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/UDP/172.20.65.159:53
I0120 16:41:25.008828   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/UDP/172.20.65.159:53: 0 ActiveConn, 13855 InactiveConn
I0120 16:41:25.008840   11442 graceful_termination.go:160] Trying to delete rs: 169.169.0.2:53/UDP/172.20.41.162:53
I0120 16:41:25.008882   11442 graceful_termination.go:171] Not deleting, RS 169.169.0.2:53/UDP/172.20.41.162:53: 0 ActiveConn, 13920 InactiveConn
I0120 16:41:25.222215   11442 service.go:309] Adding new service port "kube-system/kube-dns:dns" at 169.169.0.2:53/UDP
I0120 16:41:25.222232   11442 service.go:309] Adding new service port "kube-system/kube-dns:dns-tcp" at 169.169.0.2:53/TCP
I0120 16:41:25.222845   11442 proxier.go:714] Stale udp service kube-system/kube-dns:dns -> 169.169.0.2
I0120 16:41:25.327280   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:41:25.327413   11442 proxier.go:1559] new ep "169.169.0.2:53/TCP/172.20.38.193:53" is in graceful delete list
I0120 16:41:25.327493   11442 proxier.go:1559] new ep "169.169.0.2:53/TCP/172.20.41.162:53" is in graceful delete list
I0120 16:41:25.327563   11442 proxier.go:1559] new ep "169.169.0.2:53/TCP/172.20.65.159:53" is in graceful delete list
I0120 16:41:25.334400   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:41:25.334502   11442 proxier.go:1559] new ep "169.169.0.2:53/UDP/172.20.38.193:53" is in graceful delete list
I0120 16:41:25.334696   11442 proxier.go:1559] new ep "169.169.0.2:53/UDP/172.20.41.162:53" is in graceful delete list
I0120 16:41:25.334769   11442 proxier.go:1559] new ep "169.169.0.2:53/UDP/172.20.65.159:53" is in graceful delete list
I0120 16:41:55.182752   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:41:55.185431   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:42:25.323615   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:42:25.327466   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:42:55.453234   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:42:55.466214   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:43:25.597583   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:43:25.601359   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:43:55.766259   11442 proxier.go:1496] Bind addr 169.169.0.2
I0120 16:43:55.768480   11442 proxier.go:1496] Bind addr 169.169.0.2
02-工程实践/kubernetes/issue/kube-proxy_1.12.5_network.txt · 最后更改: 2020/04/07 06:34 由 annhe

页面工具

除额外注明的地方外，本维基上的内容按下列许可协议发布： CC Attribution-Noncommercial-Share Alike 4.0 International
CC Attribution-Noncommercial-Share Alike 4.0 International Valid HTML5 Valid CSS Driven by DokuWiki