用户工具

站点工具


02-工程实践:kubernetes:issue:multiether

双网卡容器网络不通

现象

calico ipip模式,双网卡机器,ping不通172.20容器网段

解决方案

添加内网路由,calico组网需要是互通的网络

# ip route add 10.0.0.0/8 via 10.112.0.1

分析

如果没有添加内网路由,内网不同机房网段不互通

# 机房1 -> 机房2 (tcpdump只能看到 ICMP echo request,看不到reply)
[root@k8s-node.10-112-3-6 ~]# ping 10.124.2.4 -c 5
PING 10.124.2.4 (10.124.2.4) 56(84) bytes of data.
 
--- 10.124.2.4 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4000ms
 
# 机房1 -> 机房1 不同网段(tcpdump需抓公网接口eth1,可以看到reply,但是发包是eth1公网ip,回包也是给公网ip的)
[root@k8s-node-pushd.10-112-3-6 ~]# ping 10.110.14.5 -c 4
PING 10.110.14.5 (10.110.14.5) 56(84) bytes of data.
64 bytes from 10.110.14.5: icmp_seq=1 ttl=63 time=0.497 ms
64 bytes from 10.110.14.5: icmp_seq=2 ttl=63 time=0.262 ms
64 bytes from 10.110.14.5: icmp_seq=3 ttl=63 time=0.237 ms
64 bytes from 10.110.14.5: icmp_seq=4 ttl=63 time=0.203 ms
 
# 同机房添加路由之后再次抓包,可以在eth0(内网)上抓到icmp包

同机房即使内网互通,calico 网络也不通,还是需要加内网路由。

[root@k8s-node-pushd.10-112-3-6 ~]# route -n |grep 172.20.39
172.20.39.0     10.110.14.6   255.255.255.0   UG    0      0        0 tunl0
[root@k8s-node-pushd.10-112-32-216 ~]# ping 172.20.39.1 -c 3
PING 172.20.39.1 (172.20.39.1) 56(84) bytes of data.
 
--- 172.20.39.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

监控

监控各节点到coredns的联通性,通过Zabbix dns监控

配置管理

通过puppet管理带公网机器的内网路由

        # 带公网ip的机器需要设置内网路由,否则可能到其他机房的内网不通
        'int_gw':
            command => "ip route add 10.0.0.0/8 via `ip addr show |grep 'global eth' |grep 'inet 10\.' |awk '{print \$4}' |sed 's/255/0/g' |awk -F'.' '{print \$1\".\"\$2\".\"\$3\".1\"}'`",
            path    => ["/usr/bin", "/usr/sbin","/bin","/sbin"],
            unless => 'ip route get 10.0.0.0 |grep "via 10\."';

Update:

经询问IDC,正常交付应该是有个软路由文件定义内网路由

[root@k8s-node.10-110-15-9 network-scripts]# readlink -f route-eth2 
/etc/sysconfig/network-scripts/route-eth2
[root@k8s-node.10-110-15-9 network-scripts]# cat route-eth2 
10.0.0.0/8 via 10.110.0.1
02-工程实践/kubernetes/issue/multiether.txt · 最后更改: 2020/04/07 06:34 由 annhe