calico ipip模式,双网卡机器,ping不通172.20容器网段
添加内网路由,calico组网需要是互通的网络
# ip route add 10.0.0.0/8 via 10.112.0.1
如果没有添加内网路由,内网不同机房网段不互通
# 机房1 -> 机房2 (tcpdump只能看到 ICMP echo request,看不到reply) [root@k8s-node.10-112-3-6 ~]# ping 10.124.2.4 -c 5 PING 10.124.2.4 (10.124.2.4) 56(84) bytes of data. --- 10.124.2.4 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4000ms # 机房1 -> 机房1 不同网段(tcpdump需抓公网接口eth1,可以看到reply,但是发包是eth1公网ip,回包也是给公网ip的) [root@k8s-node-pushd.10-112-3-6 ~]# ping 10.110.14.5 -c 4 PING 10.110.14.5 (10.110.14.5) 56(84) bytes of data. 64 bytes from 10.110.14.5: icmp_seq=1 ttl=63 time=0.497 ms 64 bytes from 10.110.14.5: icmp_seq=2 ttl=63 time=0.262 ms 64 bytes from 10.110.14.5: icmp_seq=3 ttl=63 time=0.237 ms 64 bytes from 10.110.14.5: icmp_seq=4 ttl=63 time=0.203 ms # 同机房添加路由之后再次抓包,可以在eth0(内网)上抓到icmp包
同机房即使内网互通,calico 网络也不通,还是需要加内网路由。
[root@k8s-node-pushd.10-112-3-6 ~]# route -n |grep 172.20.39 172.20.39.0 10.110.14.6 255.255.255.0 UG 0 0 0 tunl0 [root@k8s-node-pushd.10-112-32-216 ~]# ping 172.20.39.1 -c 3 PING 172.20.39.1 (172.20.39.1) 56(84) bytes of data. --- 172.20.39.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 1999ms
监控各节点到coredns的联通性,通过Zabbix dns监控
通过puppet管理带公网机器的内网路由
# 带公网ip的机器需要设置内网路由,否则可能到其他机房的内网不通 'int_gw': command => "ip route add 10.0.0.0/8 via `ip addr show |grep 'global eth' |grep 'inet 10\.' |awk '{print \$4}' |sed 's/255/0/g' |awk -F'.' '{print \$1\".\"\$2\".\"\$3\".1\"}'`", path => ["/usr/bin", "/usr/sbin","/bin","/sbin"], unless => 'ip route get 10.0.0.0 |grep "via 10\."';
Update:
经询问IDC,正常交付应该是有个软路由文件定义内网路由
[root@k8s-node.10-110-15-9 network-scripts]# readlink -f route-eth2 /etc/sysconfig/network-scripts/route-eth2 [root@k8s-node.10-110-15-9 network-scripts]# cat route-eth2 10.0.0.0/8 via 10.110.0.1