主要参考每天5分钟玩转docker容器技术博客。
记录下自己的实验过程。
如果想学习，请直接参考该博客http://www.cnblogs.com/CloudMan6/网络概述
flannel为每个host分配一个subnet，容器从此subnet中分配ip。这些ip可以在host间路由，容器间无需nat和port mapping就可以跨主机通信。
每个subnet都是从一个更大的ip池中划分的，flannel会在每个主机上运行一个叫flanneld的agent，职责是从池中分配subnet。为了在各个主机间共享信息，flannel用etcd存放网络配置、已分配的subnet、host的ip等信息。
数据包如何在主机间转发是由 backend 实现的。flannel 提供了多种 backend，最常用的有 vxlan 和 host-gw
环境搭建
安装配置etcd
在192.168.19.120上运行如下脚本：
ETCD_VER=v2.3.7
DOWNLOAD_URL=https://github.com/coreos/etcd/releases/download
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
mkdir -p /tmp/test-etcd && tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/test-etcd --strip-components=1
cp /tmp/test-etcd/etcd* /usr/local/bin/运行etcd
etcd -listen-client-urls http://192.168.19.120:2379 -advertise-client-urls http://192.168.19.120:2379build flannel
- 下载并重命名 image。
docker pull cloudman6/kube-cross:v1.6.2-2
docker tag cloudman6/kube-cross:v1.6.2-2 gcr.io/google_containers/kube-cross:v1.6.2-2
- 下载 flannel 源码。
git clone https://github.com/coreos/fla…
- 开始构建。
cd flannel
make dist/flanneld-amd64
- 将flannel执行文件拷贝到host2 和host3
scp dist/flanneld-amd64 192.168.19.121:/usr/local/bin/flanneld
scp dist/flanneld-amd64 192.168.19.122:/usr/local/bin/flanneld
将flannel网络的配置文件保存到etcd中
配置文件如下：
{
"Network":"10.2.0.0/16",
"SubnetLen":24,
"Backend":{
"Type":"vxlan"
}
}解释：
- Network 定义该网络的ip池
- SubnetLen 指定每个主机分配到的subnet大小为24位，即 10.2.x.0/24
- Backend为vxlan，即主机间通过vxlan通信。
将配置存入etcd
etcdctl --endpoint=192.168.19.120:2379 set /docker-test/network/config < flannel_config.json启动flannel
启动命令
在两台机器上运行如下命令：
flanneld -etcd-endpoints=http://192.168.19.120:2379 -iface=enp0s3 -etcd-prefix=/docker-test/network-etcd-endpoints 指定 etcd url
-iface 指定主机间数据传输使用的interface
-etcd-prefix 指定etcd存放flannel网络配置文件的key
输出如下：
root@host3:~# flanneld -etcd-endpoints=http://192.168.19.120:2379 -iface=enp0s3 -etcd-prefix=/docker-test/network
I0520 02:24:57.594377 2463 main.go:499] Using interface with name enp0s3 and address 192.168.19.122
I0520 02:24:57.594766 2463 main.go:516] Defaulting external address to interface address (192.168.19.122)
I0520 02:24:57.595348 2463 main.go:240] Created subnet manager: Etcd Local Manager with Previous Subnet: None
I0520 02:24:57.595643 2463 main.go:243] Installing signal handlers
I0520 02:24:57.598847 2463 main.go:358] Found network config - Backend type: vxlan
I0520 02:24:57.599288 2463 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
I0520 02:24:57.649392 2463 local_manager.go:234] Picking subnet in range 10.2.1.0 ... 10.2.255.0
I0520 02:24:57.651626 2463 local_manager.go:220] Allocated lease (10.2.99.0/24) to current node (192.168.19.122)
I0520 02:24:57.653626 2463 main.go:305] Wrote subnet file to /run/flannel/subnet.env
I0520 02:24:57.654905 2463 main.go:309] Running backend.
I0520 02:24:57.660772 2463 vxlan_network.go:60] watching for new subnet leases
I0520 02:24:57.665879 2463 main.go:401] Waiting for 23h0m0.025221044s to renew lease
I0520 02:24:57.672707 2463 iptables.go:115] Some iptables rules are missing; deleting and recreating rules
I0520 02:24:57.674333 2463 iptables.go:137] Deleting iptables rule: -s 10.2.0.0/16 -j ACCEPT
I0520 02:24:57.678549 2463 iptables.go:137] Deleting iptables rule: -d 10.2.0.0/16 -j ACCEPT
I0520 02:24:57.688935 2463 iptables.go:125] Adding iptables rule: -s 10.2.0.0/16 -j ACCEPT
I0520 02:24:57.696815 2463 iptables.go:125] Adding iptables rule: -d 10.2.0.0/16 -j ACCEPT启动后的效果
- 新建了一个interface flannel.1， 且配置上subnet的第一个ip 10.2.21.0
root@host2:~# ip addr show flannel.1
8: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 6e:9f:1d:bf:5c:b9 brd ff:ff:ff:ff:ff:ff
inet 10.2.21.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::6c9f:1dff:febf:5cb9/64 scope link
valid_lft forever preferred_lft forever- 添加了一条路由：目的地址为flannel网络10.2.0.0/16的数据包都有flannel.1转发
root@host2:~# ip r
default via 192.168.19.2 dev enp0s3 onlink
10.2.99.0/24 via 10.2.99.0 dev flannel.1 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 linkdown
192.168.19.0/24 dev enp0s3 proto kernel scope link src 192.168.19.121
192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.101 配置docker连接flannel
编辑docker配置文件
*两台机器都做此操作
首先查看/run/flannel/subnet.env
root@host2:~# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.2.0.0/16
FLANNEL_SUBNET=10.2.21.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false之后，编辑docker的配置文件vim /etc/systemd/system/docker.service.d/10-machine.conf，给ExecStart命令的参数中添加–bip和–mtu，分别对应subnet.env的 FLANNEL_SUBNET 与FLANNEL_MTU
root@host2:~# cat /etc/systemd/system/docker.service.d/10-machine.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic --bip=10.2.21.1/24 --mtu=1450
Environment=重启docker
systemctl daemon-reload
systemctl restart docker.service效果
对比路由表
# docker重启前
root@host3:~# ip r
default via 192.168.19.2 dev enp0s3 onlink
10.2.21.0/24 via 10.2.21.0 dev flannel.1 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 linkdown
192.168.19.0/24 dev enp0s3 proto kernel scope link src 192.168.19.122
192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.102
root@host3:~# systemctl daemon-reload
root@host3:~# systemctl restart docker.service
# docker重启后
root@host3:~# ip r
default via 192.168.19.2 dev enp0s3 onlink
10.2.21.0/24 via 10.2.21.0 dev flannel.1 onlink
10.2.99.0/24 dev docker0 proto kernel scope link src 10.2.99.1 linkdown
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 linkdown
192.168.19.0/24 dev enp0s3 proto kernel scope link src 192.168.19.122
192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.102 flannel 没有创建新的 docker 网络，而是直接使用默认的 bridge 网络。同一主机的容器通过 docker0 连接，跨主机流量通过 flannel.1 转发。
将容器连接到flannel网络
root@host2:~# docker run -itd --name bbox1 busybox
root@host2:~# docker exec bbox1 ip r
default via 10.2.21.1 dev eth0
10.2.21.0/24 dev eth0 scope link src 10.2.21.2
root@host3:~# docker run -itd --name bbox2 busybox
root@host3:~# docker exec bbox2 ip r
default via 10.2.99.1 dev eth0
10.2.99.0/24 dev eth0 scope link src 10.2.99.2可以发现网络可通
# 10.2.99.2 为bbox2的ip
root@host2:~# docker exec bbox1 ping -c 1 10.2.99.2
PING 10.2.99.2 (10.2.99.2): 56 data bytes
64 bytes from 10.2.99.2: seq=0 ttl=62 time=0.604 ms使用traceroute跟踪包
root@host2:~# docker exec bbox1 traceroute 10.2.99.2
traceroute to 10.2.99.2 (10.2.99.2), 30 hops max, 46 byte packets
1 10.2.21.1 (10.2.21.1) 0.010 ms 0.216 ms 0.020 ms
2 10.2.99.0 (10.2.99.0) 0.938 ms 2.731 ms 1.663 ms
3 10.2.99.2 (10.2.99.2) 2.016 ms 0.647 ms 0.596 ms- 发给docker0
- 之后发给flannel.1
- flannel.1 将数据包封装成 VxLAN，通过 enp0s8 发送给 host3，
- host3 收到包解封装，发现数据包目的地址为 10.2.99.2，根据路由表（下图）将数据包发送给 flannel.1，并通过 docker0 到达 bbox2
### dns不可用
无法通过主机名ping通机器。
