金丝雀发布(又称灰度发布、灰度更新):
金丝雀发布一般是先发1台机器,或者一个小比例,例如2%的服务器,主要做流量验证用,也称为金丝雀 (Canary) 测试,国内常称灰度测试。以前旷工下矿前,会先放一只金丝雀进去用于探测洞里是否有有毒气体,看金丝雀能否活下来,金丝雀发布由此得名。简单的金丝雀测试一般通过手工测试验证,复杂的金丝雀测试需要比较完善的监控基础设施配合,通过监控指标反馈,观察金丝雀的健康状况,作为后续发布或回退的依据。如果金丝测试通过,则把剩余的 V1 版本全部升级为 V2 版本。如果金丝雀测试失败,则直接回退金丝雀,发布失败。
发布的各种策略详细介绍参考:蓝绿、ABTest、滚动部署、灰度发布、金丝雀发布介绍
实战
打开一个命令行,来监测更新过程
kubectl get pods -l app=myapp -n test -w
## 下面命令执行完之后显示如下,之前的pod还在,新创建了一个pod,没有立即删除。
(也可以使用kubectl rollout statusdeployment myapp-deploy,显示Waiting for deployment "myapp-deploy"rollout to finish: 1 out of 5 new replicas have been updated)
打开另一个命令行来操作,如下:
1、更新镜像,并暂停升级
kubectl set image deployment myapp-deploy myapp:v2 -n test && kubectl rollout pause deployment myapp-deploy -n test
注:上面的步骤解释说明
把myapp这个容器的镜像更新到
myapp:v2
版本,更新镜像之后,创建一个新的pod就立即暂停,这就是我们说的金丝雀发布;如果暂停几个小时之后没有问题,那么取消暂停,就会依次执行后面步骤,把所有pod都升级。
2、解除暂停
确认新版本没问题,则解除暂停
kubectl rollout resume deployment myapp-deploy -n test
在刚才监测的界面可以看到如下一些信息,下面过程是把余下的pod里的容器都更新到新的版本:
myapp-deploy-6bdcd6755d-llrw8 0/1 Pending 0 0s
myapp-deploy-6bdcd6755d-llrw8 0/1 ContainerCreating 0 0s
myapp-deploy-67f6f6b4dc-7cs8v 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-7cs8v 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-7cs8v 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-7cs8v 0/1 Terminating 0 1h
myapp-deploy-6bdcd6755d-llrw8 1/1 Running 0 16s
myapp-deploy-67f6f6b4dc-nhcp2 1/1 Terminating 0 1h
myapp-deploy-6bdcd6755d-r4mrl 0/1 Pending 0 0s
myapp-deploy-6bdcd6755d-r4mrl 0/1 Pending 0 1s
myapp-deploy-6bdcd6755d-r4mrl 0/1 ContainerCreating 0 1s
myapp-deploy-67f6f6b4dc-nhcp2 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-nhcp2 0/1 Terminating 0 1h
myapp-deploy-6bdcd6755d-r4mrl 1/1 Running 0 5s
myapp-deploy-67f6f6b4dc-hwx7w 1/1 Terminating 0 1h
myapp-deploy-6bdcd6755d-j8nj8 0/1 Pending 0 0s
myapp-deploy-6bdcd6755d-j8nj8 0/1 Pending 0 0s
myapp-deploy-6bdcd6755d-j8nj8 0/1 ContainerCreating 0 0s
myapp-deploy-67f6f6b4dc-nhcp2 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-nhcp2 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-hwx7w 0/1 Terminating 0 1h
myapp-deploy-6bdcd6755d-j8nj8 1/1 Running 0 4s
myapp-deploy-67f6f6b4dc-dbcqh 1/1 Terminating 0 1h
myapp-deploy-6bdcd6755d-lpk5b 0/1 Pending 0 1s
myapp-deploy-6bdcd6755d-lpk5b 0/1 Pending 0 1s
myapp-deploy-6bdcd6755d-lpk5b 0/1 ContainerCreating 0 1s
myapp-deploy-67f6f6b4dc-dbcqh 0/1 Terminating 0 1h
myapp-deploy-6bdcd6755d-lpk5b 1/1 Running 0 4s
myapp-deploy-67f6f6b4dc-b4wfc 1/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-b4wfc 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-hwx7w 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-hwx7w 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-b4wfc 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-b4wfc 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-dbcqh 0/1 Terminating 0 1h
myapp-deploy-67f6f6b4dc-dbcqh 0/1 Terminating 0 1h
kubectl get rs -n test
可以看到replicaset控制器有2个了
回滚
如果发现刚才升级的这个版本有问题可以回滚,查看当前有哪几个版本:
kubectl rollout history deployment myapp-deploy -n test
上面可以看到第一版没了,被还原成了第三版,第三版的前一版是第二版
kubectl get rs -n test -o wide
可以看到上面的rs
已经用第一个了,这个就是还原之后的rs