Kubernetes 1.16大版本发布说明：自定义资源、全面调整的指标，以及存储卷扩展 [译]

半兽人 发表于: 2019-09-19   最后更新时间: 2019-09-24 01:07:35  
{{totalSubscript}} 订阅， 5,617 游览

今天（9月18日），我们自豪地宣布 Kubernetes 1.16 正式发布，这也是我们在2019年之内推出的第三个版本！ Kubernetes 1.16 包含 31 项增强功能，其中8项增强性功能已经稳定运行，另有 8 项增强性功能处于beta测试阶段，15 项处于alpha测试阶段。

kubernetes 1.16 主要更新了什么？

自定义资源 - CustomResourceDefinition（CRD） 一直作为Kubernetes的可扩展机制得到广泛使用，但自1.7 版本发布以来一直处于beta测试阶段。本次 Kubernetes 1.16 的中，CRD 迎来了自己的通用（GA）版本。

全面调整的指标 - Kubernetes此前一直广泛采用一套全局指标注册表，用于注册需要公开的各项指标。通过这样一套指标注册表，各项指标能够以更透明的方式进行注册。之前，各类稳定性要求都禁止使用Kubernetes指标。

存储卷扩展 - 新版本中包含一些与存储卷和存储卷修改有关的增强功能。 在CSI specs中，Volume 调整大小的支持到了beta测试阶段，其支持任何CSI spec皆可调整 Volume 大小。

其他值得注意的功能更新

  • Topology Manager（拓扑管理）, 是一种新的Kubelet组件， 用于协调资源分配的决策，优化资源分配
  • IPv4/IPv6双栈，可将IPv4和IPv6 分配给 Pod 和 Services。
  • API Server Network Proxy 变更为alpha。
  • 为Cloud Controller Manager Migration 提供更多的扩展。
  • 在1.16版本，弃用 extensions/v1beta1，apps/v1beta1和apps/v1beta2 APIs。

已知的问题

etcd 和 KMS 插件运行状况检查不会在新的 livez 和 readyz endpoints点中暴露。这将在 1.16.1 中修复。

运行 iptables 1.8.0或更高版本的系统应在传统模式下启动它。请注意，这将影响 Kubernetes 的所有版本，而不仅仅是 v1.16.0。有关该问题以及如何应用解决方法的更多详细信息，请参阅官方文档。

吉祥物发布

Kubernetes 1.16版本的发布徽章，灵感源自阿波罗16号的任务徽章。它代表着发布团队与社区付出的辛勤努力，也是对我们在整个发布周期之内分享挑战与乐趣的肯定及鼓励。这里，感谢微软公司的Ronan Flynn-Curran创作出这件华丽的作品。

kubernetes

发布团队介绍（Release Team）

此次发行版源自数百名技术与非技术贡献者的共同努力。在这里，我们要特别感谢微软公司首席项目经理Lachlan Everson所领导的发布团队。团队中的32名成员协调了发布工作中的各项具体事务，包括文档、测试、验证以及功能完整性等等。

随着Kubernetes的不断发展，我们的发布过程也代表着开源软件开发协作领域的又一惊人成功。Kubernetes仍在不断获得新用户，快速成长本身则创造出积极折反馈周期，吸引更多贡献者提交代码，最终建立起强大且极具活力的社区群体。截至目前，Kubernetes拥有超过32000名个人贡献者，社区成员总量则超过66000人。

其他增强的功能

自定义资源迎来通用版本

CustomResourceDefinition（CRD）已经成为Kubernetes生态系统中的重要基础。作为对ThirdPartyResources原型方案的重新设计成果，CRD终于通过apiextensions.k8s.io/v1在1.16版本中步入通用阶段，并整合了大量Kubernetes发展过程中积累到的API相关演变经验。在推出通用版本中，首要关注点就是API客户端的数据一致性。

自定义资源迎来通用版本

CRD已经成为Kubernetes生态系统中的重要基础。作为对ThirdPartyResources原型方案的重新设计成果，CRD终于通过apiextensions.k8s.io/v1在1.16版本中步入通用阶段，且整合了大量Kubernetes发展过程中积累到的API相关演变经验。在推出通用版本时，我们的首要关注重点就是API客户端的数据一致性。

当大家升级到GA API时，会很快注意到以往的几个可选条目被升级为必要（或默认）选项。事实证明，结构模式、修剪未知字段、验证以及保护*.k8s.io组对于保障API正常使用确实非常重要，而现在我们更不能容忍意外状况的发生。将其转为默认选项，是API演进的另一大重要组成部分，默认情况下CRD.v1支持将直接启用。再配合CRD转换机制，足以构建起能够经受住时间考验的稳定API，正如原生Kubernetes资源能够在不破坏向下兼容性实现变更一样。

当然，CRD API的更新之路还远没有结束。我们对于任意子资源、API组迁移以及提升序列化协议效率等功能已经有了一定的改进思路，但这些变更应该会以可选形式提供，并与GA API中已经存在的内容互相补充。总之，一切以便利性为目标！

利用Windows增强开启新的大门

Beta：增强Windows容器的工作负载标识选项

Active Directory Group Managed Service Account（简称GMSA）支持即将完成beta测试阶段，而且将不推荐用户使用alpha版本阶段支持的某些注释。GMSA是一种特定类型的Active Directory账户，其允许Windows容器通过网络传输身份标识并与其它资源进行通信。Windows容器现在可以完成身份验证并访问外部资源。此外，GMSA还提供自动密码管理、经过简化的服务主体名称（SPN）管理，以及将管理委派给多服务器上其他管理员等功能。

在alpha测试阶段，我们加入了对RunAsUserName的支持。这是一条字符串，用于指定Windows中的容器标识（或者用户名）以运行该容器的入口点，其同时也是securityContext（WindowsSecurityContextOptions）中新添加的windowsOptions组件的一部分。

Alpha：利用kubeadm改进设置与节点添加体验

新版本引入了kubeadm的alpha支持，这意味着Kubernetes用户能够轻松向现有集群当中添加（以及重置）Windows工作节点，操作方式与Linux节点完全相同。用户可以利用kubeadm准备Windows节点并加入集群。在操作完成之后，该节点将处于Ready状态并能够随时运行Windows容器。此外，我们还提供一组Windows专用脚本，旨在配合节点添加的其它资源与CNI安装需求。

Alpha：引入对容器存储接口（CSI）的支持

为树外提供程序引入CSI插件支持，这意味着Kubernetes集群中的Windows节点能够利用持久存储功能运行基于Windows的工作负载。这显著扩展了Windows工作负载的存储选项范围，使用户能够在FlexVolume与树内存储插件之外获得新的选择。这一功能通过主机操作系统代理实现，该代理能够代理容器在Windows节点上执行高权限操作。

引入Endpoint切片（Introducing Endpoint Slices）

Kubernetes 1.16版本还包含一项令人兴奋的全新alpha功能：端点切片（Endpoint Slices）。这些切片实际上就是一种指向端点资源的可扩展替代方案。在幕后，这些资源为Kubernetes的网络路由提供重要支持。每个网络端点都在这些资源之内受到追踪，而kube-proxy则利用这些端点生成代理规则，从而允许各Pod在Kubernetes之内轻松实现相互通信。

提供更强大的可扩展性

Endpoint切片的一大关键目标，在于为Kubernetes服务提供更强大的可扩展性。对于现有端点资源，单一资源必须包含用于表示与某项服务相关联的全部Pod的网络端点。但如果该服务扩展至成千上万Pod，那么对应的端点资源也将无比庞大。在这种情况下，对该服务中的某一商战进行添加或删除，都会带来可观的操作成本。随着每一次端点资源的更新，代码中与该端点相关的部分都需要获取一份关于该资源的完整副本。现在，由于集群内的各个节点上都运行有kube-proxy，因此只需要面向各个节点发送副本即可。这项调整在小规模使用场景下影响不大，但在大规模集群中却会带来极为显著的影响。

举个简单的例子，假设我们拥有一个包含5000个节点与一个1 MB endpoint对象的集群，每一次更新都将带来大约5 GB的数据传输量（相当于一张DVD光盘）。考虑到部署期间滚动更新等事件的频繁出现，这无疑是巨大的资源浪费。

endpoint端点切片，服务的网络endpoint可以被拆分为多种资源，从而显著降低大规模更新所需要传输的数据总量。在默认情况下，每个endpoint切片最多包含100个端点。

例如，假设我们拥有一个包含2万个网络端点，且分布在5000个节点上的集群。利用endpoint切片对单一endpoint进行更新，将带来更高的执行效率，因为每个endpoint只代表着网络端点总数中的一小部分。相较于以往将大端点对象传输至各个节点的方式，现在我们只需要传输已经变更的小型端点切片。实际效果就是，现在更新操作的数据传输量仅相当于以往的约二百分之一。

Endpoints Endpoint Slices
资源 1 20k / 100 = 200
网络endpoints存储 1 * 20k = 20k 200 * 100 = 20k
每个资源的大小 20k * const = ~2.0 MB 100 * const = ~10 kB
监听事件数据的传输量 ~2.0MB * 5k = 10GB ~10kB * 5k = 50MB

endpoint切片的第二大主要作用，是提供一种在各类用例之内都具有高度可扩展性与实用性的资源。endpoint切片的还带来了新的拓扑属性。在默认情况下，其将填充Kubernetes中当前使用的拓扑标签，用以指示region与zone等属性。当然，这一字段也可以填充进其它自定义标签并配合更为专业的用例。

endpoint切片还实现了更强大的地址类型灵活性。每个endpoint切片都包含一份地址列表，多地址初始用例即可支持同时具有 IPv4与IPv6地址的双栈endpoint。

作为 Kubernetes 1.16 版本中的 alpha测试阶段 的功能，endpoint切片在默认情况下并未启用，但大家可以参阅说明文档了解如何在集群中将其开启。

安装 Kubernetes 1.16

Kubernetes 1.16 目前已经可以从GitHub上下载。

也可以使用kubeadm很方便的安装1.16。

升级说明

升级之前务必阅读以下内容。

集群的生命周期

  • amd64 的容器镜像tar文件现在将包含RepoTags清单中的manifest.json部分。如果使用docker manifests，则没有可见的更改。(#80266, @javier-b-perez)

  • kubeadm现在在TLS引导之后，删除bootstrap-kubelet.conf文件，依赖于bootstrap-kubelet.conf的用户应切换到包含凭据的kubelet.conf节点上 (#80676, @fabriziopandini)

  • 节点标签beta.kubernetes.io/metadata-proxy-ready，beta.kubernetes.io/metadata-proxy-ready 和 beta.kubernetes.io/kube-proxy-ds-ready 不再添加到新节点上了。

    • ip-mask-agent开始使用标签 node.kubernetes.io/masq-agent-ds-ready 而不是 beta.kubernetes.io/masq-agent-ds-ready 作为其节点选择器。
    • kube-proxy开始使用标签 node.kubernetes.io/kube-proxy-ds-ready 而不是 beta.kubernetes.io/kube-proxy-ds-ready 作为其节点选择器。
    • metadata-proxy开始使用标签 cloud.google.com/metadata-proxy-ready 而不是 beta.kubernetes.io/metadata-proxy-ready 作为其节点选择器。

存储

  • 当为 CSI 驱动启用 PodInfoOnMount 时，Volume 上下文中新的 csi.storage.k8s.io/ephemeral 参数允许驱动程序的 NodePublishVolume 实现根据具体情况确定该 Volume 是临时性的还是正常的持久卷 (#79983, @pohly)
  • 为 VerifyVolumesAreAttached 和 BulkVolume-Verify 添加 CSI Migration Shim (#80443, @davidz627)
  • 新版本将 VolumePVCDataSource（克隆）特性提升到 Beta 版 (#81792, @j-griffith)
  • 将 in-tree 和 CSI Volume 的 Volume Limits 集成到一个 scheduler predicate 中 (#77595, @bertinatto)

弃用和移除

  • API

    • The following APIs are no longer served by default:
    • All resources under apps/v1beta1 and apps/v1beta2 - use apps/v1 instead
    • daemonsets, deployments, replicasets resources under extensions/v1beta1 - use apps/v1 instead
    • networkpolicies resources under extensions/v1beta1 - use networking.k8s.io/v1 instead
    • podsecuritypolicies resources under extensions/v1beta1 - use policy/v1beta1 instead
      Serving these resources can be temporarily re-enabled using the --runtime-config apiserver flag.
    • apps/v1beta1=true
    • apps/v1beta2=true
    • extensions/v1beta1/daemonsets=true,extensions/v1beta1/deployments=true,extensions/v1beta1/replicasets=true,extensions/v1beta1/networkpolicies=true,extensions/v1beta1/podsecuritypolicies=true

    The ability to serve these resources will be completely removed in v1.18. (#70672, @liggitt)

    • Ingress resources will no longer be served from extensions/v1beta1 in v1.20. Migrate use to the networking.k8s.io/v1beta1 API, available since v1.14. Existing persisted data can be retrieved via the networking.k8s.io/v1beta1 API.
    • PriorityClass resources will no longer be served from scheduling.k8s.io/v1beta1 and scheduling.k8s.io/v1alpha1 in v1.17. Migrate to the scheduling.k8s.io/v1 API, available since v1.14. Existing persisted data can be retrieved via the scheduling.k8s.io/v1 API.
    • The export query parameter for list API calls, deprecated since v1.14, will be removed in v1.18.
    • The series.state field in the events.k8s.io/v1beta1 Event API is deprecated and will be removed in v1.18 (#75987, @yastij)
    • The apiextensions.k8s.io/v1beta1 version of CustomResourceDefinition is deprecated and will no longer be served in v1.19. Use apiextensions.k8s.io/v1 instead. (#79604, @liggitt)
    • The admissionregistration.k8s.io/v1beta1 versions of MutatingWebhookConfiguration and ValidatingWebhookConfiguration are deprecated and will no longer be served in v1.19. Use admissionregistration.k8s.io/v1 instead. (#79549, @liggitt)
    • The alpha metadata.initializers field, deprecated in 1.13, has been removed. (#79504, @yue9944882)
    • The deprecated node condition type OutOfDisk has been removed. Use the DiskPressure condition instead. (#72420, @Pingan2017)
    • The metadata.selfLink field is deprecated in individual and list objects. It will no longer be returned starting in v1.20, and the field will be removed entirely in v1.21. (#80978, @wojtek-t)
    • The deprecated cloud providers ovirt, cloudstack and photon have been removed (#72178, @dims)
    • The Cinder and ScaleIO volume providers have been deprecated and will be removed in a future release. (#80099, @dims)
    • The GA PodPriority feature gate is now on by default and cannot be disabled. The feature gate will be removed in v1.18. (#79262, @draveness)
    • Aggregated discovery requests can now timeout. Aggregated API servers must complete discovery calls within 5 seconds (other requests can take longer). Use the feature gate EnableAggregatedDiscoveryTimeout=false to temporarily revert behavior to the previous 30 second timeout if required (the temporary EnableAggregatedDiscoveryTimeout feature gate will be removed in v1.17). (#82146, @deads2k)
    • the scheduler.alpha.kubernetes.io/critical-pod annotation is removed. Pod priority (spec.priorityClassName) should be used instead to mark pods as critical. (#80342, @draveness)
    • the NormalizeScore plugin set is removed from scheduler framework config API. Use ScorePlugin only. (#80930, @liu-cong)

  • Features:

    • The following features are now GA, and the associated feature gates are deprecated and will be removed in v1.17:
    • GCERegionalPersistentDisk (since 1.15.0)
    • CustomResourcePublishOpenAPI
    • CustomResourceSubresources
    • CustomResourceValidation
    • CustomResourceWebhookConversion
    • The feature flags HugePages, VolumeScheduling, CustomPodDNS and PodReadinessGates have been removed (#79307, @draveness)

  • hyperkube

    • the --make-symlinks flag, deprecated in v1.14, has been removed. (#80017, @Pothulapati)

  • kube-apiserver

    • the --basic-auth-file flag and authentication mode is deprecated and will be removed in a future release. It is not recommended for production environments. (#81152, @tedyu)
    • the --cloud-provider-gce-lb-src-cidrs flag has been deprecated. This flag will be removed once the GCE Cloud Provider is removed from kube-apiserver. (#81094, @andrewsykim)
    • the --enable-logs-handler flag and log-serving functionality is deprecated since v1.15, and scheduled to be removed in v1.19. (#77611, @rohitsardesai83)
    • Deprecate the default service IP CIDR. The previous default was 10.0.0.0/24 which will be removed in 6 months/2 releases. Cluster admins must specify their own desired value, by using --service-cluster-ip-range on kube-apiserver. (#81668, @darshanime)

  • kube-proxy

    • the --resource-container flag has been removed from kube-proxy, and specifying it will now cause an error. The behavior is now as if you specified --resource-container="". If you previously specified a non-empty --resource-container, you can no longer do so as of kubernetes 1.16. (#78294, @vllry)

  • kube-scheduler

    • Migrate scheduler to use v1beta1 Event API. any tool targeting scheduler events needs to use v1beta1 Event API (#78447, @yastij)

  • kubeadm

  • The CoreDNS Deployment now checks readiness via the ready plugin.

  • The proxy plugin has been deprecated. The forward plugin is to be used instead.
  • kubernetes plugin removes the resyncperiod option.

  • The upstream option is deprecated and ignored if included. (#82127, @rajansandeep)

  • kubectl

    • kubectl convert, deprecated since v1.14, will be removed in v1.17.
    • The --export flag for the kubectl get command, deprecated since v1.14, will be removed in v1.18.
    • kubectl cp no longer supports copying symbolic links from containers; to support this use case, see kubectl exec --help for examples using tar directly (#82143, @soltysh)
    • Removed deprecated flag --include-uninitialized. (#80337, @draveness)

  • kubelet

    • the --containerized flag was deprecated in 1.14 and has been removed (#80043, @dims)
    • the beta.kubernetes.io/os and beta.kubernetes.io/arch labels, deprecated since v1.14, are targeted for removal in v1.18.
    • cAdvisor json endpoints have been deprecated since 1.15. (#78504, @dashpole)
    • removed the ability to set kubernetes.io- or k8s.io-prefixed labels via --node-labels, other than the specifically allowed labels/prefixes. (#79305, @paivagustavo)

  • client-go

    • Remove DirectCodecFactory (replaced with serializer.WithoutConversionCodecFactory), DirectEncoder (replaced with runtime.WithVersionEncoder) and DirectDecoder (replaced with runtime.WithoutVersionDecoder). (#79263, @draveness)

Metrics变更

添加的度量

  • Added metrics aggregator_openapi_v2_regeneration_count, aggregator_openapi_v2_regeneration_gaugeand apiextension_openapi_v2_regeneration_count counting the triggering APIService and CRDs and the reason (add, update, delete) when kube-apiserver regenerates the OpenAPI spec. (#81786, @sttts)
    Added metrics authentication_attempts that can be used to understand the attempts of authentication. (#81509, @RainbowMango)
    Add a new counter metrics apiserver_admission_webhook_rejection_count with details about the causing for a webhook rejection. (#81399, @roycaihw)
    NFS Drivers are now enabled to collect metrics, StatFS metrics provider is used to collect the metrics. (@brahmaroutu) (#75805, @brahmaroutu)
    Add container_sockets, container_threads, and container_threads_max metrics (#81972, @dashpole)
  • Add container_state label to running_container_count kubelet metrics, to get count of containers based on their state(running/exited/created/unknown) (#81573, @irajdeep)
    Added metric apiserver_watch_events_total that can be used to understand the number of watch events in the system. (#78732, @mborsz)
    Added metric apiserver_watch_events_sizes that can be used to estimate sizes of watch events in the system. (#80477, @mborsz)
  • Added a new Prometheus counter metric sync_proxy_rules_iptables_restore_failures_total for kube-proxy iptables-restore failures (both ipvs and iptables modes) (#81210, @figo)
  • kubelet now exports an kubelet_evictions metric that counts the number of pod evictions carried out by the kubelet to reclaim resources (#81377, @sjenning)

已移除的度量

  • Removed cadvisor metric labels pod_name and container_name to match instrumentation guidelines. Any Prometheus queries that match pod_name and container_name labels (e.g. cadvisor or kubelet probe metrics) must be updated to use pod and container instead. (#80376, @ehashman)

启用/更改的指标

  • kube-controller-manager and cloud-controller-manager metrics are now marked as with the ALPHA stability level. (#81624, @logicalhan)
  • kube-proxy metrics are now marked as with the ALPHA stability level. (#81626, @logicalhan)
  • kube-apiserver metrics are now marked as with the ALPHA stability level. (#81531, @logicalhan)
  • kubelet metrics for /metrics and /metrics/probes are now marked as with the ALPHA stability level. (#81534, @logicalhan)
  • Scheduler metrics are now marked as with the ALPHA stability level. (#81576, @logicalhan)
  • The rejected label in apiserver_admission_webhook_admission_duration_seconds metrices now properly indicates if the request was rejected. (#81399, @roycaihw)
  • Fixed a bug in the CSI metrics that does not return not supported error when a CSI driver does not support metrics. (#79851, @jparklab)
  • Fix disk stats in LXD using ZFS storage pool and CRI-O missing network metris bug (#81972, @dashpole)

特色功能

Beta

  • Promote WatchBookmark feature to beta and enable it by default. With WatchBookmark feature, clients are able to request watch events with BOOKMARK type. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. (#79786, @wojtek-t)
  • The server-side apply feature is now beta (#81956, @apelisse)
  • Server-side apply will now use the openapi provided in the CRD validation field to help figure out how to correctly merge objects and update ownership. (#77354, @jennybuckley)
  • The CustomResourceDefaulting feature is promoted to beta and enabled by default. Defaults may be specified in structural schemas via the apiextensions.k8s.io/v1 API. See https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#specifying-a-structural-schema for details. (#81872, @sttts)
  • Finalizer Protection for Service LoadBalancers is now in beta (enabled by default). This feature ensures the Service resource is not fully deleted until the correlating load balancer resources are deleted. (#81691, @MrHohn)
  • Graduating Windows GMSA support from alpha to beta (#82110, @wk8)

Alpha

  • Introduce a new admission controller for RuntimeClass. Initially, RuntimeClass will be used to apply the pod overhead associated with a given RuntimeClass to the Pod spec if a corresponding RuntimeClassName is specified. PodOverhead is an alpha feature as of Kubernetes 1.16. (#78484, @egernst)
  • Introduction of the pod overhead feature to the scheduler. This functionality is alpha-level as of Kubernetes v1.16, and is only honored by servers that enable the PodOverhead feature.gate. (#78319, @egernst)
  • Ephemeral containers have been added in alpha. These temporary containers can be added to running pods for purposes such as debugging, similar to how kubectl exec runs a process in an existing container. Also like kubectl exec, no resources are reserved for ephemeral containers and they are not restarted when they exit. Note that container namespace targeting is not yet implemented, so process namespace sharing must be enabled to view process from other containers in the pod. (#59484, @verb)

CLI 改进

  • the new flag --endpoint-updates-batch-period in kube-controller-manager can be used to reduce the number of endpoints updates generated by pod changes. (#80509, @mborsz)
  • the kubectl --all-namespaces flag is now honored by kubectl wait (#81468, @ashutoshgngwr)
  • kubectl get -w now takes an --output-watch-events flag to indicate the event type (ADDED, MODIFIED, DELETED) (#72416, @liggitt)
  • Adds Endpoint Slice support for kubectl when discovery API group is enabled. (#81795, @robscott)

Misc

  • Add --shutdown-delay-duration to kube-apiserver in order to delay a graceful shutdown. /healthz will keep returning success during this time and requests are normally served, but /readyz will return failure immediately. This delay can be used to allow the SDN to update iptables on all nodes and stop sending traffic. (#74416, @sttts) Kubeadm now seamlessly migrates the CoreDNS Configuration when upgrading CoreDNS. (#78033, @rajansandeep)
  • Add Endpoint Slice Controller for managing new EndpointSlice resource, disabled by default. (#81048, @robscott)
  • Adds \livez for liveness health checking for kube-apiserver. Using the parameter --maximum-startup-sequence-duration will allow the liveness endpoint to defer boot-sequence failures for the specified duration period. (#81969, @logicalhan)
  • Adds EndpointSlice integration to kube-proxy, can be enabled with EndpointSlice feature gate. (#81430, @robscott)
  • Add status condition to namespace resource (#73405, @wozniakjan)
  • Enhance Azure cloud provider code to support both AAD and ADFS authentication. (#80841, @rjaini)
  • kubeadm: implement support for concurrent add/remove of stacked etcd members (#79677, @neolit123)
  • kubeadm: support any Linux kernel version newer than 3.10 (#81623, @neolit123)
  • Volume expansion is enabled in the default GCE storageclass (#78672, @msau42)
  • kubeadm ClusterConfiguration now supports featureGates: IPv6DualStack: true (#80145, @Arvinderpal)
  • In order to enable dual-stack support within kubeadm and kubernetes components, as part of the init config file, the user should set feature-gate IPv6DualStack=true in the ClusterConfiguration. Additionally, for each worker node, the user should set the feature-gate for kubelet using either nodeRegistration.kubeletExtraArgs or KUBELET_EXTRA_ARGS. (#80531, @Arvinderpal)
  • Add possibility to configure controller manager to use IPv6 dual stack: use --cluster-cidr=",". Notes:
    1. Only the first two CIDRs are used (soft limits for Alpha, might be lifted later on).
    2. Only the "RangeAllocator" (default) is allowed as a value for --cidr-allocator-type. Cloud allocators are not compatible with IPv6 dual stack (#73977, @khenidak)
  • Add scheduling support for RuntimeClasses. RuntimeClasses can now specify nodeSelector constraints & tolerations, which are merged into the PodSpec for pods using that RuntimeClass. (#80825, @tallclair)
  • When specifying --(kube|system)-reserved-cgroup, with --cgroup-driver=systemd, it is now possible to use the fully qualified cgroupfs name (i.e. /test-cgroup.slice). (#78793, @mattjmcnaughton)
  • Adds support for vSphere volumes on Windows (#80911, @gab-satchi)

API变化

  • The MutatingWebhookConfiguration and ValidatingWebhookConfiguration APIs have been promoted to admissionregistration.k8s.io/v1:
    • failurePolicy default changed from Ignore to Fail for v1
    • matchPolicy default changed from Exact to Equivalent for v1
    • timeout default changed from 30s to 10s for v1
    • sideEffects default value is removed, and the field made required, and only None and NoneOnDryRun are permitted for v1
    • admissionReviewVersions default value is removed and the field made required for v1 (supported versions for AdmissionReview are v1 and v1beta1)
    • The name field for specified webhooks must be unique for MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects created via admissionregistration.k8s.io/v1
  • The AdmissionReview API sent to and received from admission webhooks has been promoted to admission.k8s.io/v1. Webhooks can specify a preference for receiving v1 AdmissionReview objects with admissionReviewVersions: ["v1","v1beta1"], and must respond with an API object in the same apiVersion they are sent. When webhooks use admission.k8s.io/v1, the following additional validation is performed on their responses:
    • response.patch and response.patchType are not permitted from validating admission webhooks
    • apiVersion: "admission.k8s.io/v1" is required
    • kind: "AdmissionReview" is required
    • response.uid: "" is required
    • response.patchType: "JSONPatch" is required (if response.patch is set) (#80231, @liggitt)
  • The CustomResourceDefinition API type is promoted to apiextensions.k8s.io/v1 with the following changes:
    • Use of the new default feature in validation schemas is limited to v1
    • spec.scope is no longer defaulted to Namespaced and must be explicitly specified
    • spec.version is removed in v1; use spec.versions instead
    • spec.validation is removed in v1; use spec.versions[*].schema instead
    • spec.subresources is removed in v1; use spec.versions[*].subresources instead
    • spec.additionalPrinterColumns is removed in v1; use spec.versions[*].additionalPrinterColumnsinstead
    • spec.conversion.webhookClientConfig is moved to spec.conversion.webhook.clientConfig in v1
    • spec.conversion.conversionReviewVersions is moved to spec.conversion.webhook.conversionReviewVersions in v1
    • spec.versions[*].schema.openAPIV3Schema is now required when creating v1 CustomResourceDefinitions
    • spec.preserveUnknownFields: true is disallowed when creating v1 CustomResourceDefinitions; it must be specified within schema definitions as x-kubernetes-preserve-unknown-fields: true

  • In additionalPrinterColumns items, the JSONPath field was renamed to jsonPath in v1 (fixes https://github.com/kubernetes/kubernetes/issues/66531) The apiextensions.k8s.io/v1beta1 version of CustomResourceDefinition is deprecated and will no longer be served in v1.19. (#79604, @liggitt)

  • The ConversionReview API sent to and received from custom resource CustomResourceDefinition conversion webhooks has been promoted to apiextensions.k8s.io/v1. CustomResourceDefinition conversion webhooks can now indicate they support receiving and responding with ConversionReview API objects in the apiextensions.k8s.io/v1 version by including v1 in the conversionReviewVersions list in their CustomResourceDefinition. Conversion webhooks must respond with a ConversionReview object in the same apiVersion they receive. apiextensions.k8s.io/v1 ConversionReviewresponses must specify a response.uid that matches the request.uid of the object they were sent. (#81476, @liggitt)

  • Add scheduling support for RuntimeClasses. RuntimeClasses can now specify nodeSelector constraints & tolerations, which are merged into the PodSpec for pods using that RuntimeClass. (#80825, @tallclair)
  • Kubelet should now more reliably report the same primary node IP even if the set of node IPs reported by the CloudProvider changes. (#79391, @danwinship)
  • Omit nil or empty field when calculating container hash value to avoid hash changed. For a new field with a non-nil default value in the container spec, the hash would still get changed. (#57741, @dixudx)
  • Property conditions in apiextensions.v1beta1.CustomResourceDefinitionStatus and apiextensions.v1.CustomResourceDefinitionStatus is now optional instead of required. (#64996, @roycaihw)
  • When the status of a CustomResourceDefinition condition changes, its corresponding lastTransitionTime is now updated. (#69655, @CaoShuFeng)

其他值得注意的变化

API Machinery

  • Remove GetReference() and GetPartialReference() function from pkg/api/ref, as the same function exists also in staging/src/k8s.io/client-go/tools/ref (#80361, @wojtek-t)
  • Verify that CRD default values in OpenAPI specs are pruned, with the exceptions of values under metadata. (#78829, @sttts)
  • Fixes a bug that when there is a "connection refused" error, the reflector's ListAndWatch func will return directly but what expected is that sleep 1 second and rewatch since the specified resourceVersion. (#81634, @likakuli)
  • Resolves an issue serving aggregated APIs backed by services that respond to requests to / with non-2xx HTTP responses (#79895, @deads2k)
  • The CRD handler now properly re-creates stale CR storage to reflect CRD update. (#79114, @roycaihw)
  • Fix CVE-2019-11247: API server allows access to custom resources via wrong scope (#80750, @sttts)
  • Fixed a bug with the openAPI definition for io.k8s.apimachinery.pkg.runtime.RawExtension, which previously required a field raw to be specified (#80773, @jennybuckley)
  • Property conditions in apiextensions.v1beta1.CustomResourceDefinitionStatus and apiextensions.v1.CustomResourceDefinitionStatus is now optional instead of required. (#64996, @roycaihw)
  • Resolves a transient 404 response to custom resource requests during server startup (#81244, @liggitt)
  • OpenAPI now advertises correctly supported patch types for custom resources (#81515, @liggitt)
  • When the status of a CRD Condition changes, it's corresponding LastTransitionTime is now updated. (#69655, @CaoShuFeng)
  • Add metadata.generation=1 to old CustomResources. (#82005, @sttts)
  • Fix a bug in the apiserver that could cause a valid update request to be rejected with a precondition check failure. (#82303, @roycaihw)
  • Fixes regression in logging spurious stack traces when proxied connections are closed by the backend (#82588, @liggitt)
  • RateLimiter add a context-aware method, fix client-go request goruntine backlog in async timeout scene. (#79375, @answer1991)
  • Add a Patch method to ScaleInterface (#80699, @knight42)
  • CRDs under k8s.io and kubernetes.io must have the api-approved.kubernetes.io set to either unapproved.* or a link to the pull request approving the schema. See https://github.com/kubernetes/enhancements/pull/1111 for more details. (#79992, @deads2k)
  • KMS Providers will install a healthz check for the status of kms-plugin in kube-apiservers' encryption config. (#78540, @immutableT)
  • Improves validation errors for custom resources (#81212, @liggitt)
  • Populate object name for admission attributes when CREATE (#53185, @dixudx)
  • Add Overhead field to the PodSpec and RuntimeClass types as part of the Pod Overhead KEP (#76968, @egernst)

Apps

  • Fix a bug that pods not be deleted from unmatched nodes by daemon controller (#78974, @DaiHao)
  • Fix a bug that causes DaemonSet rolling update hang when there exist failed pods. (#78170, @DaiHao)

Auth

  • Service account tokens now include the JWT Key ID field in their header. (#78502, @ahmedtd)
  • The nbf (not before) claim, if present in ID token, is now enforced. (#81413, @anderseknert)

CLI

  • Fix CVE-2019-11249: Incomplete fixes for CVE-2019-1002101 and CVE-2019-11246, kubectl cp potential directory traversal (#80436, @M00nF1sh)
  • Fix the bash completion error with override flags. (#80802, @dtaniwaki)
  • Fix a bug in server printer that could cause kube-apiserver to panic. (#79349, @roycaihw)
  • Fix invalid "time stamp is the future" error when kubectl cp-ing a file (#73982, @tanshanshan)
  • Fix a bug where kubectl set config hangs and uses 100% CPU on some invalid property names (#79000, @pswica)
  • Fix output of kubectl get --watch-only when watching a single resource (#79345, @liggitt)
  • Make kubectl get --ignore-not-found continue processing when encountering error. (#82120, @soltysh)
  • Correct a reference to a not/no longer used kustomize subcommand in the documentation (#82535, @demobox)
  • kubectl could scale custom resource again (#81342, @knight42)
  • Add PodOverhead awareness to kubectl (#81929, @egernst)

云提供商

  • When a load balancer type service is created in a k8s cluster that is backed by Azure Standard Load Balancer, the corresponding load balancer rule added in the Azure Standard Load Balancer would now have the "EnableTcpReset" property set to true. (#80624, @xuto2)
  • Switch to VM Update call in attach/detach disk operation, original CreateOrUpdate call may lead to orphaned VMs or blocked resources (#81208, @andyzhangx)
  • Fix azure disk naming matching issue due to case sensitive comparison (#81720, @andyzhangx)
  • Fix retry issues when the nodes are under deleting on Azure (#80419, @feiskyer)
  • Fix conflicted cache when the requests are canceled by other Azure operations. (#81282, @feiskyer)
  • Fix make azure disk URI as case insensitive (#79020, @andyzhangx)
  • Fix VMSS LoadBalancer backend pools so that the network won't be broken when instances are upgraded to latest model (#81411, @nilo19)
  • Default resourceGroup should be used when the value of annotation azure-load-balancer-resource-group is an empty string. (#79514, @feiskyer)
  • Kubelet could be run with no Azure identity without subscriptionId configured now. A sample cloud provider configure is: '{"vmType": "vmss", "useInstanceMetadata": true}'. (#81500, @feiskyer)
  • Fix public IP not found issues for VMSS nodes (#80703, @feiskyer)
  • Fix Azure client requests stuck issues on http.StatusTooManyRequests (HTTP Code 429). (#81279, @feiskyer)
  • Add a service annotation service.beta.kubernetes.io/azure-pip-name to specify the public IP name for Azure load balancer. (#81213, @nilo19)
  • Optimize EC2 DescribeInstances API calls in aws cloud provider library by querying instance ID instead of EC2 filters when possible (#78140, @zhan849)
  • Creates an annotation service.beta.kubernetes.io/aws-load-balancer-eip-allocations to assign AWS EIP to the newly created Network Load Balancer. Number of allocations and subnets must match. (#69263, @brooksgarrett)
  • Add an azure cloud configuration LoadBalancerName and LoadBalancerResourceGroup to allow the corresponding customizations of azure load balancer. (#81054, @nilo19)

集群生命周期

  • Fix error handling and potential go null pointer exception in kubeadm upgrade diff (#80648, @odinuge)
  • kubeadm: fall back to client version in case of certain HTTP errors (#80024, @RainbowMango)
  • kubeadm: fix a potential panic if kubeadm discovers an invalid, existing kubeconfig file (#79165, @neolit123)
  • kubeadm: treat non-fatal errors as warnings when doing reset (#80862, @drpaneas)
  • kubeadm: prevent PSP blocking of upgrade image prepull by using a non-root user (#77792, @neolit123)
  • kubeadm: fix "certificate-authority" files not being pre-loaded when using file discovery (#80966, @neolit123)
  • Add instruction to setup "Application Default Credentials" to run GCE Windows e2e tests locally. (#81337, @YangLu1031)
  • Fix error in kubeadm join --discovery-file when using discovery files with embedded credentials (#80675, @fabriziopandini)
  • Fix remove the etcd member from the cluster during a kubeadm reset. (#79326, @bradbeam)
  • kubeadm: the permissions of generated CSR files are changed from 0644 to 0600 (#81217, @SataQiu)
  • kubeadm: avoid double deletion of the upgrade prepull DaemonSet (#80798, @xlgao-zju)
  • kubeadm: introduce deterministic ordering for the certificates generation in the phase command kubeadm init phase certs. (#78556, @neolit123)
  • kubeadm: implement retry logic for certain ConfigMap failures when joining nodes (#78915, @ereslibre)
  • kubeadm: use etcd's /health endpoint for a HTTP liveness probe on localhost instead of having a custom health check using etcdctl (#81385, @neolit123)
  • kubeadm reset: unmount directories under /var/lib/kubelet for Linux only (#81494, @Klaven)
  • kubeadm: fix the bug that --cri-socket flag does not work for kubeadm reset (#79498, @SataQiu)
  • kubeadm: produce errors if they occur when resetting cluster status for a control-plane node (#80573, @bart0sh)
  • Fix an error when using external etcd but storing etcd certificates in the same folder with the same name used by kubeadm for local etcd certificates; for an older version of kubeadm, the workaround is to avoid file name used by kubeadm for local etcd. (#80867, @fabriziopandini)
  • kubeadm join fails if file-based discovery is too long, with a default timeout of 5 minutes. (#80804, @olivierlemasle)
  • kubeadm: fixed ignoring errors when pulling control plane images (#80529, @bart0sh)
  • Fix a bug in kube-addon-manager's leader election logic that made all replicas active. (#80575, @mborsz)
  • kubeadm: prevent overriding of certain kubelet security configuration parameters if the user wished to modify them (#81903, @jfbai)
  • kubeadm no longer performs IPVS checks as part of its preflight checks (#81791, @yastij)
  • kubeadm: fix for HTTPProxy check for IPv6 addresses (#82267, @kad)
  • kubeadm: Allow users to skip the kube-proxy init addon phase during init and still be able to join a cluster and perform some other minor operations (but not upgrade). (#82248, @rosti)
  • Mounts /home/kubernetes/bin/nvidia/vulkan/icd.d on the host to /etc/vulkan/icd.d inside containers requesting GPU. (#78868, @chardch)
  • kubeadm: use the --pod-network-cidr flag to init or use the podSubnet field in the kubeadm config to pass a comma separated list of pod CIDRs. (#79033, @Arvinderpal)
  • kubeadm: provide --control-plane-endpoint flag for controlPlaneEndpoint (#79270, @SataQiu)
  • kubeadm: enable secure serving for the kube-scheduler (#80951, @neolit123)
  • kubeadm: print the stack trace of an error for klog level --v>=5 (#80937, @neolit123)
  • Add --kubernetes-version to kubeadm init phase certs ca and kubeadm init phase kubeconfig(#80115, @gyuho)
  • kubeadm: support fetching configuration from the original cluster for upgrade diff (#80025, @SataQiu)
  • When using the conformance test image, a new environment variable E2E_USE_GO_RUNNER will cause the tests to be run with the new golang-based test runner rather than the current bash wrapper. (#79284, @johnSchnake)
  • Implement a new feature that allows applying kustomize patches to static pod manifests generated by kubeadm. (#80905, @fabriziopandini)
  • The 404 request handler for the GCE Ingress load balancer now exports prometheus metrics, including:
    • http_404_request_total (the number of 404 requests handled)
    • http_404_request_duration_ms (the amount of time the server took to respond in ms)
  • Also includes percentile groupings. The directory for the default 404 handler includes instructions on how to enable prometheus for monitoring and setting alerts. (#79106, @vbannai)

仪表盘

  • Kibana has been slightly revamped/improved in the latest version (#80421, @lostick)

    网络

  • Fix a string comparison bug in IPVS graceful termination where UDP real servers are not deleted. (#78999, @andrewsykim)
  • kube-proxy --cleanup will return the correct exit code if the cleanup was successful (#78775, @johscheuer)
  • Fix a bug in the IPVS proxier where virtual servers are not cleaned up even though the corresponding Service object was deleted. (#80942, @gongguan)
  • kube-proxy waits for some duration for the node to be defined. (#77167, @paulsubrata55)
  • Increase log level for graceful termination to v=5 (#80100, @andrewsykim)
  • Reduce kube-proxy CPU usage in IPVS mode when a large number of nodePort services exist. (#79444, @cezarsa)
  • Fix in kube-proxy for SCTP nodeport service which only works for node's InternalIP, but doesn't work for other IPs present in the node when ipvs is enabled. (#81477, @paulsubrata55)
  • Ensure the KUBE-MARK-DROP chain in kube-proxy IPVS mode. The chain is ensured for both IPv4 and IPv6 in dual-stack operation. (#82214, @uablrek)
  • Introduce node.kubernetes.io/exclude-balancer and node.kubernetes.io/exclude-disruption labels in alpha to prevent cluster deployers from being dependent on the optional node-role labels which not all clusters may provide. (#80238, @smarterclayton)
    If targetPort is changed that will process by service controller (#77712, @Sn0rt)

    节点

  • Remove PIDs cgroup controller requirement when related feature gates are disabled (#79073, @rafatio)
  • Fix kubelet NodeLease potential performance issues. Kubelet now will try to update lease using cached one instead of get from API Server every time. (#81174, @answer1991)
  • Passing an invalid policy name in the --cpu-manager-policy flag will now cause the kubelet to fail instead of simply ignoring the flag and running the cpumanager’s default policy instead. (#80294, @klueska)
  • Make node lease renew interval more heuristic based on node-status-update-frequency in kubelet (#80173, @gaorong)
  • Kubelet should now more reliably report the same primary node IP even if the set of node IPs reported by the CloudProvider changes. (#79391, @danwinship)
  • Omit nil or empty field when calculating container hash value to avoid hash changed. For a new field with a non-nil default value in the container spec, the hash would still get changed. (#57741, @dixudx)
    Fix a bug where kubelet would not retry pod sandbox creation when the restart policy of the pod is Never (#79451, @yujuhong)
  • Limit the body length of exec readiness/liveness probes. remote CRIs and Docker shim read a max of 16MB output of which the exec probe itself inspects 10kb. (#82514, @dims)
  • Single static pod files and pod files from http endpoints cannot be larger than 10 MB. HTTP probe payloads are now truncated to 10KB. (#82669, @rphillips)
  • Introduce support for applying pod overhead to pod cgroups, if the PodOverhead feature is enabled. (#79247, @egernst)
  • Node-Problem-Detector v0.7.1 is used on GCI (#80726, @wangzhen127)
  • Node-Problem-Detector v0.7.1 is used for addon daemonset. (#82140, @wangzhen127)
  • Enable cAdvisor ProcessMetrics collecting. (#79002, @jiayingz)
  • kubelet: change node-lease-renew-interval to 0.25 of lease-renew-duration (#80429, @gaorong)
  • Attempt to set the kubelet's hostname & internal IP if --cloud-provider=external and no node addresses exists (#75229, @andrewsykim)

调度

  • Scheduler should terminate when it loses leader lock. (#81306, @ravisantoshgudimetla)
  • If scheduler extender filtered a not found node, current scheduling round for this pod will just be skipped. (#79641, @yqwang-ms)
  • Extender bind should respect IsInterested (#79804, @yqwang-ms)
  • Fix an issue with toleration merging & whitelist checking in the PodTolerationRestriction admission controller. (#81732, @tallclair)
  • Add a helper function to decode scheduler plugin args. (#80696, @hex108)
  • Fix filter plugins are not been called during preemption (#81876, @wgliang)
  • Fix an issue that the correct PluginConfig.Args is not passed to the corresponding PluginFactory in kube-scheduler when multiple PluginConfig items are defined. (#82483, @everpeace)
  • Take the context as the first argument of Schedule. (#82119, @wgliang)
  • Implement post-filter extension point for scheduling framework (#78097, @draveness)
  • Add Bind extension point of the scheduling framework (#78513, @chenchun)
  • Add Filter extension point to the scheduling framework. (#78477, @YoubingLi)
  • Return error when the scoring plugin returns score out of range [0, 100]. (#81015, @draveness)
  • Use a named array instead of a score array in normalizing-score phase. (#80901, @draveness)
  • Updates the requestedToCapacityRatioArguments to add resources parameter that allows the users to specify the resource name along with weights for each resource to score nodes based on the request to capacity ratio. (#77688, @sudeshsh)
  • Add UnschedulableAndUnresolvable status code for scheduling framework (#82034, @alculquicondor)
  • Add normalize plugin extension point for the scheduling framework. (#80383, @liu-cong)
  • Add Bind extension point to the scheduling framework. (#79313, @chenchun)
  • Add Score extension point to the scheduling framework. (#79109, @ahg-g)
  • Add Pre-filter extension point to the scheduling framework. (#78005, @ahg-g)
  • Add support for writing out of tree custom scheduler plugins. (#78162, @hex108)

    存储

  • Fix possible file descriptor leak and closing of dirs in doSafeMakeDir (#79534, @odinuge)
  • Azure disks of shared kind will no longer fail if they do not contain skuname or storageaccounttype. (#80837, @rmweir)
  • Fix CSI plugin supporting raw block that does not need attach mounted failed (#79920, @cwdsuzhou)
  • Reduces GCE PD Node Attach Limits by 1 since the node boot disk is considered an attachable disk (#80923, @davidz627)
  • Remove iSCSI volume storage cleartext secrets in logs (#81215, @zouyee)
  • Fixes validation of VolumeAttachment API objects created with inline volume sources. (#80945, @tedyu)
  • Changes timeout value in csi plugin from 15s to 2min which fixes the timeout issue (#79529, @andyzhangx)
  • Fix kubelet fail to delete orphaned pod directory when the kubelet's pods directory (default is /var/lib/kubelet/pods) symbolically links to another disk device's directory (#79094, @gaorong)

    测试

  • Fix pod list return value of framework.WaitForPodsWithLabelRunningReady (#78687, @pohly)
  • Adding TerminationGracePeriodSeconds to the test framework API (#82170, @vivekbagade)
  • /test/e2e/framework: Adds a flag non-blocking-taints which allows tests to run in environments with tainted nodes. String value should be a comma-separated list. (#81043, @johnSchnake)
  • Move CSI volume expansion to beta. (#81467, @bertinatto)
  • Added E2E tests validating WindowsOptions.RunAsUserName. (#79539, @bclau)
  • framework.ExpectNoError no longer logs the error and instead relies on using the new log.Fail as gomega fail handler. (#80253, @pohly)

Windows

  • On Windows systems, %USERPROFILE% is now preferred over %HOMEDRIVE%\%HOMEPATH% as the home folder if %HOMEDRIVE%\%HOMEPATH% does not contain a .kube\config file, and %USERPROFILE% exists and is writable. (#73923, @liggitt)
  • Add support for AWS EBS on windows (#79552, @wongma7)
  • Support Kubelet plugin watcher on Windows nodes. (#81397, @ddebroy)
    Dependencies

变化的

  • the default Go version was updated to v1.12.9. (#78958, #79966, #81390, #81489)
  • etcd has been updated to v3.3.15 (#82199, @dims)
  • CoreDNS for kubeadm and kube-up has been updated to v1.6.2 (#82127)
  • Cluster Autoscaler has been updated to v1.16.0 (#82501, @losipiuk)
  • fluentd has been updated to v1.5.1 (#79014)
  • fluentd-elasticsearch plugin has been updated to v3.5.3 (#79014)
  • elasticsearch has been updated to v7.1.1 (#79014)
  • kibana has been updated to v7.1.1 (#79014)
  • Azure SDK and go-autorest API versions have been updated (#79574)
  • Azure API versions have been updated (container registry to 2018-09-01, network to 2018-08-01) (#79583)
  • kube-addon-manager has been updated to v9.0.2 (#80861)
  • golang/x/net has been updated to bring in fixes for CVE-2019-9512, CVE-2019-9514 (#81394)
  • GCE windows node image has been updated. (#81106)
  • portworx plugin has been updated on libopenstorage/openstorage to v1.0.0 (#80495)
  • metrics-server has been updated to v0.3.4 (#82322, @olagacek)
  • klog has been updated to v0.4.0 (#81164)

    未变更的

  • The list of validated docker versions remains unchanged.
    • The current list is 1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09. (#72823, #72831)
  • CNI remains unchanged at v0.7.5. (#75455)
  • cri-tools remains unchanged at v1.14.0. (#75658)
  • CAdvisor remains unchanged at v0.33.2. (#76291)
  • event-exporter remains unchanged at v0.2.5. (#77815)
  • ip-masq-agent remains unchanged at v2.4.1. (#77844)
  • k8s-dns-node-cache remains unchanged at v1.15.1 (#76640, @george-angel)
  • CSI remains unchanged at to v1.1.0. (#75391)
  • The dashboard add-on remains unchanged at v1.10.1. (#72495)
  • kube-dns is unchanged at v1.14.13 as of Kubernetes 1.12. (#68900)
  • Influxdb is unchanged at v1.3.3 as of Kubernetes 1.10. (#53319)
  • Grafana is unchanged at v4.4.3 as of Kubernetes 1.10. (#53319)
  • The fluent-plugin-kubernetesmetadatafilter plugin in fluentd-elasticsearch is unchanged at v2.1.6. (#71180)
  • fluentd-gcp is unchanged at v3.2.0 as of Kubernetes 1.13. (#70954)
  • OIDC authentication is unchanged at coreos/go-oidc v2 as of Kubernetes 1.10. (#58544)
  • Calico is unchanged at v3.3.1 as of Kubernetes 1.13. (#70932)
  • GLBC remains unchanged at v1.2.3 as of Kubernetes 1.12. (#66793)
  • Ingress-gce remains unchanged at v1.2.3 as of Kubernetes 1.12. (#66793)

    已移除的

  • Remove deprecated github.com/kardianos/osext dependency (#80142)

详细的go依赖项变更

已经添加的

  • github.com/Azure/go-autorest/autorest/adal: v0.5.0
  • github.com/Azure/go-autorest/autorest/date: v0.1.0
  • github.com/Azure/go-autorest/autorest/mocks: v0.2.0
  • github.com/Azure/go-autorest/autorest/to: v0.2.0
  • github.com/Azure/go-autorest/autorest/validation: v0.1.0
  • github.com/Azure/go-autorest/autorest: v0.9.0
  • github.com/Azure/go-autorest/logger: v0.1.0
  • github.com/Azure/go-autorest/tracing: v0.5.0
  • github.com/armon/consul-api: eb2c6b5
  • github.com/bifurcation/mint: 93c51c6
  • github.com/caddyserver/caddy: v1.0.3
  • github.com/cenkalti/backoff: v2.1.1+incompatible
  • github.com/checkpoint-restore/go-criu: bdb7599
  • github.com/cheekybits/genny: 9127e81
  • github.com/coredns/corefile-migration: v1.0.2
  • github.com/coreos/go-etcd: v2.0.0+incompatible
  • github.com/dustin/go-humanize: v1.0.0
  • github.com/fatih/color: v1.6.0
  • github.com/flynn/go-shlex: 3f9db97
  • github.com/go-acme/lego: v2.5.0+incompatible
  • github.com/go-bindata/go-bindata: v3.1.1+incompatible
  • github.com/go-logr/logr: v0.1.0
  • github.com/google/martian: v2.1.0+incompatible
  • github.com/google/pprof: 3ea8567
  • github.com/google/renameio: v0.1.0
  • github.com/googleapis/gax-go/v2: v2.0.4
  • github.com/hashicorp/go-syslog: v1.0.0
  • github.com/jimstudt/http-authentication: 3eca13d
  • github.com/kisielk/errcheck: v1.2.0
  • github.com/kisielk/gotool: v1.0.0
  • github.com/klauspost/cpuid: v1.2.0
  • github.com/kr/pty: v1.1.5
  • github.com/kylelemons/godebug: d65d576
  • github.com/lucas-clemente/aes12: cd47fb3
  • github.com/lucas-clemente/quic-clients: v0.1.0
  • github.com/lucas-clemente/quic-go-certificates: d2f8652
  • github.com/lucas-clemente/quic-go: v0.10.2
  • github.com/marten-seemann/qtls: v0.2.3
  • github.com/mattn/go-colorable: v0.0.9
  • github.com/mattn/go-isatty: v0.0.3
  • github.com/mholt/certmagic: 6a42ef9
  • github.com/mitchellh/go-homedir: v1.1.0
  • github.com/naoina/go-stringutil: v0.1.0
  • github.com/naoina/toml: v0.1.1
  • github.com/rogpeppe/go-internal: v1.3.0
  • github.com/thecodeteam/goscaleio: v0.1.0
  • github.com/ugorji/go/codec: d75b2dc
  • github.com/xordataexchange/crypt: b2862e3
  • go.opencensus.io: v0.21.0
  • golang.org/x/mod: 4bf6d31
  • gopkg.in/airbrake/gobrake.v2: v2.0.9
  • gopkg.in/errgo.v2: v2.1.0
  • gopkg.in/gemnasium/logrus-airbrake-hook.v2: v2.1.2
  • gopkg.in/mcuadros/go-syslog.v2: v2.2.1
  • gotest.tools/gotestsum: v0.3.5
  • honnef.co/go/tools: v0.0.1-2019.2.2

已更改的

  • cloud.google.com/go: v0.34.0 → v0.38.0
  • github.com/Azure/azure-sdk-for-go: v21.4.0+incompatible → v32.5.0+incompatible
  • github.com/BurntSushi/toml: v0.3.0 → v0.3.1
  • github.com/GoogleCloudPlatform/k8s-cloud-provider: f8e9959 → 27a4ced
  • github.com/PuerkitoBio/purell: v1.1.0 → v1.1.1
  • github.com/asaskevich/govalidator: f9ffefc → f61b66f
  • github.com/client9/misspell: 9ce5d97 → v0.3.4
  • github.com/containernetworking/cni: v0.6.0 → v0.7.1
  • github.com/coreos/etcd: v3.3.13+incompatible → v3.3.15+incompatible
  • github.com/coreos/go-oidc: 065b426 → v2.1.0+incompatible
  • github.com/coreos/go-semver: e214231 → v0.3.0
  • github.com/cpuguy83/go-md2man: v1.0.4 → v1.0.10
  • github.com/cyphar/filepath-securejoin: ae69057 → v0.2.2
  • github.com/dgrijalva/jwt-go: 01aeca5 → v3.2.0+incompatible
  • github.com/docker/distribution: edc3ab2 → v2.7.1+incompatible
  • github.com/emicklei/go-restful: ff4f55a → v2.9.5+incompatible
  • github.com/evanphx/json-patch: 5858425 → v4.2.0+incompatible
  • github.com/fatih/camelcase: f6a740d → v1.0.0
  • github.com/go-openapi/analysis: v0.17.2 → v0.19.2
  • github.com/go-openapi/errors: v0.17.2 → v0.19.2
  • github.com/go-openapi/jsonpointer: v0.19.0 → v0.19.2
  • github.com/go-openapi/jsonreference: v0.19.0 → v0.19.2
  • github.com/go-openapi/loads: v0.17.2 → v0.19.2
  • github.com/go-openapi/runtime: v0.17.2 → v0.19.0
  • github.com/go-openapi/spec: v0.17.2 → v0.19.2
  • github.com/go-openapi/strfmt: v0.17.0 → v0.19.0
  • github.com/go-openapi/swag: v0.17.2 → v0.19.2
  • github.com/go-openapi/validate: v0.18.0 → v0.19.2
  • github.com/godbus/dbus: c7fdd8b → v4.1.0+incompatible
  • github.com/gogo/protobuf: 342cbe0 → 65acae2
  • github.com/golang/mock: bd3c8e8 → v1.2.0
  • github.com/golang/protobuf: v1.2.0 → v1.3.1
  • github.com/google/btree: 7d79101 → 4030bb1
  • github.com/google/cadvisor: 9db8c7d → v0.34.0
  • github.com/google/gofuzz: 24818f7 → v1.0.0
  • github.com/google/uuid: v1.0.0 → v1.1.1
  • github.com/gophercloud/gophercloud: c818fa6 → v0.1.0
  • github.com/gorilla/websocket: 4201258 → v1.4.0
  • github.com/grpc-ecosystem/go-grpc-prometheus: 2500245 → v1.2.0
  • github.com/hashicorp/golang-lru: v0.5.0 → v0.5.1
  • github.com/hashicorp/hcl: d8c773c → v1.0.0
  • github.com/heketi/heketi: 558b292 → v9.0.0+incompatible
  • github.com/jonboulle/clockwork: 72f9bd7 → v0.1.0
  • github.com/json-iterator/go: ab8a2e0 → v1.1.7
  • github.com/kr/pretty: f31442d → v0.1.0
  • github.com/kr/text: 6807e77 → v0.1.0
  • github.com/libopenstorage/openstorage: 093a0c3 → v1.0.0
  • github.com/magiconair/properties: 61b492c → v1.8.1
  • github.com/mailru/easyjson: 60711f1 → 94de47d
  • github.com/mattn/go-shellwords: f8471b0 → v1.0.5
  • github.com/miekg/dns: 5d001d0 → v1.1.4
  • github.com/mistifyio/go-zfs: 1b4ae6f → v2.1.1+incompatible
  • github.com/mitchellh/go-wordwrap: ad45545 → v1.0.0
  • github.com/mvdan/xurls: 1b768d7 → v1.1.0
  • github.com/onsi/ginkgo: v1.6.0 → v1.8.0
  • github.com/onsi/gomega: 5533ce8 → v1.5.0
  • github.com/opencontainers/go-digest: a6d0ee4 → v1.0.0-rc1
  • github.com/opencontainers/image-spec: 372ad78 → v1.0.1
  • github.com/opencontainers/runc: f000fe1 → 6cc5158
  • github.com/opencontainers/selinux: 4a2974b → v1.2.2
  • github.com/robfig/cron: df38d32 → v1.1.0
  • github.com/russross/blackfriday: 300106c → v1.5.2
  • github.com/seccomp/libseccomp-golang: 1b506fc → v0.9.1
  • github.com/sirupsen/logrus: v1.2.0 → v1.4.2
  • github.com/spf13/afero: b28a7ef → v1.2.2
  • github.com/spf13/cast: e31f36f → v1.3.0
  • github.com/spf13/cobra: c439c4f → v0.0.5
  • github.com/spf13/jwalterweatherman: 33c24e7 → v1.1.0
  • github.com/spf13/pflag: v1.0.1 → v1.0.3
  • github.com/spf13/viper: 7fb2782 → v1.3.2
  • github.com/stretchr/objx: v0.1.1 → v0.2.0
  • github.com/stretchr/testify: v1.2.2 → v1.3.0
  • golang.org/x/net: 65e2d4e → cdfb69a
  • golang.org/x/tools: aa82965 → 6e04913
  • google.golang.org/api: 583d854 → 5213b80
  • google.golang.org/genproto: 09f6ed2 → 54afdca
  • google.golang.org/grpc: v1.13.0 → v1.23.0
  • gopkg.in/check.v1: 20d25e2 → 788fd78
  • gopkg.in/natefinch/lumberjack.v2: 20b71e5 → v2.0.0
  • gopkg.in/square/go-jose.v2: 89060de → v2.2.2
  • gopkg.in/yaml.v2: v2.2.1 → v2.2.2
  • k8s.io/gengo: f8a0810 → 26a6646
  • k8s.io/klog: v0.3.1 → v0.4.0
  • k8s.io/kube-openapi: b3a7cee → 743ec37
  • k8s.io/utils: c2654d5 → 581e001
  • sigs.k8s.io/structured-merge-diff: e85c7b2 → 6149e45

已移除的

  • github.com/Azure/go-autorest: v11.1.2+incompatible
  • github.com/codedellemc/goscaleio: 20e2ce2
  • github.com/d2g/dhcp4: a1d1b6c
  • github.com/d2g/dhcp4client: 6e570ed
  • github.com/jteeuwen/go-bindata: a0ff256
  • github.com/kardianos/osext: 8fef92e
  • github.com/kr/fs: 2788f0d
  • github.com/marstr/guid: 8bdf7d1
  • github.com/mholt/caddy: 2de4950
  • github.com/natefinch/lumberjack: v2.0.0+incompatible
  • github.com/pkg/sftp: 4d0e916
  • github.com/shurcooL/sanitizedanchorname: 10ef21a
  • github.com/sigma/go-inotify: c87b6cf
  • github.com/vmware/photon-controller-go-sdk: 4a435da
  • github.com/xanzy/go-cloudstack: 1e2cbf6
    gopkg.in/yaml.v1: 9f9df34
    v1.16.0-rc.2
    v1.16.0-rc.1
    v1.16.0-beta.2
    v1.16.0-beta.1
    v1.16.0-alpha.3
    v1.16.0-alpha.2
    v1.16.0-alpha.1
kubernetes k8s
更新于 2019-09-24
半兽人

查看kubernetes更多相关的文章或提一个关于kubernetes的问题，也可以与我们一起分享文章。
提问

找不到想要的答案？提一个您自己的问题。