本文将介绍如何通过创建CustomResourceDefinition将自定义资源安装到Kubernetes API中。

在开始之前

验证你的版本, 输入 kubectl version.

• 如果你的Kubernetes集群是1.16.0或更高的主版本才能使用apiextensions.k8s.io/v1，否则如果是1.7.0或更高的只能使用apiextensions.k8s.io/v1beta1。

• 学习完整的自定义资源介绍

创建一个CustomResourceDefinition

创建新的CustomResourceDefinition（CRD）时，Kubernetes API Server为您指定的每个版本创建一个新的RESTful资源路径。在CRD的作用域字段中指定是命名空间，还是集群作用域。与现有的内置对象一样，删除命名空间也会删除该命名空间中的所有自定义对象。CustomResourceDefinitions本身是非命名空间的，但可用于所有的命名空间。

例如，将以下CustomResourceDefinition保存到resourcedefinition.yaml中：

apiextensions.k8s.io/v1（1.16.0或更高）

apiextensions.k8s.io/v1
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct

apiextensions.k8s.io/v1beta1（1.7.0或更高）

# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct
  preserveUnknownFields: false
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            cronSpec:
              type: string
            image:
              type: string
            replicas:
              type: integer

创建它:

kubectl apply -f resourcedefinition.yaml

然后将在以下位置创建一个新的命名空间RESTful API endpoint

/apis/stable.example.com/v1/namespaces/*/crontabs/...

然后，你可以使用该endpoint URL创建和管理自定义对象。这些对象的kind(类型)是上面创建的CustomResourceDefinition对象spec中的CronTab。

创建endpoint可能需要几秒钟。你可以观察你的CustomResourceDefinition的Established条件是否为真，或者观察你的资源的API服务器的发现信息是否有了。

创建自定义对象

创建CustomResourceDefinition对象后，你可以创建自定义对象。 自定义对象可以包含自定义字段。这些字段可以包含任意JSON。 在下面的示例中，在kind(类型)为CronTab的自定义对象中设置了cronSpec和image自定义字段。CronTab类型来自你刚刚上面创建的CustomResourceDefinition对象的spec。

将以下YAML保存到my-crontab.yaml中：

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

创建它:

kubectl apply -f my-crontab.yaml

然后，就可以使用kubectl管理CronTab对象。 例如：

kubectl get crontab

应该打印如下列表：

NAME                 AGE
my-new-cron-object   6s

使用kubectl时，资源名称不区分大小写，你可以使用CRD中定义的单数或复数形式以及任何简称。

你还可以查看原始的YAML数据：

kubectl get ct -o yaml

你可以看到它包含用于创建它的Yaml的自定义cronSpec和image字段：

apiVersion: v1
kind: List
items:
- apiVersion: stable.example.com/v1
  kind: CronTab
  metadata:
    creationTimestamp: 2017-05-31T12:56:35Z
    generation: 1
    name: my-new-cron-object
    namespace: default
    resourceVersion: "285"
    uid: 9423255b-4600-11e7-af6a-28d2447dc82b
  spec:
    cronSpec: '* * * * */5'
    image: my-awesome-cron-image
metadata:
  resourceVersion: ""

删除一个CustomResourceDefinition

删除CustomResourceDefinition时，服务器将卸载RESTful API端点并删除其中存储的所有自定义对象。

kubectl delete -f resourcedefinition.yaml
kubectl get crontabs

Error from server (NotFound): Unable to list {"stable.example.com" "v1" "crontabs"}: the server could not find the requested resource (get crontabs.stable.example.com)

如果你之后重新创建相同的CustomResourceDefinition，它也是空。

指定结构模式（Specifying a structural schema）

FEATURE STATE : Kubernetes v1.16 [stable]

传统上，CustomResources存储任意JSON（除了apiVersion，kind和metadata外，还包括API服务隐式验证的元数据）。使用OpenAPI v3.0验证，可以指定一个模式，在创建和更新过程中进行验证，下面比较一下这种模式的细节和限制。

在apiextensions.k8s.io/v1中，对于CustomResourceDefinitions来说，结构模式的定义是强制性的，而在v1beta1中，这仍然是可选的。

结构模式是一个OpenAPI v3.0的验证模式：

1. 为root、对象节点的每个指定字段（通过OpenAPI中的properties或additionalProperties）和数组节点的每个项（通过OpenAPI中的 items）指定一个非空类型（通过OpenAPI中的type），但以下情况除外。
   • 具有x-kubernetes-int-or-string: true的节点
   • 具有x-kubernetes-preserve-unknown-fields: true的节点

2. 对于对象中的每一个字段和数组中的每一个项目，如果在allOf、anyOf、oneOf或不在其中任何一个范围内指定，模式也会指定这些逻辑连接符之外的字段/项目（对比例1和例2）。

3. 在allOf、anyOf、oneOf或不设置description、type、default、additionalProperties、nullable，但x-kubernetes-int-or-string:true（见下文）的两个模式除外。

4. 如果指定了metadata，那么只允许对metadata.name和metadata.generateName进行限制。

非结构性示例1:

allOf:
- properties:
    foo:
      ...

与规则2冲突。以下是正确的：

properties:
  foo:
    ...
allOf:
- properties:
    foo:
      ...

非结构示例2：

allOf:
- items:
    properties:
      foo:
        ...

与规则2冲突。以下是正确的：

items:
  properties:
    foo:
      ...
allOf:
- items:
    properties:
      foo:
        ...

非结构示例3：

properties:
  foo:
    pattern: "abc"
  metadata:
    type: object
    properties:
      name:
        type: string
        pattern: "^a"
      finalizers:
        type: array
        items:
          type: string
          pattern: "my-finalizer"
anyOf:
- properties:
    bar:
      type: integer
      minimum: 42
  required: ["bar"]
  description: "foo bar object"

因为违反了以下规定，因此不是结构化架构：

• root类型缺失（规则1）。
• 缺少foo类型（规则1）。
• "anyOf "里面的 "bar "在外面没有指定(规则2)。
• bar'的类型在anyOf'之内(规则3)；
• description设置在`anyOf'范围内(规则3)；
• metadata.finalizer可能不受限制（规则4）。

相反，以下对应的模式是结构化的：

type: object
description: "foo bar object"
properties:
  foo:
    type: string
    pattern: "abc"
  bar:
    type: integer
  metadata:
    type: object
    properties:
      name:
        type: string
        pattern: "^a"
anyOf:
- properties:
    bar:
      minimum: 42
  required: ["bar"]

在CustomResourceDefinition中的NonStructural条件中报告违反结构模式规则的情况。

结构模式是apiextensions.k8s.io/v1的要求，并禁用了apiextensions.k8s.io/v1beta1的以下功能：

• Validation Schema Publishing（验证模式发布）
• Webhook Conversion（Webhook转换）
• Pruning

修剪与保留未知字段

FEATURE STATE : Kubernetes v1.16 [stable]

CustomResourceDefinitions传统上将任何（可能是验证过的）JSON按原样存储在etcd中，这意味着未指定的字段（如果有OpenAPI v3.0验证模式的话）被持久化。这意味着未指定的字段（如果有OpenAPI v3.0验证模式的话）将被持久化。这与原生的Kubernetes资源（如pod）相反，在pod中，未知字段在被持久化到etcd之前会被丢弃。我们称之为未知字段的 “修剪（pruning）”。

apiextensions.k8s.io/v1

对于在apiextensions.k8s.io/v1中创建的CustomResourceDefinitions，需要结构性的OpenAPI v3验证模式，并且pruning(修剪)是启用的，不能禁用（注意，从apiextensions.k8s.io/v1beta1转换到apiextensions.k8s.io/v1的CRD可能缺乏结构性模式，并且spec.preserveUnknownFields可能为true）。

apiextensions.k8s.io/v1beta1

对于在apiextensions.k8s.io/v1beta1中创建的CustomResourceDefinitions，如果在 CustomResourceDefinition 中定义了结构性的OpenAPI v3 验证模式（在apiextensions.k8s.io/v1beta1 的全局 spec.validation.openAPIV3Schema 中或为每个版本定义），可以通过将 spec.preservationUnknownFields 设置为 false 来启用pruning(修剪)。

如果启用了pruning(修剪)，那么在创建和更新时，CustomResources中未指定的字段将被删除。

比较一下上面的CustomResourceDefinitioncrontabs.stable.example.com，它已经启用了修剪功能（在apiextensions.k8s.io/v1和apiextensions.k8s.io/v1beta1中）。因此，将以下YAML保存到my-crontab.yaml。

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
  someRandomField: 42

然后创建它:

kubectl create --validate=false -f my-crontab.yaml -o yaml

应该得到以下输出:

apiVersion: stable.example.com/v1
kind: CronTab
metadata:
  creationTimestamp: 2017-05-31T12:56:35Z
  generation: 1
  name: my-new-cron-object
  namespace: default
  resourceVersion: "285"
  uid: 9423255b-4600-11e7-af6a-28d2447dc82b
spec:
  cronSpec: '* * * * */5'
  image: my-awesome-cron-image

删除了someRandomField字段。

注意，kubectl create可以使用--validate=false来跳过客户端验证。由于OpenAPI验证模式也已发布到kubectl，因此它还将检查未知字段并在这些对象被发送到API服务器之前就拒绝这些对象。

Controlling pruning

If pruning is enabled (enforced in apiextensions.k8s.io/v1, or as opt-in via spec.preserveUnknownField: false in apiextensions.k8s.io/v1beta1) in the CustomResourceDefinition, all unspecified fields in custom resources of that type and in all versions are pruned. It is possible though to opt-out of that for JSON sub-trees via x-kubernetes-preserve-unknown-fields: true in the structural OpenAPI v3 validation schema:

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true

The field json can store any JSON value, without anything being pruned.

It is possible to partially specify the permitted JSON, e.g.:

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true
    type: object
    description: this is arbitrary JSON

With this only object type values are allowed.

Pruning is enabled again for each specified property (or additionalProperties):

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true
    type: object
    properties:
      spec:
        type: object
        properties:
          foo:
            type: string
          bar:
            type: string

With this, the value:

json:
  spec:
    foo: abc
    bar: def
    something: x
  status:
    something: x

is pruned to:

json:
  spec:
    foo: abc
    bar: def
  status:
    something: x

This means that the something field in the specified spec object is pruned, but everything outside is not.

IntOrString

Nodes in a schema with x-kubernetes-int-or-string: true are excluded from rule 1, such that the following is structural:

type: object
properties:
  foo:
    x-kubernetes-int-or-string: true

Also those nodes are partially excluded from rule 3 in the sense that the following two patterns are allowed (exactly those, without variations in order to additional fields):

x-kubernetes-int-or-string: true
anyOf:
- type: integer
- type: string
...

and

x-kubernetes-int-or-string: true
allOf:
- anyOf:
  - type: integer
  - type: string
- ... # zero or more
...
With one of thos

e specification, both an integer and a string validate.

In Validation Schema Publishing, x-kubernetes-int-or-string: true is unfolded to one of the two patterns shown above.

RawExtension

RawExtensions (as in runtime.RawExtension defined in k8s.io/apimachinery) holds complete Kubernetes objects, i.e. with apiVersion and kind fields.

It is possible to specify those embedded objects (both completely without constraints or partially specified) by setting x-kubernetes-embedded-resource: true. For example:

type: object
properties:
  foo:
    x-kubernetes-embedded-resource: true
    x-kubernetes-preserve-unknown-fields: true

Here, the field foo holds a complete object, e.g.:

foo:
  apiVersion: v1
  kind: Pod
  spec:
    ...

Because x-kubernetes-preserve-unknown-fields: true is specified alongside, nothing is pruned. The use of x-kubernetes-preserve-unknown-fields: true is optional though.

With x-kubernetes-embedded-resource: true, the apiVersion, kind and metadata are implicitly specified and validated.

Serving multiple versions of a CRD

See Custom resource definition versioning for more information about serving multiple versions of your CustomResourceDefinition and migrating your objects from one version to another.

Advanced topics

Finalizers

Finalizers allow controllers to implement asynchronous pre-delete hooks. Custom objects support finalizers just like built-in objects.

You can add a finalizer to a custom object like this:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  finalizers:
  - finalizer.stable.example.com

Finalizers are arbitrary string values, that when present ensure that a hard delete of a resource is not possible while they exist.

The first delete request on an object with finalizers sets a value for the metadata.deletionTimestamp field but does not delete it. Once this value is set, entries in the finalizer list can only be removed.

When the metadata.deletionTimestamp field is set, controllers watching the object execute any finalizers they handle, by polling update requests for that object. When all finalizers have been executed, the resource is deleted.

The value of metadata.deletionGracePeriodSeconds controls the interval between polling updates.

It is the responsibility of each controller to remove its finalizer from the list.

Kubernetes only finally deletes the object if the list of finalizers is empty, meaning all finalizers have been executed.

Validation

FEATURE STATE: Kubernetes v1.16 [stable]
Validation of custom objects is possible via OpenAPI v3 schemas or validatingadmissionwebhook. In apiextensions.k8s.io/v1 schemas are required, in apiextensions.k8s.io/v1beta1 they are optional.

Additionally, the following restrictions are applied to the schema:

• These fields cannot be set:
  • definitions,
  • dependencies,
  • deprecated,
  • discriminator,
  • id,
  • patternProperties,
  • readOnly,
  • writeOnly,
  • xml,
  • $ref.
• The field uniqueItems cannot be set to true.
• The field additionalProperties cannot be set to false.
• The field additionalProperties is mutually exclusive with properties.

These fields can only be set with specific features enabled:

• default: can be set for apiextensions.k8s.io/v1 CustomResourceDefinitions. Defaulting is in GA since 1.17 (beta since 1.16 with the CustomResourceDefaulting feature gate to be enabled, which is the case automatically for many clusters for beta features). Compare Validation Schema Defaulting.

Note: Compare with structural schemas for further restriction required for certain CustomResourceDefinition features.

The schema is defined in the CustomResourceDefinition. In the following example, the CustomResourceDefinition applies the following validations on the custom object:

• spec.cronSpec must be a string and must be of the form described by the regular expression.
• spec.replicas must be an integer and must have a minimum value of 1 and a maximum value of 10.

Save the CustomResourceDefinition to resourcedefinition.yaml:

apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        # openAPIV3Schema is the schema for validating custom objects.
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                  pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
  version: v1
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
  validation:
   # openAPIV3Schema is the schema for validating custom objects.
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            cronSpec:
              type: string
              pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
            replicas:
              type: integer
              minimum: 1
              maximum: 10

And create it:

kubectl apply -f resourcedefinition.yaml
A request to create a custom object of kind CronTab will be rejected if there are invalid values in its fields. In the following example, the custom object contains fields with invalid values:

• spec.cronSpec does not match the regular expression.
• spec.replicas is greater than 10.

If you save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * *"
  image: my-awesome-cron-image
  replicas: 15

and create it:

kubectl apply -f my-crontab.yaml

you will get an error:

The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
validation failure list:
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
spec.replicas in body should be less than or equal to 10

If the fields contain valid values, the object creation request is accepted.

Save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
  replicas: 5

And create it:

kubectl apply -f my-crontab.yaml
crontab "my-new-cron-object" created

Defaulting

FEATURE STATE: Kubernetes v1.17 [stable]
Note: To use defaulting, your CustomResourceDefinition must use API version apiextensions.k8s.io/v1.
Defaulting allows to specify default values in the OpenAPI v3 validation schema:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        # openAPIV3Schema is the schema for validating custom objects.
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                  pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
                  default: "5 0 * * *"
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
                  default: 1
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

With this both cronSpec and replicas are defaulted:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  image: my-awesome-cron-image
leads to

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "5 0 * * *"
  image: my-awesome-cron-image
  replicas: 1
Defaulting happens on the object

in the request to the API server using the request version defaults,
when reading from etcd using the storage version defaults,
after mutating admission plugins with non-empty patches using the admission webhook object version defaults.
Defaults applied when reading data from etcd are not automatically written back to etcd. An update request via the API is required to persist those defaults back into etcd.

Default values must be pruned (with the exception of defaults for metadata fields) and must validate against a provided schema.

Default values for metadata fields of x-kubernetes-embedded-resources: true nodes (or parts of a default value covering metadata) are not pruned during CustomResourceDefinition creation, but through the pruning step during handling of requests.

Publish Validation Schema in OpenAPI v2
FEATURE STATE: Kubernetes v1.16 [stable]
Note: OpenAPI v2 Publishing is available as beta since 1.15, and as alpha since 1.14. The CustomResourcePublishOpenAPI feature must be enabled, which is the case automatically for many clusters for beta features. Please refer to the feature gate documentation for more information.
With the OpenAPI v2 Publishing feature enabled, CustomResourceDefinition OpenAPI v3 validation schemas which are structural and enable pruning (opt-in in v1beta1, enabled by default in v1) are published as part of the OpenAPI v2 spec from Kubernetes API server.

kubectl consumes the published schema to perform client-side validation (kubectl create and kubectl apply), schema explanation (kubectl explain) on custom resources. The published schema can be consumed for other purposes as well, like client generation or documentation.

The OpenAPI v3 validation schema is converted to OpenAPI v2 schema, and show up in definitions and paths fields in the OpenAPI v2 spec. The following modifications are applied during the conversion to keep backwards compatibility with kubectl in previous 1.13 version. These modifications prevent kubectl from being over-strict and rejecting valid OpenAPI schemas that it doesn’t understand. The conversion won’t modify the validation schema defined in CRD, and therefore won’t affect validation in the API server.

The following fields are removed as they aren’t supported by OpenAPI v2 (in future versions OpenAPI v3 will be used without these restrictions)
The fields allOf, anyOf, oneOf and not are removed
If nullable: true is set, we drop type, nullable, items and properties because OpenAPI v2 is not able to express nullable. To avoid kubectl to reject good objects, this is necessary.
Additional printer columns
Starting with Kubernetes 1.11, kubectl uses server-side printing. The server decides which columns are shown by the kubectl get command. You can customize these columns using a CustomResourceDefinition. The following example adds the Spec, Replicas, and Age columns.

Save the CustomResourceDefinition to resourcedefinition.yaml:

apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              cronSpec:
                type: string
              image:
                type: string
              replicas:
                type: integer
    additionalPrinterColumns:
    - name: Spec
      type: string
      description: The cron spec defining the interval a CronJob is run
      jsonPath: .spec.cronSpec
    - name: Replicas
      type: integer
      description: The number of jobs launched by the CronJob
      jsonPath: .spec.replicas
    - name: Age
      type: date
      jsonPath: .metadata.creationTimestamp

# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  version: v1
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            cronSpec:
              type: string
            image:
              type: string
            replicas:
              type: integer
  additionalPrinterColumns:
  - name: Spec
    type: string
    description: The cron spec defining the interval a CronJob is run
    JSONPath: .spec.cronSpec
  - name: Replicas
    type: integer
    description: The number of jobs launched by the CronJob
    JSONPath: .spec.replicas
  - name: Age
    type: date
    JSONPath: .metadata.creationTimestamp

Create the CustomResourceDefinition:

kubectl apply -f resourcedefinition.yaml

Create an instance using the my-crontab.yaml from the previous section.

Invoke the server-side printing:

kubectl get crontab my-new-cron-object
Notice the NAME, SPEC, REPLICAS, and AGE columns in the output:

NAME                 SPEC        REPLICAS   AGE
my-new-cron-object   * * * * *   1          7s

Note: The NAME column is implicit and does not need to be defined in the CustomResourceDefinition.
Priority
Each column includes a priority field for each column. Currently, the priority differentiates between columns shown in standard view or wide view (using the -o wide flag).

Columns with priority 0 are shown in standard view.
Columns with priority greater than 0 are shown only in wide view.
Type
A column’s type field can be any of the following (compare OpenAPI v3 data types):

integer – non-floating-point numbers
number – floating point numbers
string – strings
boolean – true or false
date – rendered differentially as time since this timestamp.
If the value inside a CustomResource does not match the type specified for the column, the value is omitted. Use CustomResource validation to ensure that the value types are correct.

Format
A column’s format field can be any of the following:

int32
int64
float
double
byte
date
date-time
password
The column’s format controls the style used when kubectl prints the value.

Subresources
FEATURE STATE: Kubernetes v1.16 [stable]
Custom resources support /status and /scale subresources.

You can disable this feature using the CustomResourceSubresources feature gate on the kube-apiserver:

--feature-gates=CustomResourceSubresources=false
The status and scale subresources can be optionally enabled by defining them in the CustomResourceDefinition.

Status subresource
When the status subresource is enabled, the /status subresource for the custom resource is exposed.

The status and the spec stanzas are represented by the .status and .spec JSONPaths respectively inside of a custom resource.

PUT requests to the /status subresource take a custom resource object and ignore changes to anything except the status stanza.

PUT requests to the /status subresource only validate the status stanza of the custom resource.

PUT/POST/PATCH requests to the custom resource ignore changes to the status stanza.

The .metadata.generation value is incremented for all changes, except for changes to .metadata or .status.

Only the following constructs are allowed at the root of the CRD OpenAPI validation schema:

Description
Example
ExclusiveMaximum
ExclusiveMinimum
ExternalDocs
Format
Items
Maximum
MaxItems
MaxLength
Minimum
MinItems
MinLength
MultipleOf
Pattern
Properties
Required
Title
Type
UniqueItems
Scale subresource
When the scale subresource is enabled, the /scale subresource for the custom resource is exposed. The autoscaling/v1.Scale object is sent as the payload for /scale.

To enable the scale subresource, the following values are defined in the CustomResourceDefinition.

SpecReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.

It is a required value.
Only JSONPaths under .spec and with the dot notation are allowed.
If there is no value under the SpecReplicasPath in the custom resource, the /scale subresource will return an error on GET.
StatusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.

It is a required value.
Only JSONPaths under .status and with the dot notation are allowed.
If there is no value under the StatusReplicasPath in the custom resource, the status replica value in the /scale subresource will default to 0.
LabelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.

It is an optional value.
It must be set to work with HPA.
Only JSONPaths under .status or .spec and with the dot notation are allowed.
If there is no value under the LabelSelectorPath in the custom resource, the status selector value in the /scale subresource will default to the empty string.
The field pointed by this JSON path must be a string field (not a complex selector struct) which contains a serialized label selector in string form.
In the following example, both status and scale subresources are enabled.

Save the CustomResourceDefinition to resourcedefinition.yaml:

apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:

- name: v1
  served: true
  storage: true
  schema:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            cronSpec:
              type: string
            image:
              type: string
            replicas:
              type: integer
        status:
          type: object
          properties:
            replicas:
              type: integer
            labelSelector:
              type: string
  # subresources describes the subresources for custom resources.
  subresources:
    # status enables the status subresource.
    status: {}
    # scale enables the scale subresource.
    scale:
      # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
      specReplicasPath: .spec.replicas
      # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
      statusReplicasPath: .status.replicas
      # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
      labelSelectorPath: .status.labelSelector

scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:

- ct

Deprecated in v1.16 in favor of apiextensions.k8s.io/v1

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:

- name: v1
  served: true
  storage: true

scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:

- ct

validation:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
status:
type: object
properties:
replicas:
type: integer
labelSelector:
type: string

subresources describes the subresources for custom resources.

subresources:

# status enables the status subresource.
status: {}
# scale enables the scale subresource.
scale:
  # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
  specReplicasPath: .spec.replicas
  # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
  statusReplicasPath: .status.replicas
  # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
  labelSelectorPath: .status.labelSelector

And create it:

kubectl apply -f resourcedefinition.yaml
After the CustomResourceDefinition object has been created, you can create custom objects.

If you save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: " */5"
image: my-awesome-cron-image
replicas: 3
and create it:

kubectl apply -f my-crontab.yaml
Then new namespaced RESTful API endpoints are created at:

/apis/stable.example.com/v1/namespaces/*/crontabs/status
and

/apis/stable.example.com/v1/namespaces/*/crontabs/scale
A custom resource can be scaled using the kubectl scale command. For example, the following command sets .spec.replicas of the custom resource created above to 5:

kubectl scale --replicas=5 crontabs/my-new-cron-object
crontabs "my-new-cron-object" scaled

kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
5
You can use a PodDisruptionBudget to protect custom resources that have the scale subresource enabled.

Categories
Categories is a list of grouped resources the custom resource belongs to (eg. all). You can use kubectl get to list the resources belonging to the category. This feature is beta and available for custom resources from v1.10.

The following example adds all in the list of categories in the CustomResourceDefinition and illustrates how to output the custom resource using kubectl get all.

Save the following CustomResourceDefinition to resourcedefinition.yaml:

apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
    # categories is a list of grouped resources the custom resource belongs to.
    categories:
    - all
# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            cronSpec:
              type: string
            image:
              type: string
            replicas:
              type: integer
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
    # categories is a list of grouped resources the custom resource belongs to.
    categories:
    - all

And create it:

kubectl apply -f resourcedefinition.yaml

After the CustomResourceDefinition object has been created, you can create custom objects.

Save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

and create it:

kubectl apply -f my-crontab.yaml

You can specify the category using kubectl get:

kubectl get all

and it will include the custom resources of kind CronTab:

NAME                          AGE
crontabs/my-new-cron-object   3s