mirror of
https://github.com/kubesphere/kubekey.git
synced 2025-12-26 01:22:51 +00:00
170 lines
9.0 KiB
Markdown
170 lines
9.0 KiB
Markdown
# CAPKK Developer Guide
|
|
|
|
CAPKK (Cluster-API-Provider-Kubekey) is an SSH-based infrastructure provider that follows the development documentation of the cluster-api Provider contracts. (https://cluster-api.sigs.k8s.io/developer/providers/contracts.html#provider-contract)
|
|
|
|
In the CAPKK project, there are three main resources and corresponding controllers: KKCluster, KKMachine, and KKInstance.
|
|
|
|
## Project structure
|
|
|
|
```bash
|
|
├── api
|
|
│ └── v1beta1 ## CR defined in CAPKK
|
|
├── bootstrap ## bootstrap provider for k3s version
|
|
├── config ## Kubernetes yaml files for CAPKK project
|
|
├── controllers ## controllers included in CAPKK
|
|
│ ├── kkcluster
|
|
│ ├── kkinstance
|
|
│ └── kkmachine
|
|
├── controlplane ## control-plane provider for k3s version
|
|
├── docs ## documentation
|
|
├── exp ## experimental features
|
|
├── hack
|
|
├── pkg
|
|
│ ├── clients ## clients used in CAPKK
|
|
│ │ └── ssh
|
|
│ ├── rootfs ## data storage directory for CAPKK
|
|
│ ├── scope ## interfaces defined by functionality in CAPKK, mainly used to split KKCluster, KKMachine, and KKInstance attribute fields.
|
|
│ ├── service ## concrete implementation of various operations in CAPKK
|
|
│ │ ├── binary ## operations on binary files in CAPKK
|
|
│ │ ├── bootstrap ## operations on machine environment initialization in CAPKK
|
|
│ │ ├── containermanager ## operations on container runtime in CAPKK
|
|
│ │ ├── operation ## basic operations on Linux machines in CAPKK
|
|
│ │ ├── provisioning ## parsing cloud-init or ignition files in CAPKK and mapping them to SSH commands
|
|
│ │ ├── repository ## operations on rpm packages in CAPKK
|
|
│ │ └── util
|
|
│ └── util
|
|
│ ├── filesystem
|
|
│ └── hash
|
|
├── scripts
|
|
├── templates ## used to generate automated scripts in sample yaml for GitHub releases
|
|
├── test ## e2e testing in CAPKK
|
|
│ └── e2e
|
|
├── util ## general utility class for KubeKey
|
|
└── version ## contains KubeKey's version and SHA256 of binary
|
|
````
|
|
|
|
## KKCluster
|
|
KKCluster is used to define the global basic information of a cluster.
|
|
```go
|
|
type KKClusterSpec struct {
|
|
// Define the distribution of Kubernetes, with currently supported parameters including "kubernetes" and "k3s".
|
|
Distribution string `json:"distribution,omitempty"`
|
|
|
|
// Define the SSH authentication information for the nodes.
|
|
Nodes Nodes `json:"nodes"`
|
|
|
|
// Define the endpoints for the control plane.
|
|
// Optional
|
|
ControlPlaneEndpoint clusterv1.APIEndpoint `json:"controlPlaneEndpoint"`
|
|
|
|
// Define the address for the load balancer for the control plane.
|
|
ControlPlaneLoadBalancer *KKLoadBalancerSpec `json:"controlPlaneLoadBalancer,omitempty"`
|
|
|
|
// Define the download address for the cluster component binaries, which is usually used to overwrite default addresses in offline deployment scenarios.
|
|
// Optional
|
|
Component *Component `json:"component,omitempty"`
|
|
|
|
// Define information for the private image repository.
|
|
// Optional
|
|
Registry Registry `json:"registry,omitempty"`
|
|
}
|
|
```
|
|
In the above content, the following points should be noted:
|
|
* The nodes field specifies the SSH information of all nodes included in the cluster.
|
|
* CAPKK uses kube-vip by default to achieve high availability of the cluster control plane, so the controlPlaneLoadBalancer can be set to any unused IP within the subnet.
|
|
* CAPKK uses the official download address of binary components by default. If you want to use the domestic resource address maintained by the KubeSphere team, you can configure it as follows:
|
|
```yaml
|
|
spec:
|
|
component:
|
|
zone: cn
|
|
```
|
|
* The component field can also be used to specify a custom file server address and configure the overrides array to modify the path and SHA256 checksum of different binaries in the file server used by CAPKK.
|
|
|
|
The following is a common KKCluster sample YAML file:
|
|
```yaml
|
|
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
|
|
kind: KKCluster
|
|
metadata:
|
|
name: quick-start
|
|
spec:
|
|
component:
|
|
zone: ""
|
|
nodes:
|
|
auth:
|
|
user: ubuntu
|
|
password: P@88w0rd
|
|
instances:
|
|
- name: test1
|
|
address: 192.168.0.3
|
|
- name: test2
|
|
address: 192.168.0.4
|
|
- name: test3
|
|
address: 192.168.0.5
|
|
controlPlaneLoadBalancer:
|
|
host: 192.168.0.100
|
|
```
|
|
After applying this YAML file, the webhook of KKCluster will add some default values, such as `roles: [control-plane, worker]`.
|
|
## KKMachine
|
|
KKMachine is mainly used to represent the mapping between Kubernetes node and infrastructure host.
|
|
```go
|
|
type KKMachineSpec struct {
|
|
// Unique ID of the infrastructure provider
|
|
ProviderID *string `json:"providerID,omitempty"`
|
|
|
|
// Name of the corresponding KKInstance resource
|
|
InstanceID *string `json:"instanceID,omitempty"`
|
|
|
|
// Role of the node
|
|
// Optional
|
|
Roles []Role `json:"roles"`
|
|
|
|
// Container runtime of the corresponding node for this resource
|
|
// Optional
|
|
ContainerManager ContainerManager `json:"containerManager"`
|
|
|
|
// software packages of the corresponding node for this resource
|
|
// Optional
|
|
Repository *Repository `json:"repository,omitempty"`
|
|
}
|
|
```
|
|
Usually, we do not directly use KKMachine resources, but declare KKMachineTemplate resources for reference by cluster-api resources.
|
|
|
|
|
|
### KKMachineTemplate
|
|
The xxxTemplate resource is the template type of this type of resource, which contains all the contents in the spec of the corresponding resource. In the usage of cluster-api, we declare resources with the replica field, such as KubeadmControlPlane (KCP) and MachineDeployment (MD), indicating that this type of resource can be dynamically scaled. These resources also contain an infrastructureRef field, which refers to a specific infrastructure template resource (xxxMachineTemplate). When scaling up, KCP and MD resources will create corresponding, real single instances of the resource based on the template resource.
|
|
The following is an example of a common control plane KKMachineTemplate yaml:
|
|
```yaml
|
|
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
|
|
kind: KKMachineTemplate
|
|
metadata:
|
|
name: quick-start
|
|
namespace: default
|
|
spec:
|
|
template:
|
|
spec:
|
|
roles:
|
|
- control-plane
|
|
containerManager:
|
|
type: containerd
|
|
repository:
|
|
iso: "none"
|
|
update: false
|
|
```
|
|
Notes:
|
|
- The roles specify control-plane, which means that the resources that reference this KKMachineTemplate will only select machines with corresponding roles in KKCluster for bootstrapping.
|
|
- containerManager specifies that the container runtime configured in the KKMachine created by this template is containerd.
|
|
- repository specifies the policy for the KKMachine to operate rpm software sources created by this template. In this case, the policy indicates that CAPKK will use the default software source on the corresponding Linux machine to install default software dependencies (conntrack, socat, etc.).
|
|
## KKInstance
|
|
In the developer documentation of cluster-api, the definition of infra only includes xxxCluster and xxxMachine resources, while KKInstance is a resource exclusive to CAPKK. The additional definition of KKInstance in CAPKK aims to decouple the logic of the operator and controller, i.e., KKMachine is focused on maintaining interactions with cluster-api CR, while KKInstance focuses on maintaining Linux machines (by performing command-based operations on the machines via SSH).
|
|
The property fields of KKInstance mainly consist of the collection of nodes and KKMachine fields in KKCluster, which will not be elaborated here.
|
|
|
|
The controller of KKInstance mainly performs the following operations on the machine:
|
|
|
|
* Bootstrop: Some machine environment initialization or cleanup operations defined in CAPKK, such as creating corresponding directories, adding Linux users, running initialization shell scripts, resetting the network, deleting files, and resetting Kubeadm.
|
|
* Repository: RPM software source handling operations defined in CAPKK, such as mounting ISO software packages, updating sources, and installing dependency software packages.
|
|
* Binary: Cluster component binary file handling operations defined in CAPKK, such as downloading binary files.
|
|
* ContainerManager: Machine container runtime operations defined in CAPKK, such as checking container runtime and installing container runtime.
|
|
* Provisioning: Parsing cloud-init or ignition files provided by cluster-api for the corresponding machine in CAPKK and mapping them to SSH commands. This cloud-init file will include operations such as "kubeadm init" and "kubeadm join".
|
|
|
|
For the interface definitions of these operations, see [interface](https://github.com/kubesphere/kubekey/blob/master/pkg/service/interface.go).
|