diff --git a/content/en/docs/v3.4/cluster-administration/cluster-settings/log-collections/add-opensearch-as-receiver.md b/content/en/docs/v3.4/cluster-administration/cluster-settings/log-collections/add-opensearch-as-receiver.md new file mode 100755 index 000000000..2854ce699 --- /dev/null +++ b/content/en/docs/v3.4/cluster-administration/cluster-settings/log-collections/add-opensearch-as-receiver.md @@ -0,0 +1,126 @@ +--- +title: "Add OpenSearch as a Log Receiver" +keywords: 'Kubernetes, Logs, OpenSearch, Pod, Container, Fluentbit, Output' +description: 'Learn how to add OpenSearch to receive container logs, resource events, or audit logs.' +linkTitle: "Add OpenSearch as a Log Receiver" +weight: 8625 +--- + +[OpenSearch](https://opensearch.org/) is a distributed, community-driven, and 100% open-source search and analytics suite licensed under Apache 2.0. It can be used for real-time application monitoring, log analysis, website search, and more. + +OpenSearch is backed by the Apache Lucene search library and supports a range of search and analysis features such as k-Nearest Neighbors (KNN) search, SQL, anomaly detection, Machine Learning Commons, Trace Analytics, full-text search, and more. + +OpenSearch provides a highly scalable system, allowing users to explore their data easily through integrated visualization tools. + +Starting from KubeSphere v3.4.0, OpenSearch v1 and v2 are integrated into it and set as the default backend storage for the `logging`, `events`, and `auditing` components. + +## Prerequisites + +- You need a user with cluster management permissions. For example, you can log in to the console directly with the `admin` user or create a role with cluster management permissions and assign it to a user. + +- Before adding a log receiver, you need to enable the `logging`, `events`, or `auditing` components. For more information, see [Enable Pluggable Components](../../../../pluggable-components/). This tutorial enables `logging` as an example. + +## Use OpenSearch as a Log Receiver + +In KubeSphere v3.4.0 and later, OpenSearch is the default backend storage for `logging`, `events`, or `auditing` components. Configure as follows: + +```shell +$ kubectl edit cc -n kubesphere-system ks-installer + +apiVersion: installer.kubesphere.io/v1alpha1 +kind: ClusterConfiguration +metadata: + name: ks-installer + namespace: kubesphere-system +spec: + common: + opensearch: # Storage backend for logging, events, and auditing. + enabled: true + logMaxAge: 7 # Log retention time in built-in Opensearch. It is 7 days by default. + opensearchPrefix: whizard # The string making up index names. The index name will be formatted as ks--logging. +``` + +For KubeSphere versions below `v3.4.0`, please [upgrade](https://github.com/kubesphere/ks-installer/tree/release-3.4#upgrade) first. + +### Enable Logging via the Console and Use OpenSearch as the Backend Storage + +1. Log in to the console as the `admin` user, click **Platform** in the upper left corner, and select **Cluster Management**. + +2. Click **CRDs**, enter `clusterconfiguration` in the search bar, and click the search result to view its detailed page. + +![](/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/cc-search.png) + +3. Under **Custom Resources**, click the three dots on the right side of `ks-installer`, select **Edit YAML**. + +![](/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/cc-ks-installer.png) + +4. In the YAML file, search for `logging`, change the `enabled` from `false` to `true`. After that, click **OK** in the lower-right corner to save the configuration. + + ```yaml + common: + opensearch: + enabled: true + + logging: + enabled: true + ``` + +## Change Log Storage to External OpenSearch and Disable Internal OpenSearch + +If you are using KubeSphere's internal OpenSearch and want to change it to your external OpenSearch, follow these steps: + +1. Run the following command to update the cluster configuration: + + ```shell + kubectl edit cc -n kubesphere-system ks-installer + ``` + +2. Set `opensearch:externalOpensearchHost` to the address of your external OpenSearch and set `opensearch:externalOpensearchPort` to its port. Comment or delete the `status:logging` field. Here is an example: + + ```yaml + apiVersion: installer.kubesphere.io/v1alpha1 + kind: ClusterConfiguration + metadata: + name: ks-installer + namespace: kubesphere-system + spec: + common: + opensearch: + enabled: true + externalOpensearchHost: "" + externalOpensearchPort: "" + dashboard: + enabled: false + status: + # logging: + # enabledTime: 2023-08-21T21:05:13UTC + # status: enabled + ``` + + If you want to use the visualization tools of `OpenSearch`, set `opensearch.dashboard.enabled` to `true`. + +3. Run the following command to restart `ks-installer`. + + ```shell + kubectl rollout restart deploy -n kubesphere-system ks-installer + ``` + +4. Run the following command to delete the internal OpenSearch. Make sure you have backed up the data in it. + + ```shell + helm uninstall opensearch-master -n kubesphere-logging-system && helm uninstall opensearch-data -n kubesphere-logging-system && helm uninstall opensearch-logging-curator -n kubesphere-logging-system + ``` + +## Query Logs in KubeSphere + +1. All users can use the log query feature. Log in with an account, hover over the icon in the lower right corner, and select Log Query from the pop-up menu. + +![](/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-icon.png) + +2. In the pop-up window, you can see the time histogram of the number of logs, the cluster dropdown list, and the log search bar. + +![](/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-dashboard.png) + +3. Click on the search bar and search for logs by message, workspace, project, resource type, resource name, reason, category, or time range (e.g., enter the time range: last 10 minutes to search for logs in the last 10 minutes). Alternatively, click on the bar in the time histogram, and KubeSphere will show logs in the time range of that bar. + +![](/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-filter.png) \ No newline at end of file diff --git a/content/en/docs/v3.4/cluster-administration/logs-dashboard.md b/content/en/docs/v3.4/cluster-administration/logs-dashboard.md new file mode 100644 index 000000000..66fdb5a6e --- /dev/null +++ b/content/en/docs/v3.4/cluster-administration/logs-dashboard.md @@ -0,0 +1,178 @@ +--- +title: "KubeSphere Log Dashboard" +keywords: "Kubernetes, KubeSphere, OpenSearch, Dashboard" +description: "Learn how to enable log dashboard, a graphical interface tool similar to ElasticSearch Kibana." +linkTitle: "KubeSphere Log Dashboard" +weight: 6200 +--- + +As an open-source, application-centric container platform, KubeSphere v3.4.1 uses [OpenSearch](https://opensearch.org/) instead of ElasticSearch as the backend storage for logs, events, and auditing. By default, we can use the query tool provided in the lower-right corner on the KubeSphere console to retrieve logs, events, and auditing records. + +If you want an experience similar to the Kibana page, such as log chart drawing, you can enable OpenSearch Dashboard. + + +## Enable Log Dashboard Before KubeSphere Installation + +### Install on Linux + +When installing KubeSphere with multiple nodes on Linux, you should create a configuration file that lists all KubeSphere components. + +1. When you [Install KubeSphere on Linux](../../installing-on-linux/introduction/multioverview/), create a file `config-sample.yaml` and change it by executing the following command: + + ```bash + vi config-sample.yaml + ``` + +2. Before enabling the OpenSearch Dashboard, you need to enable components `logging`, `opensearch`, and `events` or `auditing` in the yaml file. In this example, enable `events` as follows: + + ```yaml + opensearch: + basicAuth: + enabled: true + password: admin + username: admin + dashboard: + enabled: true # Change "false" to "true". + enabled: true # Change "false" to "true". + externalOpensearchHost: "" + externalOpensearchPort: "" + logMaxAge: 7 + opensearchPrefix: whizard + ``` + + ```yaml + logging: + enabled: true # Change "false" to "true". + logsidecar: + enabled: true + replicas: 2 + ``` + + ```yaml + events: + enabled: true # Change "false" to "true". + ruler: + enabled: true + replicas: 2 + ``` + +3. Execute the following command to create the cluster using this configuration file: + + ```bash + ./kk create cluster -f config-sample.yaml + ``` + +### Install on Kubernetes + +When you [Install KubeSphere on Kubernetes](../../installing-on-kubernetes/introduction/overview/), you need to enable the relevant components in the [cluster-configuration.yaml](https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml) file. + +1. Download the [cluster-configuration.yaml](https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml) file and edit the file using the following command: + + ```bash + vi cluster-configuration.yaml + ``` + +2. Before enabling the OpenSearch Dashboard, you need to enable components `logging`, `opensearch`, and `events` or `auditing` in the yaml file. In this example, enable `events` as follows: + + ```yaml + opensearch: + basicAuth: + enabled: true + password: admin + username: admin + dashboard: + enabled: true # Change "false" to "true". + enabled: true # Change "false" to "true". + externalOpensearchHost: "" + externalOpensearchPort: "" + logMaxAge: 7 + opensearchPrefix: whizard + ``` + + ```yaml + logging: + enabled: true # Change "false" to "true". + logsidecar: + enabled: true + replicas: 2 + ``` + + ```yaml + events: + enabled: true # Change "false" to "true". + ruler: + enabled: true + replicas: 2 + ``` + +3. Execute the following command to start KubeSphere installation: + + ```bash + kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/kubesphere-installer.yaml + + kubectl apply -f cluster-configuration.yaml + ``` + +## Enable Log Dashboard After KubeSphere Installation + +1. Log in to the console as the `admin` user, click **Platform** in the upper left corner, and select **Cluster Management**. + +2. Click **CRDs**, enter `clusterconfiguration` in the search bar, and click the search result to view its detailed page. + + {{< notice info >}} + Custom Resource Definitions (CRDs) allow users to create a new resource type without adding an additional API server. Users can use these custom resources just like other native Kubernetes objects. + {{}} + +3. Under **Custom Resources**, click the three dots on the right side of `ks-installer`, select **Edit YAML**. + +4. In the YAML file, edit as follows, and then click **OK** to save the configuration. + + ```yaml + opensearch: + basicAuth: + enabled: true + password: admin + username: admin + dashboard: + enabled: true # Change "false" to "true". + enabled: true # Change "false" to "true". + externalOpensearchHost: "" + externalOpensearchPort: "" + logMaxAge: 7 + opensearchPrefix: whizard + ``` + + ```yaml + logging: + enabled: true # Change "false" to "true". + logsidecar: + enabled: true + replicas: 2 + ``` + + ```yaml + events: + enabled: true # Change "false" to "true". + ruler: + enabled: true + replicas: 2 + ``` + +5. Check the installation process in kubectl by executing the following command: + + ```bash + kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f + ``` + + {{< notice note >}} + You can find the kubectl tool by clicking on the lower-right corner of the console. + + + + {{}} + +## Access Log Dashboard + +After logging in to the console, expose the 5601 port of the OpenSearch dashboard through NodePort or other forms such as Ingress to an accessible network as below. + + \ No newline at end of file diff --git a/content/en/docs/v3.4/devops-user-guide/how-to-use/pipelines/use-pipeline-step-template.md b/content/en/docs/v3.4/devops-user-guide/how-to-use/pipelines/use-pipeline-step-template.md new file mode 100644 index 000000000..4f20a6ee7 --- /dev/null +++ b/content/en/docs/v3.4/devops-user-guide/how-to-use/pipelines/use-pipeline-step-template.md @@ -0,0 +1,202 @@ +--- +title: "Use Pipeline Step Templates" +keywords: 'KubeSphere, Kubernetes, Jenkins, pipeline step templates' +description: 'Learn how to configure and use pipeline step templates in KubeSphere.' +linkTitle: "Use Pipeline Step Templates" +weight: 11214 +--- + + +In KubeSphere 3.4.x, the DevOps project supports using step templates in pipelines. + +This section describes how to configure and use pipeline step templates in KubeSphere. + +## Prerequisites + +- Create a workspace and a user (`project-admin`). Invite this user to the workspace and assign the role "workspace-self-provisioner" to the user. For more information, please refer to [Create Workspaces, Projects, Users and Roles](../../../../quick-start/create-workspace-and-project/). + +- [Enable KubeSphere DevOps system.](../../../../pluggable-components/devops/). + +### Enable DevOps + +1. Log in to the console as the `admin` user, click **Platform** in the upper left corner, and select **Cluster Management**. + +2. Click **CRDs**, enter `clusterconfiguration` in the search bar, and click the search result to view its detailed page. + +3. Under **Custom Resources**, click the three dots on the right side of `ks-installer`, select **Edit YAML**, and change the 'enabled' configuration under 'devops' to true. + + ``` + devops: + enabled: true # Change 'false' to 'true'. + + ``` +4. Run the 'kubectl' command to check the DevOps installation process. + + ``` + kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f + + ``` +5. Run the 'kubectl' command to verify if the installation is completed. + + ``` + kubectl get pod -n kubesphere-devops-system + ``` + + If the corresponding pod is in the 'Running' state, it indicates a successful installation. + + ``` + devops-apiserver-7576cfc79c-j9kdz 1/1 Running 0 23h + devops-controller-7bcbbfc546-lszkt 1/1 Running 0 23h + devops-jenkins-79b59bdd5-tjrj8 1/1 Running 0 23h + s2ioperator-0 1/1 Running 0 23h + ``` + +## Create and Use Custom Step Templates + +### Create Custom Step Templates + +Currently, custom step templates can only be created through the console. + +1. Use the 'kubectl' command to view existing step templates. + + ``` + kubectl get clustersteptemplates + ``` + + ``` + NAME AGE + archiveartifacts 6d7h + build 6d7h + cd 6d7h + checkout 6d7h + container 6d7h + echo 6d7h + error 6d7h + git 6d7h + input 6d7h + junit 6d7h + mail 6d7h + retry 6d7h + script 6d7h + shell 6d7h + sleep 6d7h + timeout 6d7h + waitforqualitygate 6d7h + withcredentials 6d7h + withsonarqubeenv 6d7h + ``` + +2. To create a custom step template, create a YAML file first to support simple file writing. + + ``` + apiVersion: devops.kubesphere.io/v1alpha3 + kind: ClusterStepTemplate + metadata: + annotations: + devops.kubesphere.io/descriptionEN: Write message to file in the build + devops.kubesphere.io/descriptionZH: Write a file during the build process + devops.kubesphere.io/displayNameEN: writeFile + devops.kubesphere.io/displayNameZH: Write a file + meta.helm.sh/release-name: devops + meta.helm.sh/release-namespace: kubesphere-devops-system + step.devops.kubesphere.io/icon: loudspeaker + generation: 1 + labels: + app.kubernetes.io/managed-by: Helm + step.devops.kubesphere.io/category: General + name: writefile + spec: + parameters: + - display: file + name: file + required: true + type: string + - display: text + name: text + required: true + type: string + runtime: dsl + template: | + { + "arguments": [ + { + "key": "file", + "value": { + "isLiteral": true, + "value": "{{.param.file}}" + } + }, + { + "key": "text", + "value": { + "isLiteral": true, + "value": "{{.param.text}}" + } + } + ], + "name": "writeFile" + } + ``` + + {{< notice note >}} + + a. Step templates are implemented through CRDs (Custom Resource Definitions). For more details, please refer to [CRD for Step Templates](https://github.com/kubesphere-sigs/ks-devops-helm-chart/blob/master/charts/ks-devops/crds/devops.kubesphere.io_clustersteptemplates.yaml). + + b. `metadata.name` in the YAML file should be consistent with `spec.template.name`. Additionally, the `name` field relies on functions within Jenkins to achieve the corresponding functionality. In the provided YAML file, the `writeFile` function is used to implement the output functionality. For more details, please refer to [pipeline steps](https://www.jenkins.io/doc/pipeline/steps/). + + {{}} + +3. Create a custom step using the kubectl command. + + ``` + kubectl apply -f test-writefile.yaml + ``` + +4. Run the below command to check if the custom step template "writefile" has been created. + + ``` + kubectl get clustersteptemplates + + NAME AGE + archiveartifacts 37d + build 37d + cd 37d + checkout 37d + container 37d + echo 37d + error 37d + git 37d + input 37d + junit 37d + mail 37d + pwd 28d + retry 37d + script 37d + shell 37d + sleep 37d + timeout 37d + waitforqualitygate 37d + withcredentials 37d + withsonarqubeenv 37d + writefile 28s + ``` + +### Use Custom Step Templates + +1. After entering the DevOps project, create a new pipeline. + + ![](/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-pipeline-1.png) + +2. In the pipeline editor, select predefined templates such as Node.js, Maven, Golang, etc. based on your requirements. Or create a custom pipeline. + + ![](/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-pipeline-2.png) + +3. Select the template "Golang" to create a pipeline. In the pipeline, create a notification stage at the end of the pipeline. + + ![](/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-step-1.png) + +4. In the notification stage, add steps. There are several step templates available here, and you can choose the custom step `writefile`. + + ![](/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/use-step-1.png) + +Now the configuration of a custom step template is completed. \ No newline at end of file diff --git a/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/cc-ks-installer.png b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/cc-ks-installer.png new file mode 100644 index 000000000..ae163f125 Binary files /dev/null and b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/cc-ks-installer.png differ diff --git a/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/cc-search.png b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/cc-search.png new file mode 100644 index 000000000..77c3a781b Binary files /dev/null and b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/cc-search.png differ diff --git a/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-dashboard.png b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-dashboard.png new file mode 100644 index 000000000..fef63489e Binary files /dev/null and b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-dashboard.png differ diff --git a/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-filter.png b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-filter.png new file mode 100644 index 000000000..d48477a23 Binary files /dev/null and b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-filter.png differ diff --git a/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-icon.png b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-icon.png new file mode 100644 index 000000000..d1c731c8a Binary files /dev/null and b/static/images/docs/v3.x/cluster-administration/cluster-settings/log-collections/logging-icon.png differ diff --git a/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-1.png b/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-1.png new file mode 100644 index 000000000..627a82fbc Binary files /dev/null and b/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-1.png differ diff --git a/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-2.png b/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-2.png new file mode 100644 index 000000000..2cd87f915 Binary files /dev/null and b/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-2.png differ diff --git a/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-3.png b/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-3.png new file mode 100644 index 000000000..84713426d Binary files /dev/null and b/static/images/docs/v3.x/cluster-administration/logs-dashboard/logs-dashboard-3.png differ diff --git a/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-pipeline-1.png b/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-pipeline-1.png new file mode 100644 index 000000000..1ceb4a492 Binary files /dev/null and b/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-pipeline-1.png differ diff --git a/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-pipeline-2.png b/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-pipeline-2.png new file mode 100644 index 000000000..4d19ddf22 Binary files /dev/null and b/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-pipeline-2.png differ diff --git a/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-step-1.png b/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-step-1.png new file mode 100644 index 000000000..92d915613 Binary files /dev/null and b/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/create-step-1.png differ diff --git a/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/use-step-1.png b/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/use-step-1.png new file mode 100644 index 000000000..861bde322 Binary files /dev/null and b/static/images/docs/v3.x/devops-user-guide/using-devops/use-step-templates/use-step-1.png differ