mirror of
https://github.com/kubesphere/website.git
synced 2025-12-26 08:32:55 +00:00
Merge pull request #503 from Sherlock113/filename
fix file name typo and add some en source files to zh folder
This commit is contained in:
KubeSphere CI Bot
GitHub
commit
276df9f295
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
linkTitle: "Cluster-wide Alerting and Notification"
|
||||
weight: 2000
|
||||
|
||||
_build:
|
||||
render: false
|
||||
---
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
---
|
||||
title: "Alerting Message (Node Level)"
|
||||
keywords: 'KubeSphere, Kubernetes, Node, Alerting, Message, Notification'
|
||||
description: 'How to view alerting messages at the node level.'
|
||||
|
||||
linkTitle: "Alerting Message (Node Level)"
|
||||
weight: 4170
|
||||
---
|
||||
|
||||
Alert messages record detailed information of alerts triggered based on alert rules, including monitoring targets, alert policies, recent notifications and comments.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
You have created a node-level alert policy and received alert notifications of it. If it is not ready, please refer to [Alert Policy (Node Level)](../alerting-policy/) to create one first.
|
||||
|
||||
## Hands-on Lab
|
||||
|
||||
### Task 1: View Alert Message
|
||||
|
||||
1. Log in the console with one account granted the role `platform-admin`.
|
||||
|
||||
2. Click **Platform** in the top left corner and select **Clusters Management**.
|
||||
|
||||
|
||||
|
||||
3. Select a cluster from the list and enter it (If you do not enable the [multi-cluster feature](../../../multicluster-management/), you will directly go to the **Overview** page).
|
||||
|
||||
4. Navigate to **Alerting Messages** under **Monitoring & Alerting**, and you can see alert messages in the list. In the example of [Alert Policy (Node Level)](../alerting-policy/), you set one node as the monitoring target, and its memory utilization rate is higher than the threshold of `50%`, so you can see an alert message of it.
|
||||
|
||||
|
||||
|
||||
5. Click the alert message to enter the detail page. In **Alerting Detail**, you can see the graph of memory utilization rate of the node over time, which has been continuously higher than the threshold of `50%` set in the alert rule, so the alert was triggered.
|
||||
|
||||
|
||||
|
||||
### Task 2: View Alert Policy
|
||||
|
||||
Switch to **Alerting Policy** to view the alert policy corresponding to this alert message, and you can see the triggering rule of it set in the example of [Alert Policy (Node Level)](../alerting-policy/).
|
||||
|
||||
|
||||
|
||||
### Task 3: View Recent Notification
|
||||
|
||||
1. Switch to **Recent Notification**. It can be seen that 3 notifications have been received, because the notification rule was set with a repetition period of `Alert once every 5 minutes` and retransmission of `Resend up to 3 times`.
|
||||
|
||||
|
||||
|
||||
2. Log in your email to see alert notification mails sent by the KubeSphere mail server. You have received a total of 3 emails.
|
||||
|
||||
### Task 4: Add Comment
|
||||
|
||||
Click **Comment** to add comments to current alert messages. For example, as memory utilization rate of the node is higher than the threshold set based on the alert rule, you can add a comment in the dialog below: `The node needs to be tainted and new pod is not allowed to be scheduled to it`.
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,100 @@
|
|||
---
|
||||
title: "Alerting Policy (Node Level)"
|
||||
keywords: 'KubeSphere, Kubernetes, Node, Alerting, Policy, Notification'
|
||||
description: 'How to set alerting policies at the node level.'
|
||||
|
||||
linkTitle: "Alerting Policy (Node Level)"
|
||||
weight: 4160
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
KubeSphere provides alert policies for nodes and workloads. This guide demonstrates how you can create alert policies for nodes in the cluster and configure mail notifications. See [Alerting Policy (Workload Level)](../../../project-user-guide/alerting/alerting-policy/) to learn how to configure alert policies for workloads.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [KubeSphere Alerting and Notification](../../../pluggable-components/alerting-notification/) needs to be enabled.
|
||||
- [Mail Server](../../../cluster-administration/cluster-settings/mail-server/) needs to be configured.
|
||||
|
||||
## Hands-on Lab
|
||||
|
||||
### Task 1: Create an Alert Policy
|
||||
|
||||
1. Log in the console with one account granted the role `platform-admin`.
|
||||
|
||||
2. Click **Platform** in the top left corner and select **Clusters Management**.
|
||||
|
||||
|
||||
|
||||
3. Select a cluster from the list and enter it (If you do not enable the [multi-cluster feature](../../../multicluster-management/), you will directly go to the **Overview** page).
|
||||
|
||||
4. Navigate to **Alerting Policies** under **Monitoring & Alerting**, and click **Create**.
|
||||
|
||||
|
||||
|
||||
### Task 2: Provide Basic Information
|
||||
|
||||
In the dialog that appears, fill in the basic information as follows. Click **Next** after you finish.
|
||||
|
||||
- **Name**: a concise and clear name as its unique identifier, such as `alert-demo`.
|
||||
- **Alias**: to help you distinguish alert policies better. Chinese is supported.
|
||||
- **Description**: a brief introduction to the alert policy.
|
||||
|
||||
|
||||
|
||||
### Task 3: Select Monitoring Targets
|
||||
|
||||
Select several nodes in the node list or use Node Selector to choose a group of nodes as the monitoring targets. Here a node is selected for the convenience of demonstration. Click **Next** when you finish.
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
|
||||
You can sort nodes in the node list from the drop-down menu through the following three ways: `Sort By CPU`, `Sort By Memory`, `Sort By Pod Utilization`.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
### Task 4: Add Alerting Rules
|
||||
|
||||
1. Click **Add Rule** to begin to create an alerting rule. The rule defines parameters such as metric type, check period, consecutive times, metric threshold and alert level to provide rich configurations. The check period (the second field under **Rule**) means the time interval between 2 consecutive checks of the metric. For example, `2 minutes/period` means the metric is checked every two minutes. The consecutive times (the third field under **Rule**) means the number of consecutive times that the metric meets the threshold when checked. An alert is only triggered when the actual time is equal to or is greater than the number of consecutive times set in the alert policy.
|
||||
|
||||
|
||||
|
||||
2. In this example, set those parameters to `memory utilization rate`, `1 minute/period`, `2 consecutive times`, `>` and `50%`, and `Major Alert` in turn. It means KubeSphere checks the memory utilization rate every minute, and a major alert is triggered if it is larger than 50% for 2 consecutive times.
|
||||
|
||||
3. Click **√** to save the rule when you finish and click **Next** to continue.
|
||||
|
||||
{{< notice note >}}
|
||||
|
||||
You can create node-level alert policies for the following metrics:
|
||||
|
||||
- CPU: `cpu utilization rate`, `cpu load average 1 minute`, `cpu load average 5 minutes`, `cpu load average 15 minutes`
|
||||
- Memory: `memory utilization rate`, `memory available`
|
||||
- Disk: `inode utilization rate`, `disk space available`, `local disk space utilization rate`, `disk write throughput`, `disk read throughput`, `disk read iops`, `disk write iops`
|
||||
- Network: `network data transmitting rate`, `network data receiving rate`
|
||||
- Pod: `pod abnormal ratio`, `pod utilization rate`
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
### Task 5: Set Notification Rule
|
||||
|
||||
1. **Effective Notification Time Range** is used to set sending time of notification emails, such as `09:00 ~ 19:00`. **Notification Channel** currently only supports **Email**. You can add email addresses of members to be notified to **Notification List**.
|
||||
|
||||
2. **Customize Repetition Rules** defines sending period and retransmission times of notification emails. If alerts have not been resolved, the notification will be sent repeatedly after a certain period of time. Different repetition rules can also be set for different levels of alerts. Since the alert level set in the previous step is `Major Alert`, select `Alert once every 5 miniutes` (sending period) in the second field for **Major Alert** and `Resend up to 3 times` in the third field (retransmission times). Refer to the following image to set notification rules:
|
||||
|
||||
|
||||
|
||||
3. Click **Create**, and you can see that the alert policy is successfully created.
|
||||
|
||||
{{< notice note >}}
|
||||
|
||||
*Waiting Time for Alerting* **=** *Check Period* **x** *Consecutive Times*. For example, if the check period is 1 minute/period, and the number of consecutive times is 2, you need to wait for 2 minutes before the alert message appears.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
### Task 6: View Alert Policy
|
||||
|
||||
After an alert policy is successfully created, you can enter its detail information page to view the status, alert rules, monitoring targets, notification rule, alert history, etc. Click **More** and select **Change Status** from the drop-down menu to enable or disable this alert policy.
|
||||
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
---
|
||||
title: "Manage alerts with Alertmanager in KubeSphere"
|
||||
keywords: 'kubernetes, prometheus, alertmanager, alerting'
|
||||
description: 'Manage alerts with Alertmanager in KubeSphere'
|
||||
|
||||
linkTitle: "Alertmanager in KubeSphere"
|
||||
weight: 2010
|
||||
---
|
||||
|
||||
Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts. For more details, please refer to [Alertmanager guide](https://prometheus.io/docs/alerting/latest/alertmanager/).
|
||||
|
||||
KubeSphere has been using Prometheus as its monitoring service's backend from the first release. Starting from v3.0, KubeSphere adds Alertmanager to its monitoring stack to manage alerts sent from Prometheus as well as other components such as [kube-events](https://github.com/kubesphere/kube-events) and kube-auditing.
|
||||
|
||||
|
||||
|
||||
## Use Alertmanager to manage Prometheus alerts
|
||||
|
||||
Alerting with Prometheus is separated into two parts. Alerting rules in Prometheus servers send alerts to an Alertmanager. The Alertmanager then manages those alerts, including silencing, inhibition, aggregation and sending out notifications via methods such as email, on-call notification systems, and chat platforms.
|
||||
|
||||
Starting from v3.0, KubeSphere adds popular alert rules in the open source community to its Prometheus offering as builtin alert rules. And by default Prometheus in KubeSphere v3.0 evaluates these builtin alert rules continuously and then sends alerts to Alertmanager.
|
||||
|
||||
## Use Alertmanager to manage K8s events alerts
|
||||
|
||||
Alertmanager can be used to manage alerts sent from sources other than Prometheus. In KubeSphere v3.0 and above, user can use it to manage alerts triggered by K8s events. For more details, please refer to [kube-events](https://github.com/kubesphere/kube-events).
|
||||
|
||||
## Use Alertmanager to manage KubeSphere auditing alerts
|
||||
|
||||
In KubeSphere v3.0 and above, user can also use Alertmanager to manage alerts triggered by K8s/KubeSphere audit events.
|
||||
|
||||
## Receiving notifications for Alertmanager alerts
|
||||
|
||||
Generally, to receive notifications for Alertmanager alerts, users have to edit Alertmanager's configuration files manually to configure receiver settings such as Email and Slack.
|
||||
|
||||
This is not convenient for K8s users and it breaks the multi-tenant principle/architecture of KubeSphere. More specifically, alerts triggered by workloads in different namespaces belonging to different users might be sent to the same user.
|
||||
|
||||
To use Alertmanager to manage alerts on the platform, KubeSphere offers [Notification Manager](https://github.com/kubesphere/notification-manager), a Kubernetes native notification management tool, which is completely open source. It complies with the multi-tenancy principle, providing user-friendly experiences of Kubernetes notifications. It's installed by default in KubeSphere v3.0 and above.
|
||||
|
||||
For more details about using Notification Manager to receive Alertmanager Notifications, please refer to [Notification Manager](../notification-manager)
|
||||
|
|
@ -0,0 +1,450 @@
|
|||
---
|
||||
title: "Manage multi-tenant notifications with Notification Manager"
|
||||
keywords: "kubernetes, KubeSphere, notification manager, email, wechat, slack"
|
||||
description: "K8s native notification management with multi-tenancy support"
|
||||
|
||||
linkTitle: "Notification Manager"
|
||||
weight: 2020
|
||||
---
|
||||
|
||||
[Notification Manager](https://github.com/kubesphere/notification-manager) manages notifications in KubeSphere. It receives alerts or notifications from different senders and then sends notifications to different users.
|
||||
|
||||
Supported senders includes:
|
||||
|
||||
- Prometheus Alertmanager
|
||||
- Custom sender (Coming soon)
|
||||
|
||||
Supported receivers includes:
|
||||
|
||||
- Email
|
||||
- [Wechat Work](https://work.weixin.qq.com/)
|
||||
- Slack
|
||||
- Webhook (Coming soon)
|
||||
|
||||
|
||||
|
||||
## QuickStart
|
||||
|
||||
### Config Prometheus Alertmanager to send alerts to Notification Manager
|
||||
|
||||
Notification Manager uses port `19093` and API path `/api/v2/alerts` to receive alerts sent from Prometheus Alertmanager of Kubesphere.
|
||||
|
||||
To receive Alertmanager alerts, **KubeSphere already added Alertmanager webhook and route configurations like below** ( by editing the Secret alertmanager-main in the namespace `kubesphere-monitoring-system` ):
|
||||
|
||||
Send Prometheus alerts to Notification Manager:
|
||||
|
||||
```yaml
|
||||
"receivers":
|
||||
- "name": "prometheus"
|
||||
"webhook_configs":
|
||||
- "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
|
||||
"route":
|
||||
"routes":
|
||||
- "match":
|
||||
"alerttype": ""
|
||||
"receiver": "prometheus"
|
||||
```
|
||||
|
||||
Send event alerts to Notification Manager:
|
||||
|
||||
```yaml
|
||||
"receivers":
|
||||
- "name": "event"
|
||||
"webhook_configs":
|
||||
- "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
|
||||
"send_resolved": false
|
||||
"route":
|
||||
"routes":
|
||||
- "match":
|
||||
"alerttype": "event"
|
||||
"receiver": "event"
|
||||
"group_interval": "30s"
|
||||
```
|
||||
|
||||
Send auditing alerts to Notification Manager:
|
||||
|
||||
```yaml
|
||||
"receivers":
|
||||
- "name": "auditing"
|
||||
"webhook_configs":
|
||||
- "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
|
||||
"send_resolved": false
|
||||
"route":
|
||||
"routes":
|
||||
- "match":
|
||||
"alerttype": "auditing"
|
||||
"receiver": "auditing"
|
||||
"group_interval": "30s"
|
||||
```
|
||||
|
||||
{{< notice note >}}
|
||||
|
||||
The above is the default configuration. If you do not want to receive a certain type of alert, you can delete the corresponding configuration.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
### Configure receivers
|
||||
|
||||
Notification Manager now supports three types of receivers: Email, WeChat Work and Slack. Only the administrator can configure receivers.
|
||||
|
||||
#### Email
|
||||
|
||||
If a tenant named **test-user** who wants to receive notifications from email, just create an email receiver like this.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
data:
|
||||
password: dGVzdA==
|
||||
kind: Secret
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
name: test-user-email-secret
|
||||
namespace: kubesphere-monitoring-system
|
||||
type: Opaque
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: EmailConfig
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: tenant
|
||||
user: test-user
|
||||
name: test-user-config
|
||||
namespace: kubesphere-monitoring-system
|
||||
spec:
|
||||
authPassword:
|
||||
key: password
|
||||
name: test-user-email-secret
|
||||
authUsername: abc1
|
||||
from: abc1@xyz.com
|
||||
requireTLS: true
|
||||
smartHost:
|
||||
host: imap.xyz.com
|
||||
port: "25"
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: EmailReceiver
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: tenant
|
||||
user: test-user
|
||||
name: test-user-receiver
|
||||
namespace: kubesphere-monitoring-system
|
||||
spec:
|
||||
emailConfigSelector:
|
||||
matchLabels:
|
||||
type: tenant
|
||||
user: test-user
|
||||
to:
|
||||
- abc2@xyz.com
|
||||
- abc3@xyz.com
|
||||
EOF
|
||||
```
|
||||
|
||||
The emailConfigSelector is a selector to select EmailConfig for email receiver, if the emailConfigSelector is not set, receiver will use the default email config. You can create a default email config like this.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
data:
|
||||
password: dGVzdA==
|
||||
kind: Secret
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
name: default-email-secret
|
||||
namespace: kubesphere-monitoring-system
|
||||
type: Opaque
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: EmailConfig
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: default
|
||||
name: default-email-config
|
||||
namespace: kubesphere-monitoring-system
|
||||
spec:
|
||||
authPassword:
|
||||
key: password
|
||||
name: default-email-secret
|
||||
authUsername: default
|
||||
from: default@xyz.com
|
||||
requireTLS: true
|
||||
smartHost:
|
||||
host: imap.xyz.com
|
||||
port: "25"
|
||||
EOF
|
||||
```
|
||||
|
||||
Email receivers with label `type: tenant` only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global email receiver with label `type: global` as below::
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: EmailReceiver
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: global
|
||||
name: global-email-receiver
|
||||
namespace: kubesphere-monitoring-system
|
||||
spec:
|
||||
to:
|
||||
- global@xyz.com
|
||||
EOF
|
||||
```
|
||||
|
||||
{{< notice note >}}
|
||||
|
||||
Global email receiver will use the default email config.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
#### Wechat Work
|
||||
|
||||
Notification Manager supports sending notification to Wechat Work.
|
||||
If a tenant named **test-user** who wants to receive notifications from Wechat Work, just create a wechat receiver like this.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
data:
|
||||
wechat: dGVzdA==
|
||||
kind: Secret
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
name: test-user-wechat-secret
|
||||
namespace: kubesphere-monitoring-system
|
||||
type: Opaque
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: WechatConfig
|
||||
metadata:
|
||||
name: test-user-config
|
||||
namespace: kubesphere-monitoring-system
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: tenant
|
||||
user: test-user
|
||||
spec:
|
||||
wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
|
||||
wechatApiSecret:
|
||||
key: wechat
|
||||
name: test-user-wehat-secret
|
||||
wechatApiCorpId: wwfd76b24f06513578
|
||||
wechatApiAgentId: "1000002"
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: WechatReceiver
|
||||
metadata:
|
||||
name: test-user-wechat
|
||||
namespace: kubesphere-monitoring-system
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: tenant
|
||||
user: test-user
|
||||
spec:
|
||||
wechatConfigSelector:
|
||||
matchLabels:
|
||||
type: tenant
|
||||
user: test-user
|
||||
# optional
|
||||
# One of toUser, toParty, toParty should be specified.
|
||||
toUser: user1 | user2
|
||||
toParty: party1 | party2
|
||||
toTag: tag1 | tag2
|
||||
EOF
|
||||
```
|
||||
|
||||
{{< notice info >}}
|
||||
|
||||
- wechatApiCorpId is the id of your Wechat Work.
|
||||
- wechatApiAgentId is the id of app sending message to user in your Wechat Work
|
||||
- wechatApiSecret is the secret of this app, you can get these two parameters in App Management of your Wechat Work.
|
||||
- Any user, party or tag who wants to receive notifications must be in the allowed users list of this app.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
The wechatConfigSelector is a selector to select WechatConfig for wechat receiver, if the wechatConfigSelector is not set, wechat receiver will use the default wechat config. You can create a default wechat config like this.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
data:
|
||||
wechat: dGVzdA==
|
||||
kind: Secret
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
name: default-wechat-secret
|
||||
namespace: kubesphere-monitoring-system
|
||||
type: Opaque
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: WechatConfig
|
||||
metadata:
|
||||
name: default-wechat-config
|
||||
namespace: kubesphere-monitoring-system
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: default
|
||||
spec:
|
||||
wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
|
||||
wechatApiSecret:
|
||||
key: wechat
|
||||
name: default-wechat-secret
|
||||
wechatApiCorpId: wwfd76b24f06513578
|
||||
wechatApiAgentId: "1000002"
|
||||
EOF
|
||||
```
|
||||
|
||||
Wechat receivers with label `type: tenant` can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global wechat receiver with label `type: global` as below:
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: WechatReceiver
|
||||
metadata:
|
||||
name: global-wechat-wechat
|
||||
namespace: kubesphere-monitoring-system
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: global
|
||||
spec:
|
||||
# optional
|
||||
# One of toUser, toParty, toParty should be specified.
|
||||
toUser: global
|
||||
toParty: global
|
||||
toTag: global
|
||||
EOF
|
||||
```
|
||||
|
||||
{{< notice note >}}
|
||||
|
||||
Global wechat receiver will use the default wechat config.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
#### Slack
|
||||
|
||||
Notification Manager supports sending notification to slack channels. If a tenant named **test-user** who wants to receive notifications from slack, just create a slack receiver like this.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
data:
|
||||
token: dGVzdA==
|
||||
kind: Secret
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
name: test-user-slack-secret
|
||||
namespace: kubesphere-monitoring-system
|
||||
type: Opaque
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: SlackConfig
|
||||
metadata:
|
||||
name: test-user-config
|
||||
namespace: kubesphere-monitoring-system
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: tenant
|
||||
user: test-user
|
||||
spec:
|
||||
slackTokenSecret:
|
||||
key: token
|
||||
name: test-user-slack-secret
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: SlackReceiver
|
||||
metadata:
|
||||
name: test-user-slack
|
||||
namespace: kubesphere-monitoring-system
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: tenant
|
||||
user: test-user
|
||||
spec:
|
||||
slackConfigSelector:
|
||||
matchLabels:
|
||||
type: tenant
|
||||
user: test-user
|
||||
channel: alert
|
||||
EOF
|
||||
```
|
||||
|
||||
{{< notice info>}}
|
||||
|
||||
- Slack token is the OAuth Access Token or Bot User OAuth Access Token when you create a slack app.
|
||||
- This app must have the scope chat:write.
|
||||
- The user who creates the app or bot user must be in the channel to which you want to send notifications.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
The slackConfigSelector is a selector to select SlackConfig for slack receiver, if the slackConfigSelector is not set, slack receiver will use the default slack config. You can create a default slack config like this.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
data:
|
||||
token: dGVzdA==
|
||||
kind: Secret
|
||||
metadata:
|
||||
labels:
|
||||
app: notification-manager
|
||||
name: default-slack-secret
|
||||
namespace: kubesphere-monitoring-system
|
||||
type: Opaque
|
||||
|
||||
---
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: SlackConfig
|
||||
metadata:
|
||||
name: default-slack-config
|
||||
namespace: kubesphere-monitoring-system
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: default
|
||||
spec:
|
||||
slackTokenSecret:
|
||||
key: token
|
||||
name: default-slack-secret
|
||||
EOF
|
||||
```
|
||||
|
||||
Slack receivers with label `type: tenant` can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global slack receiver with label `type: global` as below:
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: notification.kubesphere.io/v1alpha1
|
||||
kind: SlackReceiver
|
||||
metadata:
|
||||
name: global-slack-slack
|
||||
namespace: kubesphere-monitoring-system
|
||||
labels:
|
||||
app: notification-manager
|
||||
type: global
|
||||
spec:
|
||||
channel: global
|
||||
EOF
|
||||
```
|
||||
|
||||
{{< notice note>}}
|
||||
|
||||
Global slack receiver will use the default slack config.
|
||||
|
||||
{{</ notice >}}
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
---
|
||||
title: "Cluster Shutdown and Restart"
|
||||
description: "Demonstrate how to shut down and restart Kubernetes clusters gracefully"
|
||||
layout: "single"
|
||||
|
||||
linkTitle: "Cluster Shutdown and Restart"
|
||||
weight: 5000
|
||||
|
||||
icon: "/images/docs/docs.svg"
|
||||
---
|
||||
This document describes the process of gracefully shutting down your cluster and how to restart it. You might need to temporarily shut down your cluster for maintenance reasons.
|
||||
|
||||
{{< notice warning >}}
|
||||
Shutting down a cluster is very dangerous. You must fully understand the operation and its consequences. Please make an etcd backup before you proceed.
|
||||
Usually, it is recommended to maintain your nodes one by one instead of restarting the whole cluster.
|
||||
{{</ notice >}}
|
||||
|
||||
## Prerequisites
|
||||
- Take an [etcd backup](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md#snapshotting-the-keyspace) prior to shutting down a cluster.
|
||||
- SSH [passwordless login](https://man.openbsd.org/ssh.1#AUTHENTICATION) is set up between hosts.
|
||||
|
||||
## Shutting Down Cluster
|
||||
{{< notice tip >}}
|
||||
|
||||
- You must back up your etcd data before you shut down the cluster as your cluster can be restored if you encounter any issues when restarting the cluster.
|
||||
- Using the method in this tutorial can shut down a cluster gracefully, while the possibility of data corruption still exists.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
### Step 1: Get Node List
|
||||
```bash
|
||||
nodes=$(kubectl get nodes -o name)
|
||||
```
|
||||
### Step 2: Shut Down All Nodes
|
||||
```bash
|
||||
for node in ${nodes[@]}
|
||||
do
|
||||
echo "==== Shut down $node ===="
|
||||
ssh $node sudo shutdown -h 1
|
||||
done
|
||||
```
|
||||
Then you can shut down other cluster dependencies, such as external storage.
|
||||
|
||||
## Restart Cluster Gracefully
|
||||
You can restart a cluster gracefully after shutting down the cluster gracefully.
|
||||
|
||||
### Prerequisites
|
||||
You have shut down your cluster gracefully.
|
||||
|
||||
{{< notice tip >}}
|
||||
Usually, a cluster can be used after restarting, but the cluster may be unavailable due to unexpected conditions. For example:
|
||||
|
||||
- Etcd data corruption during the shutdown.
|
||||
- Node failures.
|
||||
- Unexpected network errors.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
### Step 1: Check All Cluster Dependencies' Status
|
||||
Ensure all cluster dependencies are ready, such as external storage.
|
||||
### Step 2: Power on Cluster Machines
|
||||
Wait for the cluster to be up and running, which may take about 10 minutes.
|
||||
### Step 3: Check All Master Nodes' Status
|
||||
Check the status of core components, such as etcd services, and make sure everything is ready.
|
||||
```bash
|
||||
kubectl get nodes -l node-role.kubernetes.io/master
|
||||
```
|
||||
|
||||
### Step 4: Check All Worker Nodes' Status
|
||||
```bash
|
||||
kubectl get nodes -l node-role.kubernetes.io/worker
|
||||
```
|
||||
|
||||
If your cluster fails to restart, please try to [restore the etcd cluster](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md#restoring-a-cluster).
|
||||
Loading…
Reference in New Issue