Change image path and update format in DevOps guides
Signed-off-by: Sherlock113 <sherlockxu@yunify.com>
Sherlock113
|
|
@ -1,8 +1,8 @@
|
|||
---
|
||||
title: "Integrate SonarQube into Pipeline"
|
||||
title: "Integrate SonarQube into Pipelines"
|
||||
keywords: 'Kubernetes, KubeSphere, devops, jenkins, sonarqube, pipeline'
|
||||
description: 'This tutorial demonstrates how to integrate SonarQube into pipelines.'
|
||||
linkTitle: "Integrate SonarQube into Pipeline"
|
||||
linkTitle: "Integrate SonarQube into Pipelines"
|
||||
weight: 200
|
||||
---
|
||||
|
||||
|
|
@ -18,29 +18,29 @@ You need to [enable KubeSphere DevOps System](../../../../docs/pluggable-compone
|
|||
|
||||
1. Execute the following command to install SonarQube Server if it is not ready:
|
||||
|
||||
```bash
|
||||
helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort
|
||||
```
|
||||
```bash
|
||||
helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort
|
||||
```
|
||||
|
||||
2. You will get this prompt:
|
||||
|
||||
|
||||
|
||||
|
||||
## Get Address of SonarQube Console
|
||||
|
||||
1. Execute the following command to get SonarQube NodePort.
|
||||
|
||||
```bash
|
||||
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
|
||||
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
|
||||
echo http://$NODE_IP:$NODE_PORT
|
||||
```
|
||||
```bash
|
||||
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
|
||||
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
|
||||
echo http://$NODE_IP:$NODE_PORT
|
||||
```
|
||||
|
||||
2. You can get the output as below (`31434` is the port number in this example, which may be different from yours):
|
||||
|
||||
```bash
|
||||
http://192.168.0.4:31434
|
||||
```
|
||||
```bash
|
||||
http://192.168.0.4:31434
|
||||
```
|
||||
|
||||
## Configure SonarQube Server
|
||||
|
||||
|
|
@ -48,101 +48,101 @@ http://192.168.0.4:31434
|
|||
|
||||
1. Execute the following command to view the status of SonarQube. Note that the SonarQube console is not accessible until SonarQube is up and running.
|
||||
|
||||
```bash
|
||||
$ kubectl get pod -n kubesphere-devops-system
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
ks-jenkins-68b8949bb-7zwg4 1/1 Running 0 84m
|
||||
s2ioperator-0 1/1 Running 1 84m
|
||||
sonarqube-postgresql-0 1/1 Running 0 5m31s
|
||||
sonarqube-sonarqube-bb595d88b-97594 1/1 Running 2 5m31s
|
||||
uc-jenkins-update-center-8c898f44f-m8dz2 1/1 Running 0 85m
|
||||
```
|
||||
```bash
|
||||
$ kubectl get pod -n kubesphere-devops-system
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
ks-jenkins-68b8949bb-7zwg4 1/1 Running 0 84m
|
||||
s2ioperator-0 1/1 Running 1 84m
|
||||
sonarqube-postgresql-0 1/1 Running 0 5m31s
|
||||
sonarqube-sonarqube-bb595d88b-97594 1/1 Running 2 5m31s
|
||||
uc-jenkins-update-center-8c898f44f-m8dz2 1/1 Running 0 85m
|
||||
```
|
||||
|
||||
2. Access the SonarQube console `http://{$Node IP}:{$NodePort}` in your browser and you can see its homepage as below:
|
||||
|
||||
|
||||
|
||||
|
||||
3. Click **Log in** in the top right corner and use the default account `admin/admin`.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
You may need to set up necessary port forwarding rules and open the port to access SonarQube in your security groups depending on where your instances are deployed.
|
||||
You may need to set up necessary port forwarding rules and open the port to access SonarQube in your security groups depending on where your instances are deployed.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
### Step 2: Create SonarQube Admin Token
|
||||
|
||||
1. Click the letter **A** and select **My Account** from the menu to go to the **Profile** page.
|
||||
|
||||
|
||||
|
||||
|
||||
2. Click **Security** and input a token name, such as `kubesphere`.
|
||||
|
||||
|
||||
|
||||
|
||||
3. Click **Generate** and copy the token.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice warning >}}
|
||||
{{< notice warning >}}
|
||||
|
||||
Make sure you do copy the token because you won't be able to see it again as shown in the prompt.
|
||||
Make sure you do copy the token because you won't be able to see it again as shown in the prompt.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
### Step 3: Create a Webhook Server
|
||||
|
||||
1. Execute the following command to get the address of SonarQube Webhook.
|
||||
|
||||
```bash
|
||||
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
|
||||
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
|
||||
echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/
|
||||
```
|
||||
```bash
|
||||
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
|
||||
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
|
||||
echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/
|
||||
```
|
||||
|
||||
2. Expected output:
|
||||
|
||||
```bash
|
||||
http://192.168.0.4:30180/sonarqube-webhook/
|
||||
```
|
||||
```bash
|
||||
http://192.168.0.4:30180/sonarqube-webhook/
|
||||
```
|
||||
|
||||
3. Click **Administration**, **Configuration** and **Webhooks** in turn to create a webhook.
|
||||
|
||||
|
||||
|
||||
|
||||
4. Click **Create**.
|
||||
|
||||
|
||||
|
||||
|
||||
5. Input **Name** and **Jenkins Console URL** (i.e. the SonarQube Webhook address) in the dialogue that appears. Click **Create** to finish.
|
||||
|
||||
|
||||
|
||||
|
||||
### Step 4: Add SonarQube Configuration to ks-installer
|
||||
|
||||
1. Execute the following command to edit `ks-installer`.
|
||||
|
||||
```bash
|
||||
kubectl edit cc -n kubesphere-system ks-installer
|
||||
```
|
||||
```bash
|
||||
kubectl edit cc -n kubesphere-system ks-installer
|
||||
```
|
||||
|
||||
2. Navigate to `devops`. Add the field `sonarqube` and specify `externalSonarUrl` and `externalSonarToken` under it.
|
||||
|
||||
```yaml
|
||||
devops:
|
||||
enabled: true
|
||||
jenkinsJavaOpts_MaxRAM: 2g
|
||||
jenkinsJavaOpts_Xms: 512m
|
||||
jenkinsJavaOpts_Xmx: 512m
|
||||
jenkinsMemoryLim: 2Gi
|
||||
jenkinsMemoryReq: 1500Mi
|
||||
jenkinsVolumeSize: 8Gi
|
||||
sonarqube: # Add this field manually.
|
||||
externalSonarUrl: http://192.168.0.4:31434 # The SonarQube IP address.
|
||||
externalSonarToken: f75dc3be11fd3d58debfd4e445e3de844683ad93 # The SonarQube admin token created above.
|
||||
```
|
||||
```yaml
|
||||
devops:
|
||||
enabled: true
|
||||
jenkinsJavaOpts_MaxRAM: 2g
|
||||
jenkinsJavaOpts_Xms: 512m
|
||||
jenkinsJavaOpts_Xmx: 512m
|
||||
jenkinsMemoryLim: 2Gi
|
||||
jenkinsMemoryReq: 1500Mi
|
||||
jenkinsVolumeSize: 8Gi
|
||||
sonarqube: # Add this field manually.
|
||||
externalSonarUrl: http://192.168.0.4:31434 # The SonarQube IP address.
|
||||
externalSonarToken: f75dc3be11fd3d58debfd4e445e3de844683ad93 # The SonarQube admin token created above.
|
||||
```
|
||||
|
||||
3. Save the file after you finish.
|
||||
|
||||
|
|
@ -150,43 +150,43 @@ devops:
|
|||
|
||||
1. Execute the following command to get the address of Jenkins.
|
||||
|
||||
```bash
|
||||
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
|
||||
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
|
||||
echo http://$NODE_IP:$NODE_PORT
|
||||
```
|
||||
```bash
|
||||
export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
|
||||
export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
|
||||
echo http://$NODE_IP:$NODE_PORT
|
||||
```
|
||||
|
||||
2. You can get the output as below, which tells you the port number of Jenkins.
|
||||
|
||||
```bash
|
||||
http://192.168.0.4:30180
|
||||
```
|
||||
```bash
|
||||
http://192.168.0.4:30180
|
||||
```
|
||||
|
||||
3. Access Jenkins with the address `http://Public IP:30180`. When KubeSphere is installed, the Jenkins dashboard is also installed by default. Besides, Jenkins is configured with KubeSphere LDAP, which means you can log in Jenkins with KubeSphere accounts (e.g. `admin/P@88w0rd`) directly. For more information about configuring Jenkins, see [Jenkins System Settings](../../../devops-user-guide/how-to-use/jenkins-setting/).
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
You may need to set up necessary port forwarding rules and open the port `30180` to access Jenkins in your security groups depending on where your instances are deployed.
|
||||
You may need to set up necessary port forwarding rules and open the port `30180` to access Jenkins in your security groups depending on where your instances are deployed.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
4. Click **Manage Jenkins** on the left.
|
||||
|
||||
|
||||
|
||||
|
||||
5. Scroll down to **Configure System** and click it.
|
||||
|
||||
|
||||
|
||||
|
||||
6. Navigate to **SonarQube servers** and click **Add SonarQube**.
|
||||
|
||||
|
||||
|
||||
|
||||
7. Input **Name**, **Server URL** (`http://Node IP:port`) and **Server authentication token** (the SonarQube admin token). Click **Apply** to finish.
|
||||
|
||||
|
||||
|
||||
|
||||
### Step 6: Add sonarqubeUrl to KubeSphere Console
|
||||
|
||||
|
|
@ -194,22 +194,22 @@ You need to specify `sonarqubeURL` so that you can access SonarQube directly fro
|
|||
|
||||
1. Execute the following command:
|
||||
|
||||
```bash
|
||||
kubectl edit cm -n kubesphere-system ks-console-config
|
||||
```
|
||||
```bash
|
||||
kubectl edit cm -n kubesphere-system ks-console-config
|
||||
```
|
||||
|
||||
2. Navigate to `client` and add the field `devops` with `sonarqubeURL` specified.
|
||||
|
||||
```yaml
|
||||
client:
|
||||
version:
|
||||
kubesphere: v3.0.0
|
||||
kubernetes: v1.17.9
|
||||
openpitrix: v0.3.5
|
||||
enableKubeConfig: true
|
||||
devops: # Add this field manually.
|
||||
sonarqubeURL: http://192.168.0.4:31434 # The SonarQube IP address.
|
||||
```
|
||||
```bash
|
||||
client:
|
||||
version:
|
||||
kubesphere: v3.0.0
|
||||
kubernetes: v1.17.9
|
||||
openpitrix: v0.3.5
|
||||
enableKubeConfig: true
|
||||
devops: # Add this field manually.
|
||||
sonarqubeURL: http://192.168.0.4:31434 # The SonarQube IP address.
|
||||
```
|
||||
|
||||
3. Save the file.
|
||||
|
||||
|
|
@ -231,26 +231,26 @@ You need a SonarQube token so that your pipeline can communicate with SonarQube
|
|||
|
||||
1. On the SonarQube console, click **Create new project**.
|
||||
|
||||
|
||||
|
||||
|
||||
2. Enter a project key, such as `java-demo`, and click **Set Up**.
|
||||
|
||||
|
||||
|
||||
|
||||
3. Enter a project name, such as `java-sample`, and click **Generate**.
|
||||
|
||||
|
||||
|
||||
|
||||
4. After the token is created, click **Continue**.
|
||||
|
||||
|
||||
|
||||
|
||||
5. Choose **Java** and **Maven** respectively. Copy the serial number within the green box in the image below, which needs to be added in the [Credentials](../../../devops-user-guide/how-to-use/credential-management/#create-credentials) section if it is to be used in pipelines.
|
||||
|
||||
|
||||
|
||||
|
||||
## View Results on KubeSphere Console
|
||||
|
||||
After you [create a pipeline using the graphical editing panel](../../how-to-use/create-a-pipeline-using-graphical-editing-panel) or [create a pipeline using a Jenkinsfile](../../how-to-use/create-a-pipeline-using-jenkinsfile), you can view the result of code quality analysis. For example, you may see an image as below if SonarQube runs successfully.
|
||||
|
||||
|
||||
|
||||
|
|
@ -50,7 +50,7 @@ In the current version, there are 4 types of built-in podTemplates, i.e. `base`,
|
|||
|
||||
You can use the built-in podTemplate by specifying the label for an agent. For example, to use the nodejs podTemplate, you can set the label to `nodejs` when creating the Pipeline, as shown in the example below.
|
||||
|
||||
|
||||
|
||||
|
||||
```groovy
|
||||
pipeline {
|
||||
|
|
|
|||
|
|
@ -49,60 +49,60 @@ There are eight stages as shown below in this example pipeline.
|
|||
|
||||
1. Log in the KubeSphere console as `project-regular`. Go to your DevOps project and create the following credentials in **Credentials** under **Project Management**. For more information about how to create credentials, see [Credential Management](../../../devops-user-guide/how-to-use/credential-management/).
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
If there are any special characters such as `@` and `$` in your account or password, they can cause errors as a pipeline runs because they may not be recognized. In this case, you need to encode your account or password on some third-party websites first, such as [urlencoder](https://www.urlencoder.org/). After that, copy and paste the output for your credential information.
|
||||
If there are any special characters such as `@` and `$` in your account or password, they can cause errors as a pipeline runs because they may not be recognized. In this case, you need to encode your account or password on some third-party websites first, such as [urlencoder](https://www.urlencoder.org/). After that, copy and paste the output for your credential information.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
|Credential ID| Type | Where to use |
|
||||
| --- | --- | --- |
|
||||
| dockerhub-id | Account Credentials | Docker Hub |
|
||||
| github-id | Account Credentials | GitHub |
|
||||
| demo-kubeconfig | kubeconfig | Kubernetes |
|
||||
| Credential ID | Type | Where to use |
|
||||
| --------------- | ------------------- | ------------ |
|
||||
| dockerhub-id | Account Credentials | Docker Hub |
|
||||
| github-id | Account Credentials | GitHub |
|
||||
| demo-kubeconfig | kubeconfig | Kubernetes |
|
||||
|
||||
2. You need to create an additional credential ID (`sonar-token`) for SonarQube, which is used in stage 3 (SonarQube analysis) mentioned above. Refer to [Create SonarQube Token for New Project](../../../devops-user-guide/how-to-integrate/sonarqube/#create-sonarqube-token-for-new-project) to use the token for the **secret** field below. Click **OK** to finish.
|
||||
|
||||
|
||||
|
||||
|
||||
3. In total, you have four credentials in the list.
|
||||
|
||||
|
||||
|
||||
|
||||
### Step 2: Modify Jenkinsfile in GitHub Repository
|
||||
|
||||
1. Log in GitHub. Fork [devops-java-sample](https://github.com/kubesphere/devops-java-sample) from the GitHub repository to your own GitHub account.
|
||||
|
||||
|
||||
|
||||
|
||||
2. In your own GitHub repository of **devops-java-sample**, click the file `Jenkinsfile-online` in the root directory.
|
||||
|
||||
|
||||
|
||||
|
||||
3. Click the edit icon on the right to edit environment variables.
|
||||
|
||||
|
||||
|
||||
|
||||
| Items | Value | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| DOCKER\_CREDENTIAL\_ID | dockerhub-id | The **Credential ID** you set in KubeSphere for your Docker Hub account. |
|
||||
| GITHUB\_CREDENTIAL\_ID | github-id | The **Credential ID** you set in KubeSphere for your GitHub account. It is used to push tags to your GitHub repository. |
|
||||
| KUBECONFIG\_CREDENTIAL\_ID | demo-kubeconfig | The **Credential ID** you set in KubeSphere for your kubeconfig. It is used to access a running Kubernetes cluster. |
|
||||
| REGISTRY | docker.io | It defaults to `docker.io`, serving as the address of pushing images. |
|
||||
| DOCKERHUB\_NAMESPACE | your-dockerhub-account | Replace it with your Docker Hub's account name. It can be the Organization name under the account. |
|
||||
| GITHUB\_ACCOUNT | your-github-account | Replace it with your GitHub account name. For example, your GitHub account name is `kubesphere` if your GitHub address is `https://github.com/kubesphere/`. It can also be the account's Organization name. |
|
||||
| APP\_NAME | devops-java-sample | The application name. |
|
||||
| SONAR\_CREDENTIAL\_ID | sonar-token | The **Credential ID** you set in KubeSphere for the SonarQube token. It is used for code quality test. |
|
||||
| Items | Value | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| DOCKER\_CREDENTIAL\_ID | dockerhub-id | The **Credential ID** you set in KubeSphere for your Docker Hub account. |
|
||||
| GITHUB\_CREDENTIAL\_ID | github-id | The **Credential ID** you set in KubeSphere for your GitHub account. It is used to push tags to your GitHub repository. |
|
||||
| KUBECONFIG\_CREDENTIAL\_ID | demo-kubeconfig | The **Credential ID** you set in KubeSphere for your kubeconfig. It is used to access a running Kubernetes cluster. |
|
||||
| REGISTRY | docker.io | It defaults to `docker.io`, serving as the address of pushing images. |
|
||||
| DOCKERHUB\_NAMESPACE | your-dockerhub-account | Replace it with your Docker Hub's account name. It can be the Organization name under the account. |
|
||||
| GITHUB\_ACCOUNT | your-github-account | Replace it with your GitHub account name. For example, your GitHub account name is `kubesphere` if your GitHub address is `https://github.com/kubesphere/`. It can also be the account's Organization name. |
|
||||
| APP\_NAME | devops-java-sample | The application name. |
|
||||
| SONAR\_CREDENTIAL\_ID | sonar-token | The **Credential ID** you set in KubeSphere for the SonarQube token. It is used for code quality test. |
|
||||
|
||||
{{< notice note >}}
|
||||
|
||||
The command parameter `-o` of Jenkinsfile's `mvn` indicates that the offline mode is enabled. Relevant dependencies have already been downloaded in this tutorial to save time and to adapt to network interference in certain environments. The offline mode is on by default.
|
||||
|
||||
{{</ notice >}}
|
||||
{{< notice note >}}
|
||||
|
||||
The command parameter `-o` of Jenkinsfile's `mvn` indicates that the offline mode is enabled. Relevant dependencies have already been downloaded in this tutorial to save time and to adapt to network interference in certain environments. The offline mode is on by default.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
4. After you edit the environmental variables, click **Commit changes** at the bottom of the page, which updates the file in the SonarQube branch.
|
||||
|
||||
|
||||
|
||||
|
||||
### Step 3: Create Projects
|
||||
|
||||
|
|
@ -116,200 +116,201 @@ The account `project-admin` needs to be created in advance since it is the revie
|
|||
|
||||
1. Use the account `project-admin` to log in KubeSphere. In the same workspace where you create the DevOps project, create two projects as below. Make sure you invite `project-regular` to these two projects with the role of `operator`.
|
||||
|
||||
| Project Name | Alias |
|
||||
| ---------------------- | ----------------------- |
|
||||
| kubesphere-sample-dev | development environment |
|
||||
| kubesphere-sample-prod | production environment |
|
||||
| Project Name | Alias |
|
||||
| ---------------------- | ----------------------- |
|
||||
| kubesphere-sample-dev | development environment |
|
||||
| kubesphere-sample-prod | production environment |
|
||||
|
||||
2. Check the project list. You have two projects and one DevOps project as below:
|
||||
|
||||
|
||||
|
||||
|
||||
### Step 4: Create a Pipeline
|
||||
|
||||
1. Log out of KubeSphere and log back in as `project-regular`. Go to the DevOps project `demo-devops` and click **Create** to build a new pipeline.
|
||||
|
||||
|
||||
|
||||
|
||||
2. Provide the basic information in the dialogue that appears. Name it `jenkinsfile-in-scm` and select a code repository.
|
||||
|
||||
|
||||
|
||||
|
||||
3. In the tab **GitHub**, click **Get Token** to generate a new GitHub token if you do not have one. Paste the token to the box and click **Confirm**.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
4. Choose your GitHub account. All the repositories related to this token will be listed on the right. Select **devops-java-sample** and click **Select this repo**. Click **Next** to continue.
|
||||
|
||||
|
||||
|
||||
|
||||
5. In **Advanced Settings**, check the box next to **Discard old branch**. In this tutorial, you can use the default value of **Days to keep old branches** and **Maximum number branches to keep**.
|
||||
|
||||
|
||||
|
||||
|
||||
Discarding old branches means that you will discard the branch record all together. The branch record includes console output, archived artifacts and other relevant metadata of specific branches. Fewer branches mean that you can save the disk space that Jenkins is using. KubeSphere provides two options to determine when old branches are discarded:
|
||||
Discarding old branches means that you will discard the branch record all together. The branch record includes console output, archived artifacts and other relevant metadata of specific branches. Fewer branches mean that you can save the disk space that Jenkins is using. KubeSphere provides two options to determine when old branches are discarded:
|
||||
|
||||
- **Days to keep old branches**. Branches will be discarded after a certain number of days.
|
||||
- **Maximum number of branches to keep**. The oldest branches will be discarded after branches reach a certain amount.
|
||||
- Days to keep old branches. Branches will be discarded after a certain number of days.
|
||||
|
||||
{{< notice note >}}
|
||||
- Maximum number of branches to keep. The oldest branches will be discarded after branches reach a certain amount.
|
||||
|
||||
**Days to keep old branches** and **Maximum number of branches to keep** apply to branches at the same time. As long as a branch meets the condition of either field, it will be discarded. For example, if you specify 2 as the number of retention days and 3 as the maximum number of branches, any branches that exceed either number will be discarded. KubeSphere repopulates these two fields with -1 by default, which means deleted branches will be discarded.
|
||||
|
||||
{{</ notice >}}
|
||||
{{< notice note >}}
|
||||
|
||||
**Days to keep old branches** and **Maximum number of branches to keep** apply to branches at the same time. As long as a branch meets the condition of either field, it will be discarded. For example, if you specify 2 as the number of retention days and 3 as the maximum number of branches, any branches that exceed either number will be discarded. KubeSphere repopulates these two fields with -1 by default, which means deleted branches will be discarded.
|
||||
|
||||
{{</ notice >}}
|
||||
|
||||
6. In **Behavioral strategy**, KubeSphere offers three strategies by default. You can delete **Discover PR from Forks** as this strategy will not be used in this example. You do not need to change the setting and can use the default value directly.
|
||||
|
||||
|
||||
|
||||
|
||||
As a Jenkins pipeline runs, the Pull Request (PR) submitted by developers will also be regarded as a separate branch.
|
||||
As a Jenkins pipeline runs, the Pull Request (PR) submitted by developers will also be regarded as a separate branch.
|
||||
|
||||
**Discover Branches**
|
||||
**Discover Branches**
|
||||
|
||||
- **Exclude branches that are also filed as PRs**. The source branch is not scanned such as the origin's master branch. These branches need to be merged.
|
||||
- **Only branches that are also filed as PRs**. Only scan the PR branch.
|
||||
- **All branches**. Pull all the branches from the repository origin.
|
||||
- **Exclude branches that are also filed as PRs**. The source branch is not scanned such as the origin's master branch. These branches need to be merged.
|
||||
- **Only branches that are also filed as PRs**. Only scan the PR branch.
|
||||
- **All branches**. Pull all the branches from the repository origin.
|
||||
|
||||
**Discover PR from Origin**
|
||||
**Discover PR from Origin**
|
||||
|
||||
- **Source code version of PR merged with target branch**. A pipeline is created and runs based on the source code after the PR is merged into the target branch.
|
||||
- **Source code version of PR itself**. A pipeline is created and runs based on the source code of the PR itself.
|
||||
- **Two pipelines are created when a PR is discovered**. KubeSphere creates two pipelines, one based on the source code after the PR is merged into the target branch, and the other based on the source code of the PR itself.
|
||||
- **Source code version of PR merged with target branch**. A pipeline is created and runs based on the source code after the PR is merged into the target branch.
|
||||
- **Source code version of PR itself**. A pipeline is created and runs based on the source code of the PR itself.
|
||||
- **Two pipelines are created when a PR is discovered**. KubeSphere creates two pipelines, one based on the source code after the PR is merged into the target branch, and the other based on the source code of the PR itself.
|
||||
|
||||
7. Scroll down to **Script Path**. The field specifies the Jenkinsfile path in the code repository. It indicates the repository's root directory. If the file location changes, the script path also needs to be changed. Please change it to `Jenkinsfile-online`, which is the file name of Jenkinsfile in the example repository located in the root directory.
|
||||
|
||||
|
||||
|
||||
|
||||
8. In **Scan Repo Trigger**, check **If not, scan regularly** and set the interval to **5 minutes**. Click **Create** to finish.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
You can set a specific interval to allow pipelines to scan remote repositories, so that any code updates or new PRs can be detected based on the strategy you set in **Behavioral strategy**.
|
||||
You can set a specific interval to allow pipelines to scan remote repositories, so that any code updates or new PRs can be detected based on the strategy you set in **Behavioral strategy**.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
### Step 5: Run a Pipeline
|
||||
|
||||
1. After a pipeline is created, it displays in the list below. Click it to go to its detail page.
|
||||
|
||||
|
||||
|
||||
|
||||
2. Under **Activity**, three branches are being scanned. Click **Run** on the right and the pipeline runs based on the behavioral strategy you set. Select **sonarqube** from the drop-down list and add a tag number such as `v0.0.2`. Click **OK** to trigger a new activity.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
- If you do need see any activity on this page, you need to refresh your browser manually or click **Scan Repository** from the drop-down menu (the **More** button).
|
||||
- The tag name is used to generate releases and images with the tag in GitHub and Docker Hub. An existing tag name cannot be used again for the field TAG_NAME. Otherwise, the pipeline will not be running successfully.
|
||||
- If you do need see any activity on this page, you need to refresh your browser manually or click **Scan Repository** from the drop-down menu (the **More** button).
|
||||
- The tag name is used to generate releases and images with the tag in GitHub and Docker Hub. An existing tag name cannot be used again for the field TAG_NAME. Otherwise, the pipeline will not be running successfully.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
3. Wait for a while and you can see some activities stop and some fail. Click the first one to view details.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
Activity failures may be caused by different factors. In this example, only the Jenkinsfile of the branch sonarqube is changed as you edit the environment variables in it in the steps above. On the contrary, these variables in the dependency and master branch remain changed (namely, wrong GitHub and Docker Hub account), resulting in the failure. You can click it and inspect its logs to see details. Other reasons for failures may be network issues, incorrect coding in the Jenkinsfile and so on.
|
||||
Activity failures may be caused by different factors. In this example, only the Jenkinsfile of the branch sonarqube is changed as you edit the environment variables in it in the steps above. On the contrary, these variables in the dependency and master branch remain changed (namely, wrong GitHub and Docker Hub account), resulting in the failure. You can click it and inspect its logs to see details. Other reasons for failures may be network issues, incorrect coding in the Jenkinsfile and so on.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
4. The pipeline pauses at the stage `deploy to dev`. You need to click **Proceed** manually. Note that the pipeline will be reviewed three times as `deploy to dev`, `push with tag`, and `deploy to production` are defined in the Jenkinsfile respectively.
|
||||
|
||||
|
||||
|
||||
|
||||
In a development or production environment, it requires someone who has higher authority (e.g. release manager) to review the pipeline, images, as well as the code analysis result. They have the authority to determine whether the pipeline can go to the next stage. In the Jenkinsfile, you use the section `input` to specify who reviews the pipeline. If you want to specify a user (e.g. `project-admin`) to review it, you can add a field in the Jenkinsfile. If there are multiple users, you need to use commas to separate them as follows:
|
||||
In a development or production environment, it requires someone who has higher authority (e.g. release manager) to review the pipeline, images, as well as the code analysis result. They have the authority to determine whether the pipeline can go to the next stage. In the Jenkinsfile, you use the section `input` to specify who reviews the pipeline. If you want to specify a user (e.g. `project-admin`) to review it, you can add a field in the Jenkinsfile. If there are multiple users, you need to use commas to separate them as follows:
|
||||
|
||||
```groovy
|
||||
···
|
||||
input(id: 'release-image-with-tag', message: 'release image with tag?', submitter: 'project-admin,project-admin1')
|
||||
···
|
||||
```
|
||||
```groovy
|
||||
···
|
||||
input(id: 'release-image-with-tag', message: 'release image with tag?', submitter: 'project-admin,project-admin1')
|
||||
···
|
||||
```
|
||||
|
||||
### Step 6: Check Pipeline Status
|
||||
|
||||
1. In **Task Status**, you can see how a pipeline is running. Please note that the pipeline will keep initializing for several minutes after it is just created. There are eight stages in the sample pipeline and they have been defined separately in [Jenkinsfile-online](https://github.com/kubesphere/devops-java-sample/blob/sonarqube/Jenkinsfile-online).
|
||||
|
||||
|
||||
|
||||
|
||||
2. Check the pipeline running logs by clicking **Show Logs** in the top right corner. You can see the dynamic log output of the pipeline, including any errors that may stop the pipeline from running. For each stage, you click it to inspect logs, which can be downloaded to your local machine for further analysis.
|
||||
|
||||
|
||||
|
||||
|
||||
### Step 7: Verify Results
|
||||
|
||||
1. Once you successfully executed the pipeline, click **Code Quality** to check the results through SonarQube as follows.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
2. The Docker image built through the pipeline has also been successfully pushed to Docker Hub, as it is defined in the Jenkinsfile. In Docker Hub, you will find the image with the tag `v0.0.2` that is specified before the pipeline runs.
|
||||
|
||||
|
||||
|
||||
|
||||
3. At the same time, a new tag and a new release have been generated in GitHub.
|
||||
|
||||
|
||||
|
||||
|
||||
4. The sample application will be deployed to `kubesphere-sample-dev` and `kubesphere-sample-prod` with corresponding Deployments and Services created. Go to these two projects and here are the expected result:
|
||||
|
||||
| Environment | URL | Namespace | Deployment | Service |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| Development | `http://{NodeIP}:{$30861}` | kubesphere-sample-dev | ks-sample-dev | ks-sample-dev |
|
||||
| Production | `http://{$NodeIP}:{$30961}` | kubesphere-sample-prod | ks-sample | ks-sample |
|
||||
| Environment | URL | Namespace | Deployment | Service |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| Development | `http://{NodeIP}:{$30861}` | kubesphere-sample-dev | ks-sample-dev | ks-sample-dev |
|
||||
| Production | `http://{$NodeIP}:{$30961}` | kubesphere-sample-prod | ks-sample | ks-sample |
|
||||
|
||||
#### Deployments
|
||||
#### Deployments
|
||||
|
||||
|
||||
|
||||
|
||||
#### Services
|
||||
#### Services
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
You may need to open the port in your security groups so that you can access the app with the URL.
|
||||
You may need to open the port in your security groups so that you can access the app with the URL.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
### Step 8: Access Sample Service
|
||||
|
||||
1. To access the service, log in KubeSphere as `admin` to use the **web kubectl** from **Toolbox**. Go to the project `kubesphere-sample-dev`, and select `ks-sample-dev` in **Services** under **Application Workloads**. The endpoint can be used to access the service.
|
||||
1. To access the Service, log in KubeSphere as `admin` to use the **web kubectl** from **Toolbox**. Go to the project `kubesphere-sample-dev`, and select `ks-sample-dev` in **Services** under **Application Workloads**. The endpoint can be used to access the Service.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
2. Use the **web kubectl** from **Toolbox** in the bottom right corner by executing the following command:
|
||||
|
||||
```bash
|
||||
$ curl 10.10.128.169:8080
|
||||
```
|
||||
```bash
|
||||
$ curl 10.10.128.169:8080
|
||||
```
|
||||
|
||||
3. Expected output:
|
||||
|
||||
```bash
|
||||
Really appreciate your star, that's the power of our life.
|
||||
```
|
||||
```bash
|
||||
Really appreciate your star, that's the power of our life.
|
||||
```
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
Use `curl` endpoints or {$Virtual IP}:{$Port} or {$Node IP}:{$NodePort}
|
||||
Use `curl` endpoints or {$Virtual IP}:{$Port} or {$Node IP}:{$NodePort}
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
4. Similarly, you can test the service in the project `kubesphere-sample-prod` and you will see the same result.
|
||||
4. Similarly, you can test the Service in the project `kubesphere-sample-prod` and you will see the same result.
|
||||
|
||||
```bash
|
||||
$ curl 10.10.128.170:8080
|
||||
Really appreciate your star, that's the power of our life.
|
||||
```
|
||||
```bash
|
||||
$ curl 10.10.128.170:8080
|
||||
Really appreciate your star, that's the power of our life.
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -12,14 +12,14 @@ A DevOps project user with necessary permissions can configure credentials for J
|
|||
|
||||
Currently, you can store the following 4 types of credentials in a DevOps project:
|
||||
|
||||
|
||||
|
||||
|
||||
- **Account Credentials**: Username and password which can be handled as separate components or as a colon-separated string in the format `username:password`, such as accounts of GitHub, GitLab, and Docker Hub.
|
||||
- **SSH**: Username with a private key, an SSH public/private key pair.
|
||||
- **Secret Text**: Secret content in a file.
|
||||
- **kubeconfig**: It is used to configure cross-cluster authentication. If you select this type, the dialogue will auto-populate the field with the kubeconfig file of the current Kubernetes cluster.
|
||||
|
||||
This tutorial demonstrates how to create and manage credentials in a DevOps project.
|
||||
This tutorial demonstrates how to create and manage credentials in a DevOps project. For more information about how credentials are used, see [Create a Pipeline Using a Jenkinsfile](../create-a-pipeline-using-jenkinsfile/) and [Create a Pipeline Using Graphical Editing Panels](../create-a-pipeline-using-graphical-editing-panel).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
|
|
@ -30,19 +30,19 @@ This tutorial demonstrates how to create and manage credentials in a DevOps proj
|
|||
|
||||
Log in the console of KubeSphere as `project-regular`. Navigate to your DevOps project, choose **Credentials** and click **Create**.
|
||||
|
||||
|
||||
|
||||
|
||||
### Create Docker Hub Credentials
|
||||
|
||||
1. In the dialogue that appears, provide the following information.
|
||||
|
||||
|
||||
|
||||
|
||||
- **Credential ID**: Set an ID, such as `dockerhub-id`, which can be used in pipelines.
|
||||
- **Type**: Select **Account Credentials**.
|
||||
- **Username**: Your Docker Hub account (i.e Docker ID).
|
||||
- **Token/Password**: Your Docker Hub password.
|
||||
- **Description**: A brief introduction to the credentials.
|
||||
- **Credential ID**: Set an ID, such as `dockerhub-id`, which can be used in pipelines.
|
||||
- **Type**: Select **Account Credentials**.
|
||||
- **Username**: Your Docker Hub account (i.e Docker ID).
|
||||
- **Token/Password**: Your Docker Hub password.
|
||||
- **Description**: A brief introduction to the credentials.
|
||||
|
||||
2. Click **OK** when you finish.
|
||||
|
||||
|
|
@ -70,15 +70,18 @@ A file that is used to configure access to clusters is called a kubeconfig file.
|
|||
|
||||
1. Credentials created display in the list as below.
|
||||
|
||||
|
||||
|
||||
|
||||
2. Click any of them to go to its detail page, where you can see account details and all the events related to the credentials.
|
||||
|
||||
|
||||
|
||||
|
||||
3. You can also edit or delete credentials on this page. Note that when you edit credentials, KubeSphere does not display the existing username or password information. The previous one will be overwritten if you enter a new username and password.
|
||||
|
||||
|
||||
|
||||
|
||||
For more information about how credentials are used, see [Create a Pipeline Using Jenkinsfile](../create-a-pipeline-using-jenkinsfile/) and [Create a Pipeline Using Graphical Editing Panel](../create-a-pipeline-using-graphical-editing-panel).
|
||||
## See Also
|
||||
|
||||
[Create a Pipeline Using a Jenkinsfile](../create-a-pipeline-using-jenkinsfile/)
|
||||
|
||||
[Create a Pipeline Using Graphical Editing Panels](../create-a-pipeline-using-graphical-editing-panel)
|
||||
|
|
@ -18,28 +18,29 @@ The built-in Jenkins cannot share the same email configuration with the platform
|
|||
|
||||
1. Click **Platform** in the top left corner and select **Clusters Management**.
|
||||
|
||||
|
||||
|
||||
|
||||
2. If you have enabled the [multi-cluster feature](../../../multicluster-management) with member clusters imported, you can select a specific cluster to view its nodes. If you have not enabled the feature, refer to the next step directly.
|
||||
|
||||
3. Go to **Workloads** under **Application Workloads**, and choose the project **kubesphere-devops-system** from the drop-down list. Click the three dots on the right of **ks-jenkins** to edit its YAML.
|
||||
|
||||
|
||||
|
||||
|
||||
4. Scroll down to the fields in the image below which you need to specify. Click **Update** when you finish to save changes.
|
||||
|
||||
{{< notice warning >}}
|
||||
{{< notice warning >}}
|
||||
|
||||
Once you modify the Email server in the `ks-jenkins` Deployment, it will restart itself. Consequently, the DevOps system will be unavailable for a few minutes. Please make such modification at an appropriate time.
|
||||
Once you modify the Email server in the `ks-jenkins` Deployment, it will restart itself. Consequently, the DevOps system will be unavailable for a few minutes. Please make such modification at an appropriate time.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
|
||||
|
||||
|
||||
| Environment Variable Name | Description |
|
||||
|---|---|
|
||||
|EMAIL\_SMTP\_HOST | SMTP server address |
|
||||
|EMAIL\_SMTP\_PORT | SMTP server port (e.g. 25) |
|
||||
|EMAIL\_FROM\_ADDR | Email sender address |
|
||||
|EMAIL\_FROM\_NAME | Email sender name |
|
||||
|EMAIL\_FROM\_PASS | Email sender password |
|
||||
|EMAIL\_USE\_SSL | SSL configuration enabled or not |
|
||||
| Environment Variable Name | Description |
|
||||
| ------------------------- | -------------------------------- |
|
||||
| EMAIL\_SMTP\_HOST | SMTP server address |
|
||||
| EMAIL\_SMTP\_PORT | SMTP server port (e.g. 25) |
|
||||
| EMAIL\_FROM\_ADDR | Email sender address |
|
||||
| EMAIL\_FROM\_NAME | Email sender name |
|
||||
| EMAIL\_FROM\_PASS | Email sender password |
|
||||
| EMAIL\_USE\_SSL | SSL configuration enabled or not |
|
||||
|
|
|
|||
|
|
@ -33,51 +33,53 @@ In **Project Roles**, there are three available built-in roles as shown below. B
|
|||
|
||||
1. Log in the console as `devops-admin` and select a DevOps project (e.g. `demo-devops`) under **DevOps Projects** list.
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
The account `devops-admin` is used as an example. As long as the account you are using is granted a role including the authorization of **Project Members View**, **Project Roles Management** and **Project Roles View** in **Access Control** at DevOps project level, it can create a DevOps project role.
|
||||
The account `devops-admin` is used as an example. As long as the account you are using is granted a role including the authorization of **Project Members View**, **Project Roles Management** and **Project Roles View** in **Access Control** at DevOps project level, it can create a DevOps project role.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
2. Go to **Project Roles** in **Project Management**, click **Create** and set a **Role Identifier**. In this example, a role named `pipeline-creator` will be created. Click **Edit Authorization** to continue.
|
||||
|
||||
|
||||
|
||||
|
||||
3. In **Pipelines Management**, select the authorization that you want the user granted this role to have. For example, **Pipelines Management** and **Pipelines View** are selected for this role. Click **OK** to finish.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
**Depend on** means the major authorization (the one listed after **Depend on**) needs to be selected first so that the affiliated authorization can be assigned.
|
||||
**Depend on** means the major authorization (the one listed after **Depend on**) needs to be selected first so that the affiliated authorization can be assigned.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
4. Newly-created roles will be listed in **Project Roles**. You can click the three dots on the right to edit it.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
The role of `pipeline-creator` is only granted **Pipelines Management** and **Pipelines View**, which may not satisfy your need. This example is only for demonstration purpose. You can create customized roles based on your needs.
|
||||
The role of `pipeline-creator` is only granted **Pipelines Management** and **Pipelines View**, which may not satisfy your need. This example is only for demonstration purpose. You can create customized roles based on your needs.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
## Invite a New Member
|
||||
|
||||
1. In **Project Management**, select **Project Members** and click **Invite Member**.
|
||||
|
||||
2. Invite a user to the DevOps project. Grant the role of `pipeline-creator` to the user.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
The user must be invited to the DevOps project's workspace first.
|
||||
The user must be invited to the DevOps project's workspace first.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
3. After you add a user to the DevOps project, click **OK**. In **Project Members**, you can see the newly invited member listed.
|
||||
|
||||
4. You can also change the role of an existing member by editing it or remove it from the DevOps project.
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -18,26 +18,27 @@ You need an account granted a role including the authorization of **Clusters Man
|
|||
|
||||
1. Click **Platform** in the top left corner and select **Clusters Management**.
|
||||
|
||||
|
||||
|
||||
|
||||
2. If you have enabled the [multi-cluster feature](../../../multicluster-management) with member clusters imported, you can select a specific cluster to view its nodes. If you have not enabled the feature, refer to the next step directly.
|
||||
|
||||
3. Navigate to **Cluster Nodes** under **Nodes**, where you can see existing nodes in the current cluster.
|
||||
|
||||
|
||||
|
||||
|
||||
4. Choose a node from the list to run CI tasks. For example, select `node2` here and click it to go to its detail page. Click **More** and select **Edit Labels**.
|
||||
|
||||
|
||||
|
||||
|
||||
5. In the dialogue that appears, click **Add Labels**. Add a new label with key `node-role.kubernetes.io/worker` and value `ci`, and click **Save**.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice note >}}
|
||||
{{< notice note >}}
|
||||
|
||||
The node may already have the key without a value. You can input the value `ci` directly.
|
||||
The node may already have the key without a value. You can input the value `ci` directly.
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
||||
## Add a Taint to a CI Node
|
||||
|
||||
|
|
@ -45,18 +46,18 @@ Basically, pipelines and S2I/B2I workflows will be scheduled to this node accord
|
|||
|
||||
1. Click **More** and select **Taint Management**.
|
||||
|
||||
|
||||
|
||||
|
||||
2. Click **Add Taint** and enter a key `node.kubernetes.io/ci` without specifying a value. You can choose `NoSchedule` or `PreferNoSchedule` based on your needs.
|
||||
|
||||
|
||||
|
||||
|
||||
3. Click **Save**. KubeSphere will schedule tasks according to the taint you set. You can go back to work on your DevOps pipeline now.
|
||||
|
||||
|
||||
|
||||
|
||||
{{< notice tip >}}
|
||||
{{< notice tip >}}
|
||||
|
||||
This tutorial also covers the operation related to node management. For detailed information, see [Node Management](../../../cluster-administration/nodes/).
|
||||
This tutorial also covers the operation related to node management. For detailed information, see [Node Management](../../../cluster-administration/nodes/).
|
||||
|
||||
{{</ notice >}}
|
||||
{{</ notice >}}
|
||||
|
|
|
|||
|
Before Width: | Height: | Size: 91 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 50 KiB After Width: | Height: | Size: 50 KiB |
|
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 96 KiB |
|
Before Width: | Height: | Size: 87 KiB After Width: | Height: | Size: 87 KiB |
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 79 KiB After Width: | Height: | Size: 79 KiB |
|
Before Width: | Height: | Size: 95 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 36 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 48 KiB |
|
Before Width: | Height: | Size: 28 KiB After Width: | Height: | Size: 28 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 89 KiB After Width: | Height: | Size: 89 KiB |
|
Before Width: | Height: | Size: 92 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 81 KiB After Width: | Height: | Size: 81 KiB |
|
Before Width: | Height: | Size: 75 KiB After Width: | Height: | Size: 75 KiB |
|
Before Width: | Height: | Size: 91 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 88 KiB After Width: | Height: | Size: 88 KiB |
|
Before Width: | Height: | Size: 95 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 62 KiB After Width: | Height: | Size: 62 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 97 KiB |
|
Before Width: | Height: | Size: 98 KiB After Width: | Height: | Size: 98 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 86 KiB After Width: | Height: | Size: 86 KiB |
|
Before Width: | Height: | Size: 92 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 71 KiB After Width: | Height: | Size: 71 KiB |
|
Before Width: | Height: | Size: 75 KiB After Width: | Height: | Size: 75 KiB |
|
Before Width: | Height: | Size: 79 KiB After Width: | Height: | Size: 79 KiB |
|
Before Width: | Height: | Size: 88 KiB After Width: | Height: | Size: 88 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 80 KiB After Width: | Height: | Size: 80 KiB |
|
Before Width: | Height: | Size: 92 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 88 KiB After Width: | Height: | Size: 88 KiB |
|
Before Width: | Height: | Size: 82 KiB After Width: | Height: | Size: 82 KiB |
|
Before Width: | Height: | Size: 92 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 96 KiB |
|
Before Width: | Height: | Size: 95 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 86 KiB After Width: | Height: | Size: 86 KiB |
|
Before Width: | Height: | Size: 26 KiB After Width: | Height: | Size: 26 KiB |
|
Before Width: | Height: | Size: 86 KiB After Width: | Height: | Size: 86 KiB |
|
Before Width: | Height: | Size: 82 KiB After Width: | Height: | Size: 82 KiB |
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 91 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 91 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 56 KiB After Width: | Height: | Size: 56 KiB |
|
Before Width: | Height: | Size: 51 KiB After Width: | Height: | Size: 51 KiB |
|
Before Width: | Height: | Size: 51 KiB After Width: | Height: | Size: 51 KiB |
|
Before Width: | Height: | Size: 82 KiB After Width: | Height: | Size: 82 KiB |
|
Before Width: | Height: | Size: 82 KiB After Width: | Height: | Size: 82 KiB |
|
Before Width: | Height: | Size: 85 KiB After Width: | Height: | Size: 85 KiB |
|
Before Width: | Height: | Size: 50 KiB After Width: | Height: | Size: 50 KiB |
|
Before Width: | Height: | Size: 51 KiB After Width: | Height: | Size: 51 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 56 KiB After Width: | Height: | Size: 56 KiB |
|
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 96 KiB |
|
Before Width: | Height: | Size: 93 KiB After Width: | Height: | Size: 93 KiB |
|
After Width: | Height: | Size: 56 KiB |
|
Before Width: | Height: | Size: 87 KiB After Width: | Height: | Size: 87 KiB |
|
Before Width: | Height: | Size: 104 KiB After Width: | Height: | Size: 104 KiB |
|
Before Width: | Height: | Size: 69 KiB After Width: | Height: | Size: 69 KiB |
|
Before Width: | Height: | Size: 69 KiB After Width: | Height: | Size: 69 KiB |
|
Before Width: | Height: | Size: 39 KiB After Width: | Height: | Size: 39 KiB |