15308555518/ seafile-admin-docs
1
0
Fork 0
mirror of https://github.com/haiwen/seafile-admin-docs.git synced 2025-12-26 02:32:50 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

feat: k8s support routing logs and sending logs to loki

This commit is contained in:
Junxiang Huang 2025-01-21 12:13:35 +08:00
parent 266a99baf5
commit 12cd7dc0fb
1 changed files with 98 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

114
manual/setup/cluster_deploy_with_k8s.md
View File

@ -246,7 +246,7 @@ FILE_SERVER_ROOT = 'https://seafile.example.com/seafhttp'
Similar to [Single-pod Seafile](./k8s_single_node.md), you can browse the log files of Seafile running directly in the persistent volume directory. The difference is that when using K8S to deploy a Seafile cluster (especially in a cloud environment), the persistent volume created is usually shared and synchronized for all nodes. However, ***the logs generated by the Seafile service do not record the specific node information where these logs are located***, so browsing the files in the above folder may make it difficult to identify which node these logs are generated from. Therefore, one solution proposed here is:
1. Record the generated logs to the standard output. In this way, the logs can be distinguished under each node, but all types of logs will be merged together. You can enable this feature (**it should be enabled by default in K8S Seafile cluster but not in K8S single-pod Seafile**) by modifing `SEAFILE_LOG_TO_STDOUT` to `true` in `seafile-env.yaml`:
1. Record the generated logs to the standard output. In this way, the logs can be distinguished under each node by `kubectl logs` (but all types of logs will be output together now). You can enable this feature (**it should be enabled by default in K8S Seafile cluster but not in K8S single-pod Seafile**) by modifing `SEAFILE_LOG_TO_STDOUT` to `true` in `seafile-env.yaml`:
```yaml
...
@ -263,7 +263,7 @@ Similar to [Single-pod Seafile](./k8s_single_node.md), you can browse the log fi
kubectl apply -f /opt/seafile-k8s-yaml/
```
2. Route the standard output logs and re-record them in a new file or upload them to a log aggregation system (e.g., [*Loki*](https://grafana.com/oss/loki/)).
2. Since the logs in step 1 can be distinguished between nodes, but they are aggregated and output together, it is not convenient for log retrieval. So you have to route the standard output logs (i.e., distinguish logs by corresponding components name) and re-record them in a new file or upload them to a log aggregation system (e.g., [*Loki*](https://grafana.com/oss/loki/)).
Currently in the K8S environment, the commonly used log routing plugins are:
@ -313,13 +313,12 @@ data:
@INCLUDE /fluent-bit/etc/seafile/confs/*-log.conf
```
For example in here, we use `/opt/fluent-bit/confs`. What's more, the parsers will be defined in `/opt/fluent-bit/confs/parsers.conf` and for each type log (e.g., *seahub*'s log, *seafevent*'s log) will be defined in `/opt/fluent-bit/confs/*-log.conf`. Each `.conf` file defines several Fluent-Bit data pipeline components:
For example in here, we use `/opt/fluent-bit/confs` (**it has to be non-shared**). What's more, the parsers will be defined in `/opt/fluent-bit/confs/parsers.conf`, and for each type log (e.g., *seahub*'s log, *seafevent*'s log) will be defined in `/opt/fluent-bit/confs/*-log.conf`. Each `.conf` file defines several Fluent-Bit data pipeline components:
| **Pipeline** | **Description** | **Required/Optional** |
| ------------- | --------------- | --------------------- |
| **INPUT** | Specifies where and how Fluent-Bit can get the original log information, and assigns a tag for each log record after read. | Required |
| **PARSER** | Parse the read log records. For K8S Docker runtime logs, they are usually in Json format. | Required |
| **PROCESSOR** | Processes the log records of the specified tag (such as removing unnecessary parts), and assigns a new tag for the records after processed. | Optional |
| **FILTER** | Filters and selects log records with a specified tag, and assigns a new tag to new records. | Optional |
| **OUTPUT** | tells Fluent-Bit what format the log records for the specified tag will be in and where to output them (such as file, *Elasticsearch*, *Loki*, etc.). | Required |
@ -328,7 +327,7 @@ For example in here, we use `/opt/fluent-bit/confs`. What's more, the parsers wi
### Inputer
According to the above, a container will generate a log file (usually in `/var/log/containers/<container-name>-xxxxxx.log`), so you need to prepare an importer and add the following information in `/opt/fluent-bit/confs/seafile-log.conf`:
According to the above, a container will generate a log file (usually in `/var/log/containers/<container-name>-xxxxxx.log`), so you need to prepare an importer and add the following information (for more details, please refer to offical document about [*TAIL inputer*](https://docs.fluentbit.io/manual/pipeline/inputs/tail)) in `/opt/fluent-bit/confs/seafile-log.conf`:
```conf
[INPUT]
@ -339,7 +338,7 @@ According to the above, a container will generate a log file (usually in `/var/l
Docker_Mode On
Docker_Mode_Flush 5
Tag seafile.*
Parser Docker
Parser Docker # for definition, please see the next section as well
[INPUT]
Name tail
@ -352,11 +351,11 @@ According to the above, a container will generate a log file (usually in `/var/l
Parser Docker
```
The above defines two importers, which are used to monitor seafile-frontend and seafile-backend services respectively. The reason why they are written together here is that for a node, you may not know when it will run the frontend service and when it will run the backend service.
The above defines two importers, which are used to monitor seafile-frontend and seafile-backend services respectively. The reason why they are written together here is that for a node, you may not know when it will run the frontend service and when it will run the backend service, but they have the same tag prefix `seafile.`.
### Parser
Each input uses a parser to parse the logs and pass them to the filter. Here, a parser named Docker is created to parse the logs generated by the *K8S-docker-runtime container*. The parser is placed in `/opt/fluent-bit/confs/parser.conf`
Each input has to use a parser to parse the logs and pass them to the filter. Here, a parser named `Docker` is created to parse the logs generated by the *K8S-docker-runtime container*. The parser is placed in `/opt/fluent-bit/confs/parser.conf` (for more details, please refer to offical document about [JSON parser](https://docs.fluentbit.io/manual/pipeline/parsers/json)):
```conf
[PARSER]
@ -367,24 +366,107 @@ Each input uses a parser to parse the logs and pass them to the filter. Here, a
```
!!! tip "Log records after parsing"
The logs of the Docker container are saved in /var/log/containers in **Json** format (see the sample below), which is why we use the `Json` format in the above parser.
```json
{"log":"[seaf-server] [2025-01-17 07:43:48] [INFO] seafile-session.c(86): fileserver: web_token_expire_time = 3600\n","stream":"stdout","time":"2025-01-17T07:43:48.294638442Z"}
{"log":"[seaf-server] [2025-01-17 07:43:48] [INFO] seafile-session.c(98): fileserver: max_index_processing_threads= 3\n","stream":"stdout","time":"2025-01-17T07:43:48.294810145Z"}
{"log":"[seaf-server] [2025-01-17 07:43:48] [INFO] seafile-session.c(111): fileserver: fixed_block_size = 8388608\n","stream":"stdout","time":"2025-01-17T07:43:48.294879777Z"}
{"log":"[seaf-server] [2025-01-17 07:43:48] [INFO] seafile-session.c(123): fileserver: max_indexing_threads = 1\n","stream":"stdout","time":"2025-01-17T07:43:48.295002479Z"}
{"log":"[seaf-server] [2025-01-17 07:43:48] [INFO] seafile-session.c(138): fileserver: put_head_commit_request_timeout = 10\n","stream":"stdout","time":"2025-01-17T07:43:48.295082733Z"}
{"log":"[seaf-server] [2025-01-17 07:43:48] [INFO] seafile-session.c(150): fileserver: skip_block_hash = 0\n","stream":"stdout","time":"2025-01-17T07:43:48.295195843Z"}
{"log":"[seaf-server] [2025-01-17 07:43:48] [INFO] ../common/seaf-utils.c(553): Use database Mysql\n","stream":"stdout","time":"2025-01-17T07:43:48.29704895Z"}
```
When these logs are obtained by the importer and parsed by the parser, they will become independent log records with the following fields:
- `log`: The original log content (i.e., same as you seen in `kubectl logs seafile-xxx`) and an extra line break at the end (i.e., `\n`). **This is also the field we need to save or upload to the log aggregation system in the end**.
- `stream`: The original log come from. `stdout` means the *standard output*.
- `time`: The time when the log is recorded in the corresponding stream (ISO 8601 format).
### Filter
Add a filter in `/opt/fluent-bit/confs/seafile-log.conf` for log routing. Here, the `rewrite_tag` filter is used to route logs according to specific rules:
Add two filters in `/opt/fluent-bit/confs/seafile-log.conf` for records filtering and routing. Here, the [*record_modifier* filter](https://docs.fluentbit.io/manual/pipeline/filters/record-modifier) is to select useful keys (see the contents in above *tip* label, only the `log` field is what we need) in the log records and [*rewrite_tag* filter](https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag) is used to route logs according to specific rules:
```conf
[FILTER]
Name record_modifier
Match seafile.*
Allowlist_key log
[FILTER]
Name rewrite_tag
Match seafile.*
Rule $log ^.*\[seaf-server\].*$ seaf-server false # for seafile's logs
Rule $log ^.*\[seahub\].*$ seahub false # for seahub's logs
Rule $log ^.*\[seafevents\].*$ seafevents false # for seafevents' lgos
Rule $log ^.*\[seafile-slow-rpc\].*$ seafile-slow-rpc false # for slow-rpc's logs
```
The routing method (for details, please refer to [Fluent-Bit](https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag)), is to define routing through `Rule` term, such as
### Output log's to *Loki*
```conf
[FILTER]
...
Rule Key_name Regular New_tag Keep_old_tag_or_not
```
Loki is multi-tenant log aggregation system inspired by *Prometheus*. It is designed to be very cost effective and easy to operate. The Fluent-Bit *loki* built-in output plugin allows you to send your log or events to a *Loki service*. It supports data enrichment with Kubernetes labels, custom label keys and Tenant ID within others.
### Output log's to aggegation system (e.g., *Loki*)
!!! tip "Alternative Fluent-Bit Loki plugin by *Grafana*"
For sending logs to Loki, there are [two plugins](https://grafana.com/docs/loki/latest/send-data/fluentbit/) for Fluent-Bit:
- The [built-in *Loki* plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) maintained by the Fluent-Bit officially, and we will use it in this part because it provides the most complete features.
- [*Grafana-loki* plugin](https://grafana.com/docs/loki/latest/send-data/fluentbit/community-plugin/) maintained by *Grafana Labs*.
Due to each outputer dose not have a distinguishing marks in the configuration files (because Fluent-Bit takes each plugin as a tag workflow):
- ***Seaf-server log***: Add an outputer to `/opt/fluent-bit/confs/seaf-server-log.conf`:
```conf
[OUTPUT]
Name loki
Match seaf-server
Host <your Loki's host>
port <your Loki's port>
labels job=fluentbit, node_name=<your-node-name>, node_id=<your-node-id> # node_name and node_id is optional, but recommended for identifying the source node
```
- ***seahub log***: Add an outputer to `/opt/fluent-bit/confs/seahub-log.conf`:
```conf
[OUTPUT]
Name loki
Match seahub
Host <your Loki's host>
port <your Loki's port>
labels job=fluentbit, node_name=<your-node-name>, node_id=<your-node-id> # node_name and node_id is optional, but recommended for identifying the source node
```
- ***seafevents log***: Add an outputer to `/opt/fluent-bit/confs/seafevents-log.conf`:
```conf
[OUTPUT]
Name loki
Match seafevents
Host <your Loki's host>
port <your Loki's port>
labels job=fluentbit, node_name=<your-node-name>, node_id=<your-node-id> # node_name and node_id is optional, but recommended for identifying the source node
```
- ***seafile-slow-rpc log***: Add an outputer to `/opt/fluent-bit/confs/seafile-slow-rpc-log.conf`:
```conf
[OUTPUT]
Name loki
Match seafile-slow-rpc
Host <your Loki's host>
port <your Loki's port>
labels job=fluentbit, node_name=<your-node-name>, node_id=<your-node-id> # node_name and node_id is optional, but recommended for identifying the source node
```
!!! tip "Cloud Loki instance"
If you are using a cloud Loki instance, you can follow the [Fluent-Bit Loki plugin document](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to fill up all necessary fields. Usually, the following fields are **additional needs** in cloud Loki service:
- `tls`
- `tls.verify`
- `http_user`
- `http_passwd`