From fc5863e5b5cec77e9b017e7363624423be681683 Mon Sep 17 00:00:00 2001 From: Junxiang Huang Date: Mon, 24 Feb 2025 14:48:02 +0800 Subject: [PATCH] fix: metadata deployment --- manual/extension/metadata-server.md | 86 ++++++++++++++++------------- manual/repo/docker/md-server.yml | 2 +- 2 files changed, 50 insertions(+), 38 deletions(-) diff --git a/manual/extension/metadata-server.md b/manual/extension/metadata-server.md index ec071fff..2ace8f3b 100644 --- a/manual/extension/metadata-server.md +++ b/manual/extension/metadata-server.md @@ -5,69 +5,80 @@ Metadata server aims to provide metadata management for your libraries, so as to ## Deployment !!! note "Prerequisites" - The startup of Metadata server requires using ***Redis*** as the cached server (it should be the default cached server in Seafile 13.0). So you must deploy *Redis* for Seafile, then modify [`seafile.conf`](../config/seafile-conf.md#cache-pro-edition-only), [`seahub_settings.py`](https://docs.djangoproject.com/en/4.2/topics/cache/#redis) and [`seafevents.conf`](../config/seafevents-conf.md) to enable it before deploying metadata server. + The startup of Metadata server requires using ***Redis*** as the cache server (it should be the default cache server in Seafile 13.0). So you must deploy *Redis* for Seafile, then modify [`seafile.conf`](../config/seafile-conf.md#cache-pro-edition-only), [`seahub_settings.py`](https://docs.djangoproject.com/en/4.2/topics/cache/#redis) and [`seafevents.conf`](../config/seafevents-conf.md) to enable it before deploying metadata server. -### Download YML file +### Download docker-compose file -Please download YML file **in your `seafile-server.yml` directory** by following command: +Please download the file by following command: === "Deploy in the same machine with Seafile" + !!! note + You have to download this file to the directory same as `seafile-server.yml` + ```sh wget https://manual.seafile.com/13.0/repo/docker/md-server.yml ``` === "Standalone" + !!! note + For standalone deployment, the metadata server only supports Seafile using **S3** and **OSS** as storage backend + ```sh wget https://manual.seafile.com/13.0/repo/docker/metadata-server/md-server.yml wget -O .env https://manual.seafile.com/13.0/repo/docker/metadata-server/env ``` - !!! warning "Important for standalone deployment" - Starting from Seafile 13.0, metadata server supports standalone deployment, but it still needs to maintain access to the original Seafile data directory. **Please make sure that the machine where metadata server is deployed can access the directory normally**. - ### Modify `.env` !!! success "Faster configuration" Metadata-server docker we use an one-time configuration file, which will be generated directly through the environment variable, eliminating the need for you to repeatedly write configuration files (i.e., **no `md-server.conf`**), which will significantly improve your deployment efficiency. -Normally, you only need to set a `METADATA_SERVER_SECRET_KEY` for the metadata server in `.env`, and the metadata of the library will be stored locally (default is `opt/md-data`): - -```env -METADATA_SERVER_SECRET_KEY= -MD_DATA=/opt/md-data -``` - -!!! warning "About `METADATA_SERVER_SECRET_KEY`" - It is used to Seafile connect to metadata server, which can be generated by `pwgen -s 40 1`. However, we **do not recommend** to use the same as the original `JWT_PRIVATE_KEY` in `.env` **for security reasons**. - -The following table is all the related environment variables with metadata-server: +By default, you don't need to add additional variables to your `.env` (except for standalone deployment) to get the metadata server started, because it will read the exact same configuration as the Seafile server (including `JWT_PRIVATE_KEY` ) and keep the repository metadata locally (default `/opt/md-server`). The following table is all the related environment variables with metadata-server: | Variables | Description | Required | | --- | --- | --- | -| `METADATA_SERVER_SECRET_KEY` | The JWT key used to connect with Seafile server | **Required** | -| `MD_STORAGE_TYPE` | The type of backend storage. Options: `file` (local storage), `s3`, `oss`. | Optional, default `file` | -| `MD_DATA` | Directory where local data (only valid in `MD_STORAGE_TYPE=file`) and cache are located. | Optional, default `/opt/md-data` | -| `SEAFILE_VOLUME` | Directory for Seafile data | Optional, default `/opt/seafile-data` | +| `JWT_PRIVATE_KEY` | The JWT key used to connect with Seafile server | **Required** | +| `MD_DATA` | Directory where metadata and cache are located. | Optional, default `/opt/md-data` | | `MD_MAX_CACHE_SIZE` | The maximum cache size. | Optional, default `1GB` | -| `MD_S3_HOST` | Host of s3 backend. | Optional | -| `MD_S3_AWS_REGION` | Region of *AWS* s3 backend. | Optional | -| `MD_S3_USE_HTTPS` | Use https connecting to S3 backend. | Optional, default `true` | -| `MD_S3_BUCKET` | Name of S3 bucket for storaging metadata. | **Required** when `MD_STORAGE_TYPE=s3` | -| `MD_S3_PATH_STYLE_REQUEST` | S3 backend use path style request. | Optional, default `false` | -| `MD_S3_KEY_ID` | S3 backend authorization key ID. | **Required** when `MD_STORAGE_TYPE=s3` | -| `MD_S3_KEY` | S3 backend authorization key secret. | **Required** when `MD_STORAGE_TYPE=s3` | -| `MD_S3_USE_V4_SIGNATURE` | Use V4 signature to S3 storage backend. | Optional, default `true` | -| `MD_S3_SSE_C_KEY` | S3 SSE-C key. | Optional | -| `MD_OSS_HOST` | OSS backend host. | Optional | -| `MD_OSS_REGION` | OSS backend region. | Optional | -| `MD_OSS_BUCKET` | Name of OSS bucket for storaging metadata. | **Required** when `MD_STORAGE_TYPE=oss` | -| `MD_OSS_KEY_ID` | OSS backend authorization key ID. | **Required** when `MD_STORAGE_TYPE=oss` | -| `MD_OSS_KEY` | OSS backend authorization key secret. | **Required** when `MD_STORAGE_TYPE=oss` | +| `MD_STORAGE_TYPE` | The type of Seafile backend storage. Options: `file` (local storage), `s3`, `oss`. | Optional, default `file` | | `REDIS_HOST` | Your *Redis* service host. | Optional, default `redis` | | `REDIS_PORT` | Your *Redis* service port. | Optional, default `6379` | | `REDIS_PASSWORD` | Your *Redis* access password. | Optional | + +And here is other optional values according to your `MD_STORAGE_TYPE` setting: + +- `MD_STORAGE_TYPE=file` (only for deploying the metadata server in the same machine with Seafile) + + | Variables | Description | Required | + | --- | --- | --- | + | `SEAFILE_VOLUME` | Directory for Seafile data | Optional, default `/opt/seafile-data` | + + +- `MD_STORAGE_TYPE=s3` + + | Variables | Description | Required | + | --- | --- | --- | + | `MD_S3_HOST` | Host of s3 backend. | Optional | + | `MD_S3_AWS_REGION` | Region of *AWS* s3 backend. | Optional | + | `MD_S3_USE_HTTPS` | Use https connecting to S3 backend. | Optional, default `true` | + | `MD_S3_BUCKET` | Name of S3 bucket for storaging metadata. | **Required** | + | `MD_S3_PATH_STYLE_REQUEST` | S3 backend use path style request. | Optional, default `false` | + | `MD_S3_KEY_ID` | S3 backend authorization key ID. | **Required** | + | `MD_S3_KEY` | S3 backend authorization key secret. | **Required** | + | `MD_S3_USE_V4_SIGNATURE` | Use V4 signature to S3 storage backend. | Optional, default `true` | + | `MD_S3_SSE_C_KEY` | S3 SSE-C key. | Optional | + +- `MD_STORAGE_TYPE=oss` + + | Variables | Description | Required | + | --- | --- | --- | + | `MD_OSS_HOST` | OSS backend host. | Optional | + | `MD_OSS_REGION` | OSS backend region. | Optional | + | `MD_OSS_BUCKET` | Name of OSS bucket for storaging metadata. | **Required** | + | `MD_OSS_KEY_ID` | OSS backend authorization key ID. | **Required** | + | `MD_OSS_KEY` | OSS backend authorization key secret. | **Required** | !!! tip "More descriptions about *Optional* and *Required* in above table" @@ -84,13 +95,13 @@ To enable metadata server in Seafile, please add the following field in your `se === "Deploy in the same machine with Seafile" ```py ENABLE_METADATA_MANAGEMENT = True - METADATA_SERVER_SECRET_KEY = ' ' + METADATA_SERVER_SECRET_KEY = ' ' METADATA_SERVER_URL = 'http://metadata-server:8084' ``` === "Standalone" ```py ENABLE_METADATA_MANAGEMENT = True - METADATA_SERVER_SECRET_KEY = ' ' + METADATA_SERVER_SECRET_KEY = ' ' METADATA_SERVER_URL = 'http://:8084' ``` @@ -99,6 +110,7 @@ To enable metadata server in Seafile, please add the following field in your `se You can use following command to start metadata server (and the Seafile service also have to restart): ```sh +docker compose down docker compose up -d ``` diff --git a/manual/repo/docker/md-server.yml b/manual/repo/docker/md-server.yml index 993bbd37..6e7b755d 100644 --- a/manual/repo/docker/md-server.yml +++ b/manual/repo/docker/md-server.yml @@ -8,7 +8,7 @@ services: #ports: # - ${MD_PORT:-8084}:${MD_PORT:-8084} environment: - - JWT_PRIVATE_KEY=${METADATA_SERVER_SECRET_KEY:?Variable is not set or empty} + - JWT_PRIVATE_KEY=${JWT_PRIVATE_KEY:?Variable is not set or empty} - SEAFILE_MYSQL_DB_HOST=${SEAFILE_MYSQL_DB_HOST:-db} - SEAFILE_MYSQL_DB_PORT=${SEAFILE_MYSQL_DB_PORT:-3306} - SEAFILE_MYSQL_DB_USER=${SEAFILE_MYSQL_DB_USER:-seafile}