From 29609deffcf233002d44b7f258c547a5b099539b Mon Sep 17 00:00:00 2001 From: Huang Junxiang Date: Tue, 3 Jun 2025 16:37:58 +0800 Subject: [PATCH] Update configurations for S3 (#529) --- manual/config/env.md | 19 +- manual/extension/metadata-server.md | 33 ++- manual/repo/docker/cluster/env | 2 +- manual/repo/docker/cluster/seafile-server.yml | 2 +- manual/repo/docker/md-server.yml | 3 +- manual/repo/docker/metadata-server/env | 3 +- .../repo/docker/metadata-server/md-server.yml | 3 +- manual/repo/docker/pro/env | 40 +-- manual/repo/docker/pro/seafile-server.yml | 2 +- manual/repo/docker/pro/seasearch.yml | 2 +- .../cluster/seafile-backend-deployment.yaml | 4 +- manual/repo/k8s/cluster/seafile-env.yaml | 3 +- .../cluster/seafile-frontend-deployment.yaml | 4 +- manual/repo/k8s/pro/seafile-deployment.yaml | 9 +- manual/repo/k8s/pro/seafile-env.yaml | 2 +- manual/setup/setup_pro_by_docker.md | 19 +- manual/setup/setup_with_s3.md | 254 ++++++++++++------ manual/setup/use_seasearch.md | 23 +- 18 files changed, 272 insertions(+), 155 deletions(-) diff --git a/manual/config/env.md b/manual/config/env.md index 4c06ef9d..16a61f7f 100644 --- a/manual/config/env.md +++ b/manual/config/env.md @@ -71,9 +71,9 @@ This part of configurations is only valid in `CACHE_PROVIDER=memcached`: - `ENABLE_SEADOC`: Enable the SeaDoc server or not, default is `false`. - `SEADOC_SERVER_URL`: Only valid in `ENABLE_SEADOC=true`. Url of Seadoc server (e.g., http://seafile.example.com/sdoc-server). -## S3 storage backend configurations +## S3 storage backend configurations (pro) -- `USE_S3_STORAGE`: Use S3 backend storage in Seafile server +- `SEAF_SERVER_STORAGE_TYPE`: What kind of the Seafile data for storage. Available options are `disk` (i.e., local disk), `s3` and `multiple` (see the details of [multiple storage backends](../setup/setup_with_multiple_storage_backends.md)) - `S3_COMMIT_BUCKET`: S3 storage backend fs objects bucket - `S3_FS_BUCKET`: S3 storage backend block objects bucket - `S3_BLOCK_BUCKET`: S3 storage backend block objects bucket @@ -89,9 +89,18 @@ This part of configurations is only valid in `CACHE_PROVIDER=memcached`: - `S3_SSE_C_KEY`: A string of 32 characters can be generated by openssl rand -base64 24. It can be any 32-character long random string. It's required to use V4 authentication protocol and https if you enable SSE-C. !!! success "Easier to configure S3 for Seafile and its components" - Since Seafile Pro 13.0, in order to facilitate users to deploy Seafile's related extension components and other services in the future, a section will be provided in `.env` to store the **S3 Configurations** for Seafile and some extension components (such as *SeaSearch*, *Metadata server*). You can locate it with the title bar **\#\#S3**. - - However, please note that Seafile only support configuring S3 in `.env` for **single storage backend mode** (i.e., when `USE_S3_STORAGE=true`), if you would like to use multiple backends, please set `USE_S3_STORAGE=false` in `.env` and refer [here](../setup/setup_with_multiple_storage_backends.md) for the details. + Since Seafile Pro 13.0, in order to facilitate users to deploy Seafile's related extension components and other services in the future, a section will be provided in `.env` to store the **S3 Configurations** for Seafile and some extension components (such as *SeaSearch*, *Metadata server*). You can locate it with the title bar **Storage configurations for S3**. + +!!! warning "S3 configurations in `.env` only support single S3 storage backend mode" + The Seafile server only support configuring S3 in `.env` for **single S3 storage backend mode** (i.e., when `SEAF_SERVER_STORAGE_TYPE=s3`). If you would like to use other storage backend (e.g., [Ceph](./setup_with_ceph.md), [Swift](./setup_with_swift.md)) or other settings that can only be set in `seafile.conf` (like [multiple storage backends](./setup_with_multiple_storage_backends.md)), please set `SEAF_SERVER_STORAGE_TYPE` to `multiple`, and set `MD_STORAGE_TYPE` and `SS_STORAGE_TYPE` according to your configurations. + +!!! note "The S3 configurations only valid with at least one `STORAGE_TYPE` has specified to `s3`" + Now there are three (pro) and one (cluster) ***STORAGE_TYPE*** we provided in `.env`: + - SEAF_SERVER_STORAGE_TYPE (pro & cluster) + - MD_STORAGE_TYPE (pro, see the [Metadata server](#metadata-server) section for the details) + - SS_STORAGE_TYPE (pro, see the [SeaSearch](#seasearch) section for the details) + + You have to specify at least one of them as s3 for the above configuration to take effect. ## SeaSearch diff --git a/manual/extension/metadata-server.md b/manual/extension/metadata-server.md index f8169905..60a4a222 100644 --- a/manual/extension/metadata-server.md +++ b/manual/extension/metadata-server.md @@ -54,29 +54,25 @@ MD_MAX_CACHE_SIZE=1GB #### Example `.env` for Seafile data is stored in the storage backend (e.g., S3) -First you need to create a bucket for Metadata on your S3 storage backend provider. Then add or modify the following information to `.env`: - -!!! success "Easier to configure S3 for Seafile and its components" - Since Seafile Pro 13.0, in order to facilitate users to deploy Seafile's related extension components and other services in the future, a section will be provided in `.env` to store the **S3 Configurations** for Seafile and some extension components (such as *SeaSearch*, *Metadata server*). You can locate it with the title bar **\#\#S3**. - - In other words, if you deploy SeaSearch and Seafile together, and if you have deployed Seafile Pro following [here](../setup/setup_pro_by_docker.md#downloading-and-modifying-env) (and using the latest `.env`), you only need to specify the following variables in `.env` to make it work (that is, the `USE_S3_STORAGE` is set to `true`): - - ```sh - S3_COMMIT_BUCKET= - S3_FS_BUCKET= - S3_BLOCK_BUCKET= - S3_MD_BUCKET= - ``` +First you need to create a bucket for metadata on your S3 storage backend provider. Then add or modify the following information to `.env`: ```sh MD_IMAGE=seafileltd/seafile-md-server:latest -USE_S3_STORAGE=true -S3_COMMIT_BUCKET=... -S3_FS_BUCKET=... -S3_BLOCK_BUCKET=... +MD_STORAGE_TYPE=s3 S3_MD_BUCKET=... +S3_KEY_ID= +S3_SECRET_KEY= +S3_USE_V4_SIGNATURE=true +S3_PATH_STYLE_REQUEST=false +S3_AWS_REGION=us-east-1 +S3_HOST= +S3_USE_HTTPS=true +S3_SSE_C_KEY= ``` +!!! note "Data for Seafile server should be accessible for Metadata server" + In order to correctly obtain metadata information, you must ensure that the data of your Seafile server can be correctly accessed. In the case of deploying Metadata server and Seafile server together, Metadata server will be able to automatically obtain the configuration information of Seafile server, so you don't need to worry about this. But if your Metadata server is deployed in Standalone (usually in a cluster environment), then you need to ensure that the description of the Seafile server storage part in the `.env` deployed by Metadata server needs to be consistent with the `.env` deployed by Seafile server (e.g., `SEAF_SERVER_STORAGE_TYPE`), and can access the configuration file information of Seafile server (e.g., `seafile.conf`) to ensure that Metadata server can correctly obtain data from Seafile server. + #### List of environment variables for Metadata server The following table is all the related environment variables with Metadata server: @@ -87,7 +83,8 @@ The following table is all the related environment variables with Metadata serve | `REDIS_HOST` | Your *Redis* service host. | Optional, default `redis` | | `REDIS_PORT` | Your *Redis* service port. | Optional, default `6379` | | `REDIS_PASSWORD` | Your *Redis* access password. | Optional | -| `S3_MD_BUCKET` | Your S3 bucket name for the bucket storing metadata | Required when using S3 (`USE_S3_STORAGE` = `true`) | +| `MD_STORAGE_TYPE` | Where the metadata storage in. Available options are `disk` (local storage) and `s3` | `disk` | +| `S3_MD_BUCKET` | Your S3 bucket name for the bucket storing metadata | Required when using S3 (`MD_STORAGE_TYPE=s3`) | In addition, there are some environment variables **related to S3 authorization**, please refer to the part with `S3_` prefix in this [table](../setup/setup_pro_by_docker.md#downloading-and-modifying-env) (**the buckets name for Seafile are also needed**). diff --git a/manual/repo/docker/cluster/env b/manual/repo/docker/cluster/env index 2fa63ccb..75bb8280 100644 --- a/manual/repo/docker/cluster/env +++ b/manual/repo/docker/cluster/env @@ -32,7 +32,7 @@ MEMCACHED_HOST= MEMCACHED_PORT=11211 ## S3 -USE_S3_STORAGE=false +SEAF_SERVER_STORAGE_TYPE=disk S3_KEY_ID= S3_SECRET_KEY= S3_USE_V4_SIGNATURE=true diff --git a/manual/repo/docker/cluster/seafile-server.yml b/manual/repo/docker/cluster/seafile-server.yml index 7827750c..2bd0fdd6 100644 --- a/manual/repo/docker/cluster/seafile-server.yml +++ b/manual/repo/docker/cluster/seafile-server.yml @@ -28,7 +28,7 @@ services: - TIME_ZONE=${TIME_ZONE:-UTC} - INIT_SEAFILE_ADMIN_EMAIL=${INIT_SEAFILE_ADMIN_EMAIL:-me@example.com} - INIT_SEAFILE_ADMIN_PASSWORD=${INIT_SEAFILE_ADMIN_PASSWORD:-asecret} - - USE_S3_STORAGE=${USE_S3_STORAGE:-false} + - SEAF_SERVER_STORAGE_TYPE=${SEAF_SERVER_STORAGE_TYPE:-disk} - S3_COMMIT_BUCKET=${S3_COMMIT_BUCKET:-} - S3_FS_BUCKET=${S3_FS_BUCKET:-} - S3_BLOCK_BUCKET=${S3_BLOCK_BUCKET:-} diff --git a/manual/repo/docker/md-server.yml b/manual/repo/docker/md-server.yml index a00b6673..8bcd5488 100644 --- a/manual/repo/docker/md-server.yml +++ b/manual/repo/docker/md-server.yml @@ -17,7 +17,8 @@ services: - MD_PORT=${MD_PORT:-8084} - MD_LOG_LEVEL=${MD_LOG_LEVEL:-info} - MD_MAX_CACHE_SIZE=${MD_MAX_CACHE_SIZE:-1GB} - - USE_S3_STORAGE=${USE_S3_STORAGE:-false} + - SEAF_SERVER_STORAGE_TYPE=${SEAF_SERVER_STORAGE_TYPE:-disk} + - MD_STORAGE_TYPE=${MD_STORAGE_TYPE:-disk} - S3_COMMIT_BUCKET=${S3_COMMIT_BUCKET:-} - S3_FS_BUCKET=${S3_FS_BUCKET:-} - S3_BLOCK_BUCKET=${S3_BLOCK_BUCKET:-} diff --git a/manual/repo/docker/metadata-server/env b/manual/repo/docker/metadata-server/env index ad94d259..a230449a 100644 --- a/manual/repo/docker/metadata-server/env +++ b/manual/repo/docker/metadata-server/env @@ -15,7 +15,8 @@ SEAFILE_MYSQL_DB_SEAFILE_DB_NAME=seafile_db SEAFILE_LOG_TO_STDOUT=false # for s3 configurations -USE_S3_STORAGE=false +SEAF_SERVER_STORAGE_TYPE=disk +MD_STORAGE_TYPE=$SEAF_SERVER_STORAGE_TYPE S3_COMMIT_BUCKET= S3_FS_BUCKET= S3_BLOCK_BUCKET= diff --git a/manual/repo/docker/metadata-server/md-server.yml b/manual/repo/docker/metadata-server/md-server.yml index f6e4cd3d..15bdf12b 100644 --- a/manual/repo/docker/metadata-server/md-server.yml +++ b/manual/repo/docker/metadata-server/md-server.yml @@ -17,7 +17,8 @@ services: - MD_PORT=${MD_PORT:-8084} - MD_LOG_LEVEL=${MD_LOG_LEVEL:-info} - MD_MAX_CACHE_SIZE=${MD_MAX_CACHE_SIZE:-1GB} - - USE_S3_STORAGE=${USE_S3_STORAGE:-false} + - SEAF_SERVER_STORAGE_TYPE=${SEAF_SERVER_STORAGE_TYPE:-disk} + - MD_STORAGE_TYPE=${MD_STORAGE_TYPE:-disk} - S3_COMMIT_BUCKET=${S3_COMMIT_BUCKET:-} - S3_FS_BUCKET=${S3_FS_BUCKET:-} - S3_BLOCK_BUCKET=${S3_BLOCK_BUCKET:-} diff --git a/manual/repo/docker/pro/env b/manual/repo/docker/pro/env index 1c9cc020..b92b714f 100644 --- a/manual/repo/docker/pro/env +++ b/manual/repo/docker/pro/env @@ -16,7 +16,14 @@ NOTIFICATION_SERVER_IMAGE=seafileltd/notification-server:12.0-latest SEASEARCH_IMAGE=seafileltd/seasearch:1.0-latest MD_IMAGE=seafileltd/seafile-md-server:latest -## Persistent Storage +## Storage + +### Storage type +SEAF_SERVER_STORAGE_TYPE=disk # disk, s3, multiple +MD_STORAGE_TYPE=$SEAF_SERVER_STORAGE_TYPE # disk, s3 +SS_STORAGE_TYPE=$SEAF_SERVER_STORAGE_TYPE # disk, s3 + +### Local storage path BASIC_STORAGE_PATH=/opt SEAFILE_VOLUME=$BASIC_STORAGE_PATH/seafile-data SEAFILE_MYSQL_VOLUME=$BASIC_STORAGE_PATH/seafile-mysql/db @@ -26,6 +33,21 @@ NOTIFICATION_SERVER_VOLUME=$BASIC_STORAGE_PATH/notification-data SS_DATA_PATH=$BASIC_STORAGE_PATH/seasearch-data SEADOC_VOLUME=$BASIC_STORAGE_PATH/seadoc-data +### Storage configurations for S3 (only valid in XXX_STORAGE_TYPE=s3) +S3_COMMIT_BUCKET= +S3_FS_BUCKET= +S3_BLOCK_BUCKET= +S3_SS_BUCKET= # for seasearch +S3_MD_BUCKET= # for metadata-server +S3_KEY_ID= +S3_SECRET_KEY= +S3_USE_V4_SIGNATURE=true +S3_PATH_STYLE_REQUEST=false +S3_AWS_REGION=us-east-1 +S3_HOST= +S3_USE_HTTPS=true +S3_SSE_C_KEY= + ################################# # Startup parameters # ################################# @@ -58,22 +80,6 @@ REDIS_PASSWORD= MEMCACHED_HOST=memcached MEMCACHED_PORT=11211 -## S3 -USE_S3_STORAGE=false -S3_COMMIT_BUCKET= -S3_FS_BUCKET= -S3_BLOCK_BUCKET= -S3_SS_BUCKET= # for seasearch -S3_MD_BUCKET= # for metadata-server -S3_KEY_ID= -S3_SECRET_KEY= -S3_USE_V4_SIGNATURE=true -S3_PATH_STYLE_REQUEST=false -S3_AWS_REGION=us-east-1 -S3_HOST= -S3_USE_HTTPS=true -S3_SSE_C_KEY= - ###################################### # Initial variables # # (Only valid in first-time startup) # diff --git a/manual/repo/docker/pro/seafile-server.yml b/manual/repo/docker/pro/seafile-server.yml index 86f92f57..343a70d8 100644 --- a/manual/repo/docker/pro/seafile-server.yml +++ b/manual/repo/docker/pro/seafile-server.yml @@ -69,7 +69,7 @@ services: - REDIS_PASSWORD=${REDIS_PASSWORD:-} - MEMCACHED_HOST=${MEMCACHED_HOST:-memcached} - MEMCACHED_PORT=${MEMCACHED_PORT:-11211} - - USE_S3_STORAGE=${USE_S3_STORAGE:-false} + - SEAF_SERVER_STORAGE_TYPE=${SEAF_SERVER_STORAGE_TYPE:-disk} - S3_COMMIT_BUCKET=${S3_COMMIT_BUCKET:-} - S3_FS_BUCKET=${S3_FS_BUCKET:-} - S3_BLOCK_BUCKET=${S3_BLOCK_BUCKET:-} diff --git a/manual/repo/docker/pro/seasearch.yml b/manual/repo/docker/pro/seasearch.yml index a7ba9179..5d0ba397 100644 --- a/manual/repo/docker/pro/seasearch.yml +++ b/manual/repo/docker/pro/seasearch.yml @@ -8,7 +8,7 @@ services: - SS_FIRST_ADMIN_USER=${INIT_SS_ADMIN_USER:-} - SS_FIRST_ADMIN_PASSWORD=${INIT_SS_ADMIN_PASSWORD:-} - SS_MAX_OBJ_CACHE_SIZE=${SS_MAX_OBJ_CACHE_SIZE:-10GB} - - USE_S3_STORAGE=${USE_S3_STORAGE:-false} + - SS_STORAGE_TYPE=${SS_STORAGE_TYPE:-disk} - S3_SS_BUCKET=${S3_SS_BUCKET:-} - S3_KEY_ID=${S3_KEY_ID:-} - S3_USE_V4_SIGNATURE=${S3_USE_V4_SIGNATURE:-true} diff --git a/manual/repo/k8s/cluster/seafile-backend-deployment.yaml b/manual/repo/k8s/cluster/seafile-backend-deployment.yaml index f059f86c..f0749682 100644 --- a/manual/repo/k8s/cluster/seafile-backend-deployment.yaml +++ b/manual/repo/k8s/cluster/seafile-backend-deployment.yaml @@ -119,11 +119,11 @@ spec: configMapKeyRef: name: seafile-env key: CLUSTER_INIT_ES_PORT - - name: USE_S3_STORAGE + - name: SEAF_SERVER_STORAGE_TYPE valueFrom: configMapKeyRef: name: seafile-env - key: USE_S3_STORAGE + key: SEAF_SERVER_STORAGE_TYPE - name: S3_COMMIT_BUCKET valueFrom: configMapKeyRef: diff --git a/manual/repo/k8s/cluster/seafile-env.yaml b/manual/repo/k8s/cluster/seafile-env.yaml index c7638ca5..cee90893 100644 --- a/manual/repo/k8s/cluster/seafile-env.yaml +++ b/manual/repo/k8s/cluster/seafile-env.yaml @@ -23,8 +23,7 @@ data: MEMCACHED_PORT: "11211" # for s3 - USE_S3_STORAGE: "false" - S3_STORAGE_BACKEND_CONFIG: "false" + SEAF_SERVER_STORAGE_TYPE: "disk" S3_COMMIT_BUCKET: "" S3_FS_BUCKET: "" S3_BLOCK_BUCKET: "" diff --git a/manual/repo/k8s/cluster/seafile-frontend-deployment.yaml b/manual/repo/k8s/cluster/seafile-frontend-deployment.yaml index c98f078e..ccfb63b6 100644 --- a/manual/repo/k8s/cluster/seafile-frontend-deployment.yaml +++ b/manual/repo/k8s/cluster/seafile-frontend-deployment.yaml @@ -104,11 +104,11 @@ spec: configMapKeyRef: name: seafile-env key: MEMCACHED_PORT - - name: USE_S3_STORAGE + - name: SEAF_SERVER_STORAGE_TYPE valueFrom: configMapKeyRef: name: seafile-env - key: USE_S3_STORAGE + key: SEAF_SERVER_STORAGE_TYPE - name: S3_COMMIT_BUCKET valueFrom: configMapKeyRef: diff --git a/manual/repo/k8s/pro/seafile-deployment.yaml b/manual/repo/k8s/pro/seafile-deployment.yaml index c85c1852..f51bc822 100644 --- a/manual/repo/k8s/pro/seafile-deployment.yaml +++ b/manual/repo/k8s/pro/seafile-deployment.yaml @@ -81,16 +81,11 @@ spec: configMapKeyRef: name: seafile-env key: SEAFILE_MYSQL_DB_SEAHUB_DB_NAME - - name: USE_S3_STORAGE + - name: SEAF_SERVER_STORAGE_TYPE valueFrom: configMapKeyRef: name: seafile-env - key: USE_S3_STORAGE - - name: S3_STORAGE_BACKEND_CONFIG - valueFrom: - configMapKeyRef: - name: seafile-env - key: S3_STORAGE_BACKEND_CONFIG + key: SEAF_SERVER_STORAGE_TYPE - name: S3_COMMIT_BUCKET valueFrom: configMapKeyRef: diff --git a/manual/repo/k8s/pro/seafile-env.yaml b/manual/repo/k8s/pro/seafile-env.yaml index c2951562..2b5b110d 100644 --- a/manual/repo/k8s/pro/seafile-env.yaml +++ b/manual/repo/k8s/pro/seafile-env.yaml @@ -31,7 +31,7 @@ data: MEMCACHED_PORT: "11211" # for S3 - USE_S3_STORAGE: "false" + SEAF_SERVER_STORAGE_TYPE: "disk" S3_COMMIT_BUCKET: "" S3_FS_BUCKET: "" S3_BLOCK_BUCKET: "" diff --git a/manual/setup/setup_pro_by_docker.md b/manual/setup/setup_pro_by_docker.md index 8d64a4ea..276e5756 100644 --- a/manual/setup/setup_pro_by_docker.md +++ b/manual/setup/setup_pro_by_docker.md @@ -75,12 +75,12 @@ The following fields merit particular attention: | `TIME_ZONE` | Time zone | `UTC` | | `INIT_SEAFILE_ADMIN_EMAIL` | Synchronously set admin username during initialization | me@example.com | | `INIT_SEAFILE_ADMIN_PASSWORD` | Synchronously set admin password during initialization | asecret | -| `USE_S3_STORAGE` | Use S3 backend storage in Seafile server | `false` | -| `S3_COMMIT_BUCKET` | S3 storage backend commit objects bucket | (required when `USE_S3_STORAGE=true`) | -| `S3_FS_BUCKET` | S3 storage backend fs objects bucket | (required when you would like to initialize with S3) | -| `S3_BLOCK_BUCKET` | S3 storage backend block objects bucket | (required when `USE_S3_STORAGE=true`) | -| `S3_KEY_ID` | S3 storage backend key ID | (required when `USE_S3_STORAGE=true`) | -| `S3_SECRET_KEY` | S3 storage backend secret key | (required when `USE_S3_STORAGE=true`) | +| `SEAF_SERVER_STORAGE_TYPE` | What kind of the Seafile data for storage. Available options are `disk` (i.e., local disk), `s3` and `multiple` (see the details of [multiple storage backends](./setup_with_multiple_storage_backends.md)) | `disk` | +| `S3_COMMIT_BUCKET` | S3 storage backend commit objects bucket | (required when `SEAF_SERVER_STORAGE_TYPE=s3`) | +| `S3_FS_BUCKET` | S3 storage backend fs objects bucket | (required when `SEAF_SERVER_STORAGE_TYPE=s3`) | +| `S3_BLOCK_BUCKET` | S3 storage backend block objects bucket | (required when `SEAF_SERVER_STORAGE_TYPE=s3`) | +| `S3_KEY_ID` | S3 storage backend key ID | (required when `SEAF_SERVER_STORAGE_TYPE=s3`) | +| `S3_SECRET_KEY` | S3 storage backend secret key | (required when `SEAF_SERVER_STORAGE_TYPE=s3`) | | `S3_AWS_REGION` | Region of your buckets | `us-east-1` | | `S3_HOST` | Host of your buckets | (required when not use AWS) | | `S3_USE_HTTPS` | Use HTTPS connections to S3 if enabled | `true` | @@ -91,9 +91,10 @@ The following fields merit particular attention: | `NON_ROOT` | Run Seafile container without a root user | `false` | !!! success "Easier to configure S3 for Seafile and its components" - Since Seafile Pro 13.0, in order to facilitate users to deploy Seafile's related extension components and other services in the future, a section will be provided in `.env` to store the **S3 Configurations** for Seafile and some extension components (such as *SeaSearch*, *Metadata server*). You can locate it with the title bar **\#\#S3**. - - However, please note that Seafile only support configuring S3 in `.env` for **single storage backend mode** (i.e., when `USE_S3_STORAGE=true`), if you would like to use multiple backends, please set `USE_S3_STORAGE=false` in `.env` and refer [here](../setup/setup_with_multiple_storage_backends.md) for the details. + Since Seafile Pro 13.0, in order to facilitate users to deploy Seafile's related extension components and other services in the future, a section will be provided in `.env` to store the **S3 Configurations** for Seafile and some extension components (such as *SeaSearch*, *Metadata server*). You can locate it with the title bar **Storage configurations for S3**. + +!!! warning "S3 configurations in `.env` only support single S3 storage backend mode" + The Seafile server only support configuring S3 in `.env` for **single S3 storage backend mode** (i.e., when `SEAF_SERVER_STORAGE_TYPE=s3`). If you would like to use other storage backend (e.g., [Ceph](./setup_with_ceph.md), [Swift](./setup_with_swift.md)) or other settings that can only be set in `seafile.conf` (like [multiple storage backends](./setup_with_multiple_storage_backends.md)), please set `SEAF_SERVER_STORAGE_TYPE` to `multiple`, and set `MD_STORAGE_TYPE` and `SS_STORAGE_TYPE` according to your configurations. To conclude, set the directory permissions of the Elasticsearch volumne: diff --git a/manual/setup/setup_with_s3.md b/manual/setup/setup_with_s3.md index ce90ca44..25038f9f 100644 --- a/manual/setup/setup_with_s3.md +++ b/manual/setup/setup_with_s3.md @@ -1,60 +1,46 @@ ---- -status: new ---- - # Setup With S3 Storage -!!! note "Deployment notes" - - This feature is only for ***Pro Edition*** - - If your Seafile server is deployed from [binary packages](../setup_binary/installation_pro.md), you have to do the following steps before deploying: +From Seafile 13, there are two ways to configure S3 storage (**single S3 storage backend**) for Seafile server: + +- Environment variables (**recommend** since Seafile 13) +- Config file (`seafile.conf`) + +!!! note "Setup note for binary packages deployment (Pro)" + If your Seafile server is deployed from [binary packages](../setup_binary/installation_pro.md), you have to do the following steps before deploying: - 1. **install `boto3` to your machine** + 1. **install `boto3` to your machine** - ```sh - sudo pip install boto3 - ``` + ```sh + sudo pip install boto3 + ``` - 2. **Install and configure [memcached or Redis](../setup_binary/installation_pro.md#setup-memory-cache)**. - - For best performance, Seafile requires enable memory cache for objects. We recommend to **at least allocate 128MB** memory for memcached or Redis. - - The configuration options differ for different S3 storage. We'll describe the configurations in separate sections. You also need to add [memory cache configurations](../config/seafile-conf.md#cache-pro-edition-only) + 2. **Install and configure [memcached or Redis](../setup_binary/installation_pro.md#setup-memory-cache)**. + For best performance, Seafile requires enable memory cache for objects. We recommend to **at least allocate 128MB** memory for memcached or Redis. -!!! tip "New feature from 13.0 pro edition" - If your will deploy Seafile server in Docker, you can specify your S3 configurations in `S3` title bar in `.env` to deploy with S3 together. This operation will generate the same configuration file as this manual. - - However, please note that Seafile only support configuring S3 in `.env` for **single storage backend mode** (i.e., when `USE_S3_STORAGE=true`), if you would like to use multiple backends, please set `USE_S3_STORAGE=false` in `.env` and refer [here](../setup/setup_with_multiple_storage_backends.md) for the details. + The configuration options differ for different S3 storage. We'll describe the configurations in separate sections. You also need to add [memory cache configurations](../config/seafile-conf.md#cache-pro-edition-only) -## How to configure S3 in Seafile -Seafile configures S3 storage by adding or modifying the following section in `seafile.conf`: +## Setup swith environment variables (recommend) -```conf -[xxx_object_backend] -name = s3 -bucket = my-xxx-objects -key_id = your-key-id -key = your-secret-key -use_v4_signature = true -use_https = true -... ; other optional configurations -``` +From Seafile 13, configuring S3 from environment variables will be supported and will provide a more convenient way. You can refer to the detailed description of this part in the introduction of `.env` file. Generally, -You have to create at least **3** buckets for Seafile, corresponding to the sections: `commit_object_backend`, `fs_object_backend` and `block_backend`. For the configurations for each backend section, please refer to the following table: +1. Prepare at least **3** buckets for Seafile (`S3_COMMIT_BUCKET`, `S3_FS_BUCKET` and `S3_BLOCK_BUCKET`). +2. Set `SEAF_SERVER_STORAGE_TYPE` to `true` +3. Fill in the corresponding variable values in `.env` ​​according to the following table: -| Variable | Description | -| --- | --- | -| `bucket` | Bucket name for commit, fs, and block objects. Make sure it follows [S3 naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/BucketRestrictions.html#bucketnamingrules) (you can refer the notes below the table). | -| `key_id` | The `key_id` is required to authenticate you to S3. You can find the `key_id` in the "security credentials" section on your AWS account page or from your storage provider. | -| `key` | The `key` is required to authenticate you to S3. You can find the `key` in the "security credentials" section on your AWS account page or from your storage provider. | -| `use_v4_signature` | There are two versions of authentication protocols that can be used with S3 storage: Version 2 (older, may still be supported by some regions) and Version 4 (current, used by most regions). If you don't set this option, Seafile will use the v2 protocol. It's suggested to use the v4 protocol. | -| `use_https` | Use https to connect to S3. It's recommended to use https. | -| `aws_region` | (**Optional**) If you use the v4 protocol and AWS S3, set this option to the region you chose when you create the buckets. If it's not set and you're using the v4 protocol, Seafile will use `us-east-1` as the default. This option will be ignored if you use the v2 protocol. | -| `host` | (**Optional**) The endpoint by which you access the storage service. Usually it starts with the region name. It's required to provide the host address if you use storage provider other than AWS, *otherwise Seafile will use AWS's address* (i.e., `s3.us-east-1.amazonaws.com`).| -| `sse_c_key` | (**Optional**) A string of 32 characters can be generated by `openssl rand -base64 24`. It can be any 32-character long random string. It's required to use V4 authentication protocol and https if you enable SSE-C. | -| `path_style_request` | (**Optional**) This option asks Seafile to use URLs like `https://192.168.1.123:8080/bucketname/object` to access objects. In Amazon S3, the default URL format is in virtual host style, such as `https://bucketname.s3.amazonaws.com/object`. But this style relies on advanced DNS server setup. So most self-hosted storage systems only implement the path style format. So we recommend to set this option to true for self-hosted storage. | - -[1]: (Replace this placeholder with the actual link to the S3 bucket naming rules documentation if necessary) +| Variable | Description | Default Value | +| ------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------- | +| `S3_COMMIT_BUCKET` | S3 storage backend commit objects bucket | (required) | +| `S3_FS_BUCKET` | S3 storage backend fs objects bucket | (required) | +| `S3_BLOCK_BUCKET` | S3 storage backend block objects bucket | (required) | +| `S3_KEY_ID` | S3 storage backend key ID | (required) | +| `S3_SECRET_KEY` | S3 storage backend secret key | (required) | +| `S3_AWS_REGION` | Region of your buckets | `us-east-1` | +| `S3_HOST` | Host of your buckets | (required when not use AWS) | +| `S3_USE_HTTPS` | Use HTTPS connections to S3 if enabled | `true` | +| `S3_USE_V4_SIGNATURE` | Use the v4 protocol of S3 if enabled | `true` | +| `S3_PATH_STYLE_REQUEST` | This option asks Seafile to use URLs like `https://192.168.1.123:8080/bucketname/object` to access objects. In *Amazon S3*, the default URL format is in virtual host style, such as `https://bucketname.s3.amazonaws.com/object`. But this style relies on advanced DNS server setup. So most self-hosted storage systems only implement the path style format. | `false` | +| `S3_SSE_C_KEY` | A string of 32 characters can be generated by openssl rand -base64 24. It can be any 32-character long random string. It's required to use V4 authentication protocol and https if you enable SSE-C. | (none) | !!! warning "Bucket naming conventions" @@ -72,47 +58,138 @@ You have to create at least **3** buckets for Seafile, corresponding to the sect - seafileFSObject - seafile block object +!!! note "About S3_SSE_C_KEY" + `S3_SSE_C_KEY` is a string of 32 characters. -### Use server-side encryption with customer-provided keys (SSE-C) in Seafile + You can generate sse_c_key with the following command. Note that the key doesn't have to be base64 encoded. It can be any 32-character long random string. The example just show one possible way to generate such a key. -Since Pro 11.0, you can use SSE-C to S3. Add the following `sse_c_key` to seafile.conf (as shown in the above variables table): + ``` + openssl rand -base64 24 + ``` + + Howevery, if you have existing data in your S3 storage bucket, **turning on the above configuration will make your data inaccessible**. That's because Seafile server doesn't support encrypted and non-encrypted objects mixed in the same bucket. You have to create a new bucket, and migrate your data to it by following [storage backend migration documentation](./migrate_backends_data.md#migrating-to-sse-c-encrypted-s3-storage). + +!!! tip "For other S3 support extensions" + In addition to Seafile server, the following extensions (if already installed) will share the same S3 authorization information in `.env` with Seafile server: + + - [SeaSearch](./use_seasearch.md): Enable the feature by specifying `SS_STORAGE_TYPE=s3` and `S3_SS_BUCKET` + - [Metadata server](../extension/metadata-server.md): Enable the feature by specifying `MD_STORAGE_TYPE=s3` and `S3_MD_BUCKET` + +### Example configurations + +=== "AWS" + + ```sh + SEAF_SERVER_STORAGE_TYPE=s3 + S3_COMMIT_BUCKET=my-commit-objects + S3_FS_BUCKET=my-fs-objects + S3_BLOCK_BUCKET=my-block-objects + S3_KEY_ID=your-key-id + S3_SECRET_KEY=your-secret-key + S3_USE_V4_SIGNATURE=true + S3_PATH_STYLE_REQUEST=false + S3_AWS_REGION=eu-central-1 + S3_HOST= + S3_USE_HTTPS=true + ``` +=== "Exoscale" + + ```conf + SEAF_SERVER_STORAGE_TYPE=s3 + S3_COMMIT_BUCKET=my-commit-objects + S3_FS_BUCKET=my-fs-objects + S3_BLOCK_BUCKET=my-block-objects + S3_KEY_ID=your-key-id + S3_SECRET_KEY=your-secret-key + S3_USE_V4_SIGNATURE=true + S3_PATH_STYLE_REQUEST=true + S3_AWS_REGION=eu-central-1 # will be ignored when S3_HOST is specified + S3_HOST=sos-de-fra-1.exo.io + S3_USE_HTTPS=true + ``` +=== "Hetzner" + + ```conf + SEAF_SERVER_STORAGE_TYPE=s3 + S3_COMMIT_BUCKET=my-commit-objects + S3_FS_BUCKET=my-fs-objects + S3_BLOCK_BUCKET=my-block-objects + S3_KEY_ID=your-key-id + S3_SECRET_KEY=your-secret-key + S3_USE_V4_SIGNATURE=true + S3_PATH_STYLE_REQUEST=true + S3_AWS_REGION=eu-central-1 # will be ignored when S3_HOST is specified + S3_HOST=fsn1.your-objectstorage.com + S3_USE_HTTPS=true + ``` + +=== "Other Public Hosted S3 Storage" + + There are other S3-compatible cloud storage providers in the market, such as ***Blackblaze*** and ***Wasabi***. Configuration for those providers are just a bit different from AWS. We don't assure the following configuration works for all providers. If you have problems please contact our support + + ``` + SEAF_SERVER_STORAGE_TYPE=s3 + S3_COMMIT_BUCKET=my-commit-objects + S3_FS_BUCKET=my-fs-objects + S3_BLOCK_BUCKET=my-block-objects + S3_KEY_ID=your-key-id + S3_SECRET_KEY=your-secret-key + S3_USE_V4_SIGNATURE=true + S3_PATH_STYLE_REQUEST=true + S3_AWS_REGION=eu-central-1 # will be ignored when S3_HOST is specified + S3_HOST= + S3_USE_HTTPS=true + ``` +=== "Self-hosted S3 Storage" + + Many self-hosted object storage systems are now compatible with the S3 API, such as ***OpenStack Swift***, ***Ceph's RADOS Gateway*** and ***Minio***. You can use these S3-compatible storage systems as backend for Seafile. Here is an example config: + + ``` + SEAF_SERVER_STORAGE_TYPE=s3 + S3_COMMIT_BUCKET=my-commit-objects + S3_FS_BUCKET=my-fs-objects + S3_BLOCK_BUCKET=my-block-objects + S3_KEY_ID=your-key-id + S3_SECRET_KEY=your-secret-key + S3_USE_V4_SIGNATURE=true + S3_PATH_STYLE_REQUEST=true + S3_AWS_REGION=eu-central-1 # will be ignored when S3_HOST is specified + S3_HOST=: + S3_USE_HTTPS=true # according to your S3 configuration + ``` + +## Setup with config file + +Seafile configures S3 storage by adding or modifying the following section in `seafile.conf`: ```conf -[commit_object_backend] +[xxx_object_backend] name = s3 -...... +bucket = my-xxx-objects +key_id = your-key-id +key = your-secret-key use_v4_signature = true use_https = true -sse_c_key = XiqMSf3x5ja4LRibBbV0sVntVpdHXl3P - -[fs_object_backend] -name = s3 -...... -use_v4_signature = true -use_https = true -sse_c_key = XiqMSf3x5ja4LRibBbV0sVntVpdHXl3P - -[block_backend] -name = s3 -...... -use_v4_signature = true -use_https = true -sse_c_key = XiqMSf3x5ja4LRibBbV0sVntVpdHXl3P +... ; other optional configurations ``` -`sse_c_key` is a string of 32 characters. +Similar to configure in `.env`, you have to create at least **3** buckets for Seafile too, corresponding to the sections: `commit_object_backend`, `fs_object_backend` and `block_backend`. For the configurations for each backend section, please refer to the following table: -You can generate sse_c_key with the following command. Note that the key doesn't have to be base64 encoded. It can be any 32-character long random string. The example just show one possible way to generate such a key. +| Variable | Description | +| --- | --- | +| `bucket` | Bucket name for commit, fs, and block objects. Make sure it follows [S3 naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/BucketRestrictions.html#bucketnamingrules) (you can refer the notes below the table). | +| `key_id` | The `key_id` is required to authenticate you to S3. You can find the `key_id` in the "security credentials" section on your AWS account page or from your storage provider. | +| `key` | The `key` is required to authenticate you to S3. You can find the `key` in the "security credentials" section on your AWS account page or from your storage provider. | +| `use_v4_signature` | There are two versions of authentication protocols that can be used with S3 storage: Version 2 (older, may still be supported by some regions) and Version 4 (current, used by most regions). If you don't set this option, Seafile will use the v2 protocol. It's suggested to use the v4 protocol. | +| `use_https` | Use https to connect to S3. It's recommended to use https. | +| `aws_region` | (**Optional**) If you use the v4 protocol and AWS S3, set this option to the region you chose when you create the buckets. If it's not set and you're using the v4 protocol, Seafile will use `us-east-1` as the default. This option will be ignored if you use the v2 protocol. | +| `host` | (**Optional**) The endpoint by which you access the storage service. Usually it starts with the region name. It's required to provide the host address if you use storage provider other than AWS, *otherwise Seafile will use AWS's address* (i.e., `s3.us-east-1.amazonaws.com`).| +| `sse_c_key` | (**Optional**) A string of 32 characters can be generated by `openssl rand -base64 24`. It can be any 32-character long random string. It's required to use V4 authentication protocol and https if you enable SSE-C. | +| `path_style_request` | (**Optional**) This option asks Seafile to use URLs like `https://192.168.1.123:8080/bucketname/object` to access objects. In Amazon S3, the default URL format is in virtual host style, such as `https://bucketname.s3.amazonaws.com/object`. But this style relies on advanced DNS server setup. So most self-hosted storage systems only implement the path style format. So we recommend to set this option to true for self-hosted storage. | -``` -openssl rand -base64 24 -``` +[1]: (Replace this placeholder with the actual link to the S3 bucket naming rules documentation if necessary) -!!! warning - - If you have existing data in your S3 storage bucket, turning on the above configuration will make your data inaccessible. That's because Seafile server doesn't support encrypted and non-encrypted objects mixed in the same bucket. You have to create a new bucket, and migrate your data to it by following [storage backend migration documentation](./migrate_backends_data.md#migrating-to-sse-c-encrypted-s3-storage). - -## Example +### Example configurations === "AWS" @@ -278,6 +355,33 @@ openssl rand -base64 24 use_https = true ``` +!!! tip "Use server-side encryption with customer-provided keys (SSE-C) in Seafile" + + Since Pro 11.0, you can use SSE-C to S3. Add the following `sse_c_key` to seafile.conf (as shown in the above variables table): + + ```conf + [commit_object_backend] + name = s3 + ...... + use_v4_signature = true + use_https = true + sse_c_key = XiqMSf3x5ja4LRibBbV0sVntVpdHXl3P + + [fs_object_backend] + name = s3 + ...... + use_v4_signature = true + use_https = true + sse_c_key = XiqMSf3x5ja4LRibBbV0sVntVpdHXl3P + + [block_backend] + name = s3 + ...... + use_v4_signature = true + use_https = true + sse_c_key = XiqMSf3x5ja4LRibBbV0sVntVpdHXl3P + ``` + ## Run and Test ## Now you can start Seafile and test diff --git a/manual/setup/use_seasearch.md b/manual/setup/use_seasearch.md index 8a1e01a3..a50f2bb3 100644 --- a/manual/setup/use_seasearch.md +++ b/manual/setup/use_seasearch.md @@ -38,18 +38,21 @@ SEASEARCH_IMAGE=seafileltd/seasearch:1.0-latest SS_DATA_PATH=/opt/seasearch-data INIT_SS_ADMIN_USER= INIT_SS_ADMIN_PASSWORD= + + +# if you would like to use S3 for saving seasearch data +SS_STORAGE_TYPE=s3 +S3_SS_BUCKET=... +S3_KEY_ID= +S3_SECRET_KEY= +S3_USE_V4_SIGNATURE=true +S3_PATH_STYLE_REQUEST=false +S3_AWS_REGION=us-east-1 +S3_HOST= +S3_USE_HTTPS=true +S3_SSE_C_KEY= ``` -!!! success "Easier to configure S3 for Seafile and its components" - Since Seafile Pro 13.0, in order to facilitate users to deploy Seafile's related extension components and other services in the future, a section will be provided in `.env` to store the **S3 Configurations** for Seafile and some extension components (such as *SeaSearch*, *Metadata server*). You can locate it with the title bar **\#\#S3**. - - In other words, if you deploy SeaSearch and Seafile together, and if you have deployed Seafile Pro following [here](../setup/setup_pro_by_docker.md#downloading-and-modifying-env) (and using the latest `.env`), you only need to specify the following variables in `.env` to make it work (that is, the `USE_S3_STORAGE` is set to `true`): - - ```sh - USE_S3_STORAGE=true - S3_SS_BUCKET= - ``` - ## Modify `seafile-server.yml` to disable `elasticSearch` service If you would like to use *SeaSearch* as the search engine, the `elasticSearch` service can be removed, which is no longer used: remove `elasticsearch.yml` in the list variable `COMPOSE_FILE` on the file `.env`.