Major backend refactor (#599)

Backend refactor
This commit is contained in:
Zef Hemel
2023-12-13 17:52:56 +01:00
committed by GitHub
parent 60d7cc704a
commit 9f082c83a9
42 changed files with 959 additions and 503 deletions
+97
View File
@@ -0,0 +1,97 @@
SilverBullet is primarily configured via environment variables. This page gives a comprehensive overview of all configuration options. You can set these ad-hoc when running the SilverBullet server, or e.g. in your [[Install/Local$docker|docker-compose file]].
# Network
$network
* `SB_HOSTNAME`: Set to the hostname to bind to (defaults to `127.0.0.0`, set to `0.0.0.0` to accept outside connections for the local deno setup, defaults to `0.0.0.0` for docker)
* `SB_PORT`: Sets the port to listen to, e.g. `SB_PORT=1234`, default is `3000`
# Run mode
$runmode
* `SB_SYNC_ONLY`: If you want to run SilverBullet in a mode where the server purely functions as a simple file store, and doesnt index or process content on the server, you can do so by setting this environment variable to `true`. As a result, the client will always run in the Sync [[Client Modes|client mode]].
# Security
$security
SilverBullet enables plugs to run shell commands. This is used by e.g. the [[🔌 Git]] plug to perform git commands. This is potentially unsafe. If you dont need this you can disable this functionality:
* `SB_SHELL_BACKEND`: Enable/disable running of shell commands from plugs, defaults to `local` (enabled), set to `off` to disable. Only enabled when using a local folder for [[$storage]].
# Authentication
$authentication
SilverBullet supports basic authentication for a single user.
* `SB_USER`: Sets single-user credentials, e.g. `SB_USER=pete:1234` allows you to login with username “pete” and password “1234”.
* `SB_AUTH_TOKEN`: Enables `Authorization: Bearer <token>` style authentication on the [[API]] (useful for [[Sync]] and remote HTTP storage back-ends).
# Storage
$storage
SilverBullet support multiple storage back-ends for keeping your [[Space]] content.
## Disk storage
This is default and simplest back-end to use: a folder on disk. It is configured as follows:
* `SB_FOLDER`: Sets the folder to expose. In the docker container this defaults to `/space`.
## AWS S3 bucket storage
It is also possible to use an S3 bucket as storage. For this, you need to create a bucket, create an IAM user and configure access to it appropriately.
Since S3 doesnt support an efficient way to store custom meta data, this mode does require a [[$database]] configuration (see below) to keep all file meta data.
S3 is configured as follows:
* `SB_FOLDER`: Set to `s3://prefix`. `prefix` can be empty, but if set, this will prefix all files with `prefix/` to support multiple spaces being connected to a single bucket.
* `AWS_ACCESS_KEY_ID`: an AWS access key with read/write permissions to the S3 bucket
* `AWS_SECRET_ACCESS_KEY`: an AWS secret access key with read/write permissions to the S3 bucket
* `AWS_BUCKET`: the name of the S3 bucket to use (e.g `my-sb-bucket`)
* `AWS_ENDPOINT`: e.g. `s3.eu-central-1.amazonaws.com`
* `AWS_REGION`: e.g. `eu-central-1`
## Database storage
It is also possible to store space content in the [[$database]]. While not necessarily recommended, it is a viable way to setup a simple deployment of SilverBullet on e.g. [[Install/Deno Deploy]]. Large files will automatically be chunked to avoid limits the used database may have on value size.
This mode is configured as follows:
* `SB_FOLDER`: set to `db://`
The database configured via [[$database]] will be used.
## HTTP storage
While not particularly useful stand-alone (primarily for [[Sync]]), it is possible to store space content on _another_ SilverBullet installation via its [[API]].
This mode is configured as follows:
* `SB_FOLDER`: set to the URL of the other SilverBullet server, e.g. `https://mynotes.mydomain.com`
* `SB_AUTH_TOKEN`: matching the authorization token (configured via [[$authentication]] on the other end) to use for authorization.
# Database
$database
SilverBullet requires a database back-end to (potentially) keep various types of data:
* Indexes for e.g. [[Objects]]
* Storing some encryption related secrets (for [[Authentication]])
* Space content, when the “Database storage” storage back-end is used
Currently only two databases are supported: [Deno KV](https://deno.com/kv), and a dummy in-memory database.
## Deno KV database
When self-hosting SilverBullet (that is: on any other server than on [[Install/Deno Deploy]]), KV uses a local SQLite file to keep data. This efficient and performant.
KV can be configured as follows:
* `SB_DB_BACKEND`: `denokv` (default, so can be omitted)
* `SB_KV_DB`: path to the file name of the (SQLite) database to store data in, defaults to `.silverbullet.db` in the spaces folder (when kept on disk).
When SilverBullet runs on [[Install/Deno Deploy]] it automatically uses its cloud implementation of KV.
## Memory database
The in-memory database is only useful for testing.
* `SB_DB_BACKEND`: `memory`
# Docker
Configuration only relevant to docker deployments:
* `PUID`: Runs the server process with the specified UID (default: whatever user owns the `/space` mapped folder)
* `GUID`: Runs the server process with the specified GID (default: whatever group owns the `/space` mapped folder)
+24 -23
View File
@@ -1,33 +1,34 @@
**Note:** This is a highly experimental setup, take this into account.
> **warning** Experimental
> This setup is not battle tested, use at your own risk
You can deploy SilverBullet to [Deno Deploy](https://deno.com/deploy) for free, and store your data (space) in an S3 bucket.
You can deploy SilverBullet to [Deno Deploy](https://deno.com/deploy) for free, and store space content in [Deno KV](https://deno.com/kv).
This guide assumes you know how to set up the S3 bucket part and get appropriate IAM keys and secrets to access it.
# Steps
Sign up for a (free) [Deno Deploy account](https://dash.deno.com/projects) and “Create an empty project” there.
For the Deno Deploy side:
Jump to the “Settings”, give your project a nicer name and configure the following environment variables:
Sign up for a (free) [Deno Deploy account](https://dash.deno.com/projects) and create a project there.
* `SB_FOLDER`: `db://`
* `SB_PORT`: `8000`
* `SB_SYNC_ONLY`: `1` (Deno Deploy does not currently supports Workers, so running indexing etc. on the server will not work)
* `SB_USER`: (e.g. `pete:letmein`) — this is **super important** otherwise your space will be open to anybody without any authentication
* `SB_AUTH_TOKEN`: (Optional) If you would like to migrate existing content from elsewhere (e.g. a local folder) using [[Sync]], you will want to configure an authentication token here (pick something secure).
Set these environment variables in the project:
Make sure you have [installed Deno locally](https://docs.deno.com/runtime/manual/getting_started/installation) on your machine.
* AWS_ACCESS_KEY_ID
* AWS_SECRET_ACCESS_KEY
* AWS_BUCKET (e.g `my-sb-bucket`)
* AWS_ENDPOINT (e.g. `s3.eu-central-1.amazonaws.com`)
* AWS_REGION (e.g. `eu-central-1`)
* SB_FOLDER (should be `s3://`)
* SB_PORT (should be `8000`)
* SB_USER (e.g. `pete:letmein`) — this is **super important** otherwise your space will be open without any authentication
In your local environment set `DENO_DEPLOY_TOKEN` to your accounts [deploy token](https://dash.deno.com/account#access-tokens).
Install [deployctl](https://deno.com/deploy/docs/deployctl).
Then run:
Then, install `deployctl` via:
```shell
deployctl deploy --prod --include= -p your-project https://silverbullet.md/silverbullet.js
$ deno install -Arf https://deno.land/x/deploy/deployctl.ts
```
And thats it!
To deploy, run:
```shell
deployctl deploy -p=your-project --entrypoint=https://silverbullet.md/silverbullet.js --include= --prod
```
# Migrating and backing up content
If you want to migrate content _from_ or _to_ your new Deploy-based space, you can use [[Sync]]. For this be sure to also configure a `SB_AUTH_TOKEN` variable.
For backup purposes, it may be wise to synchronize your content regularly this way.
+2 -16
View File
@@ -7,6 +7,8 @@ You have two options here:
1. Installation via [[$deno|Deno]] (the awesome JavaScript runtime)
2. Installation via [[$docker|Docker]] (the awesome container runtime)
After choose either, be sure to checkout all [[Install/Configuration]] options as well.
# Installing using Deno
$deno
This consists of two steps (unless [Deno](https://deno.com/) is already installed — in which case were down to one):
@@ -126,19 +128,3 @@ docker-compose logs -f
## Building the docker image
To build your own version of the docker image, run `./scripts/build_docker.sh`.
# Configuration
SilverBullet is partially configured via flags (run it with `--help`) or alternatively via environment variables and partially via a [[SETTINGS]] page in your space.
## Environment variables
$env
You can configure SB with environment variables instead of flags, which is probably what you want to do in a docker setup. The following environment variables are supported:
* `PID`: Runs the server process with the specified UID (default: whatever user owns the `/space` mapped folder)
* `GID`: Runs the server process with the specified GID (default: whatever group owns the `/space` mapped folder)
* `SB_USER`: Sets single-user credentials (like `--user`), e.g. `SB_USER=pete:1234`
* `SB_HOSTNAME`: Set to the hostname to bind to (defaults to `127.0.0.0`, set to `0.0.0.0` to accept outside connections)
* `SB_PORT`: Sets the port to listen to, e.g. `SB_PORT=1234`
* `SB_FOLDER`: Sets the folder to expose, e.g. `SB_FOLDER=/space`
* `SB_SHELL_BACKEND`: Enable/disable running of shell commands from plugs, defaults to "local" (enabled), set to "off" to disable
* `SB_SYNC_ONLY`: Runs the server in a "dumb" space store-only mode (not indexing content or keeping other state), e.g. `SB_SYNC_ONLY=1`. This will disable the Online [[Client Modes]] altogether (and not even show the sync icon in the top bar). Conceptually, [silverbullet.md](https://silverbullet.md) runs in this mode.