Finish up the deployment docs

code/docs_page.js

@@ -17,7 +17,6 @@ const DocsPage = ({ _body, _ID, docsFor, titlePrefix, _pages }) => (
 					const toc = tocs[docsFor];
 					const ia = ka.replace(docsFor + '/', '');
 					const ib = kb.replace(docsFor + '/', '');
-					console.log("@@", ia, ib);
 					if (toc.includes(ia) && toc.includes(ib)) {
 						return toc.indexOf(ia) - toc.indexOf(ib);
 					}

content/matrix-media-repo/toc.json

@@ -4,5 +4,10 @@
   "installation",
   "installation/config",
   "installation/server-names",
-  "installation/methods"
+  "installation/methods",
+  "deployment",
+  "deployment/horizontal-scaling",
+  "deployment/high-availability",
+  "upgrading",
+  "upgrading/to-130"
 ]

content/matrix-media-repo/unstable/deployment/body.md

@@ -0,0 +1,50 @@
+---
+layout: docs_page
+docsFor: matrix-media-repo/unstable
+titlePrefix: matrix-media-repo
+---
+
+# Deployment
+
+MMR sits between your reverse proxy and underlying homeserver software, such as Synapse or Dendrite.
+This allows clients to continue making requests to the server without needing to know of an alternative
+media handling setup.
+
+The easiest way to deploy MMR is to route all `/_matrix/media` endpoints to the MMR process. Note that
+with [MSC3911](https://github.com/matrix-org/matrix-spec-proposals/pull/3911) and [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916),
+the specific set of endpoints may change.
+
+An example NGINX config would be:
+
+```
+server {
+  listen 443 ssl;
+  listen [::]:443 ssl;
+
+  # SSL options not shown - ensure the certificates are valid for your
+  # homeserver deployment.
+
+  # Redirect all traffic by default to the homeserver
+  location /_matrix {
+      proxy_read_timeout 60s;
+      proxy_set_header Host $host;
+      proxy_set_header X-Real-IP $remote_addr;
+      proxy_set_header X-Forwarded-For $remote_addr;
+      proxy_pass http://localhost:8008; # Point this towards your homeserver
+  }
+
+  # Redirect all media endpoints to MMR
+  location /_matrix/media {
+      proxy_read_timeout 60s;
+
+      # Make sure this matches your homeserver in your MMR config
+      # You may have to manually specify it if using delegation or the
+      # incoming `Host` doesn't match.
+      proxy_set_header Host $host;
+
+      proxy_set_header X-Real-IP $remote_addr;
+      proxy_set_header X-Forwarded-For $remote_addr;
+      proxy_pass http://localhost:8000; # Point this towards media-repo
+  }
+}
+```

content/matrix-media-repo/unstable/deployment/high-availability/body.md

@@ -0,0 +1,24 @@
+---
+layout: docs_page
+docsFor: matrix-media-repo/unstable
+titlePrefix: matrix-media-repo
+---
+
+# High Availability
+
+MMR supports horizontal scaling and by consequence high availability capabilities as well. Near-zero
+downtime can be achieved by employing rolling upgrades of a horizontally-scaled cluster.
+
+In all cases, machine ID `0` *must* be updated first, however all other processes can be updated in
+any order.
+
+For a given version number `vX.Y.Z`:
+
+* A change in `X` indicates a *breaking change* to MMR. The entire cluster should be brought offline
+  to perform the upgrade. Check the upgrade notes for further guidance on zero-downtime upgrades to
+  this version.
+* A change in `Y` indicates a *large but backwards compatible change* to MMR. The cluster should be
+  upgraded quickly, but does not need to be taken offline first. A higher volume of errors may happen
+  while the cluster is upgraded.
+* A change in `Z` indicates a *patch or otherwise small backwards compatible change* to MMR. The
+  cluster can be upgraded more slowly, and does not need to be taken offline first.

content/matrix-media-repo/unstable/deployment/high-availability/index.yml

@@ -0,0 +1,7 @@
+title: matrix-media-repo -  High Availability
+header:
+  - /_shared/header.md
+main:
+  - body.md
+footer:
+  - /_shared/footer.md

content/matrix-media-repo/unstable/deployment/horizontal-scaling/body.md

@@ -0,0 +1,31 @@
+---
+layout: docs_page
+docsFor: matrix-media-repo/unstable
+titlePrefix: matrix-media-repo
+---
+
+# Horizontal Scaling
+
+MMR's horizontal scaling is achieved through the `MACHINE_ID` environment variable. The default machine
+ID is `0`, and is responsible for all background tasks. Each deployed process *must* have a distinct
+machine ID in order to function. The machine ID *must* be between `0` and `1023`, allowing for a
+total of 1024 "workers".
+
+All MMR processes *must* additionally share the same config files, database, and Redis backend. The
+Redis server is used for inter-process communication. Once set up, any MMR process in the cluster can
+handle any request for load balancing.
+
+Typically, it is expected that there be roughly aligned "sub-clusters": a set of processes dedicated
+to handling each of downloads, uploads, URL previews, etc. At a minimum, it is best to at least split
+URL previews off to a dedicated process to avoid network request congestion. Admin APIs are best routed
+to machine ID `0`.
+
+Thumbnails are recommended to be moved to high-memory machines where possible in high-traffic environments.
+
+The number of machines/processes needed largely depends on the number of homeservers being hosted by
+a single cluster. For approximately every 500 homeservers there should be 1 additional MMR process,
+rounded up (assuming each process doesn't have a dedicated purpose).
+
+Horizontal scaling is *possible* with file-backed datastores, however all processes need to have access
+to the directory at the exact same path. It is instead suggested to use exclusively S3 datastores, where
+all processes can more easily fetch contents.

content/matrix-media-repo/unstable/deployment/horizontal-scaling/index.yml

@@ -0,0 +1,7 @@
+title: matrix-media-repo -  Horizontal Scaling
+header:
+  - /_shared/header.md
+main:
+  - body.md
+footer:
+  - /_shared/footer.md

content/matrix-media-repo/unstable/deployment/index.yml

@@ -0,0 +1,7 @@
+title: matrix-media-repo -  Deployment
+header:
+  - /_shared/header.md
+main:
+  - body.md
+footer:
+  - /_shared/footer.md

content/matrix-media-repo/unstable/upgrading/body.md

@@ -0,0 +1,15 @@
+---
+layout: docs_page
+docsFor: matrix-media-repo/unstable
+titlePrefix: matrix-media-repo
+---
+
+# Upgrading
+
+Upgrades are relatively easy: stop the existing process, replace the binaries/Docker image, and start
+it up again.
+
+Some versions require mandatory configuration changes or have other restrictions preventing such an
+easy upgrade. Check the notes in this section when upgrading to a newer version.
+
+A rolling changelog is available [in the GitHub repo](https://github.com/turt2live/matrix-media-repo/blob/master/CHANGELOG.md).

content/matrix-media-repo/unstable/upgrading/index.yml

@@ -0,0 +1,7 @@
+title: matrix-media-repo -  Upgrading
+header:
+  - /_shared/header.md
+main:
+  - body.md
+footer:
+  - /_shared/footer.md

content/matrix-media-repo/unstable/upgrading/to-130/body.md

@@ -0,0 +1,80 @@
+---
+layout: docs_page
+docsFor: matrix-media-repo/unstable
+titlePrefix: matrix-media-repo
+---
+
+# Upgrading to v1.3.0
+
+## Mandatory Configuration Change
+
+Datastores are no longer managed by matrix-media-repo internally, meaning you MUST specify a datastore ID on each of your
+configured datastores. If you're setting up matrix-media-repo for the first time then you can use whatever you want for
+a datastore ID (though it's recommended to stick to alphanumeric strings). If you're *upgrading* to this version however,
+you will need to pull the datastore IDs out of the matrix-media-repo and add them to your configuration.
+
+**For safety, the datastores table is *not* deleted from the database in this upgrade. A future version may drop the table,
+however.**
+
+### Getting existing datastore IDs
+
+**Before upgrading**, you can get your datastore IDs fairly easily. The best way might be to look at the startup log of
+your media repo:
+
+```text
+INFO[2023-05-21 20:58:45.116 Z] Datastores:
+INFO[2023-05-21 20:58:45.116 Z]         file (e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336): /mnt/mmr-store/location4
+INFO[2023-05-21 20:58:45.117 Z]         s3 (7669e2fb8ccaa0801e4255a417ad20884f76b8611659655069202644992): s3://redacted.r2.cloudflarestorage.com/redacted
+```
+
+This way, you're able to correlate locations to IDs. For example, the `file` datastore configured to put media at
+`/mnt/mmr-store/location4` has ID `e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336`. Add this as
+`id: "e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336"` in your media repo config file.
+
+Alternatively, you can use the admin API to get your datastores:
+
+```text
+curl -s -X GET -H "Authorization: Bearer YOUR_ACCESS_TOKEN" https://example.org/_matrix/media/unstable/admin/datastores
+{
+  "e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336": {
+    "type": "file",
+    "uri": "/mnt/mmr-store/location4"
+  },
+  "7669e2fb8ccaa0801e4255a417ad20884f76b8611659655069202644992": {
+    "type": "s3",
+    "uri": "s3://redacted.r2.cloudflarestorage.com/redacted"
+  }
+}
+```
+
+The returned object is keyed by ID over the API.
+
+In either case, take the ID and add it to the associated datastore in your config, similar to the following:
+
+```yaml
+# Your specific configuration may be different
+datastores:
+  - type: file
+    id: "e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336"  ## ADD THIS
+    #enabled: true   ## REMOVE THIS - use `forKinds: []` to disable a datastore
+    forKinds: ["archives"]
+    opts:
+      path: "/mnt/mmr-store/location4"
+  - type: s3
+    id: "7669e2fb8ccaa0801e4255a417ad20884f76b8611659655069202644992"  ## ADD THIS
+    #enabled: true   ## REMOVE THIS - use `forKinds: []` to disable a datastore
+    forKinds: ["all"]
+    opts:
+      ssl: true
+      tempPath: "/mnt/mmr-store/s3-staging"
+      endpoint: sfo2.digitaloceanspaces.com
+      accessKeyId: "redacted"
+      accessSecret: "redacted"
+      bucketName: "redacted"
+```
+
+**Note**: If matrix-media-repo detects that a datastore ID is used but not referenced in the config then it will refuse
+to start.
+
+This new configuration style additionally allows for out-of-band datastore transfers. If you move all your data to a new
+path/server, for example, then you can simply update the path in the config for that datastore.

content/matrix-media-repo/unstable/upgrading/to-130/index.yml

@@ -0,0 +1,7 @@
+title: matrix-media-repo -  Upgrading to v1.3.0
+header:
+  - /_shared/header.md
+main:
+  - body.md
+footer:
+  - /_shared/footer.md