feat: add documentation for diracx container deploment

docs/admin/how-to/install-without-kubernetes/index.md

@@ -0,0 +1,18 @@
+# Introduction
+
+These pages provide instructions for how to have a running `DiracX` instance without installing Kubernetes.
+This is an alternative to the main installation guide, which assumes you have a Kubernetes cluster available.
+
+!!! warning "This installation method is best-effort and not recommended"
+
+    This installation method is intended to assist with migrating existing `DIRAC` instances to `DiracX` without the need to set up a Kubernetes cluster.
+
+With respect to the original [minimal requirements](/docs/admin/how-to/install/minimal-requirements.md), the only addition is that you need a host on which to run the DiracX containers.
+
+- [Preparing a container node](prepare-container-node.md): how to prepare a node for a container-based deployment.
+- [Installing DiracX in a Container](installing-in-a-container.md): how to install `DiracX` in a container.
+
+!!! important "We assume you already have a DIRAC v9 installation available"
+
+    `DiracX` can only work alongside a `DIRAC` installation. We also assume that
+    you have access to a `DIRAC` client, which comes with the `dirac` cli.

docs/admin/how-to/install-without-kubernetes/installing-in-a-container.md

@@ -0,0 +1,196 @@
+# Installing DiracX in a container
+
+!!! warning "This is for container deployment only. For kubernetes deployment please refer to [Installing DiracX](/docs/admin/how-to/install/installing.md)."
+
+## Notes
+
+- We are running a 'developer' setup where we can edit the diracx code directly and run the local code
+    inside the container. This really helps with debugging ;-)
+- Similar to DIRAC this is installed as the (unprivileged) dirac user (and not root). There might be limitations to the approach that we have not found yet.
+- We give the version numbers of the code we tested, you should always check for the latest version and install this. If in doubt, please as on the DiracX [mattermost channel](https://mattermost.web.cern.ch/diracx/channels/town-square). In this example we used DIRAC v9.0.2, diracx 0.0.12 and diracx-web v0.1.0-a10.
+- If the same version number has to be specified in two different places, we indicate this explicitly.
+- We keep all of our configuration files in a folder called dirac-container. This name is chosen arbitrarily, and can be replaced with a name of your choice. However, the configuration examples assume that all files needed are kept in the same directory.
+
+## Configuration Files
+
+There are four configuration files that need to be provided to install DiracX in a container:
+`diracx.env`, `diracx-web.env`, `podman-compose.yaml` and `jwks.json`.
+In addition we also overwrite the entry point to include the database initialisation/schema management.
+You also need your OpenSearch server's `/etc/opensearch/root-ca.crt` (here called opensearch-ca.pem) to be able to connect to your OpenSearch server.
+Both `entrypoint.sh` and `opensearch-ca.pem` should be placed in the same folder as the configuration files.
+
+`diracx.env` is based on [reference/env-variables](../../reference/env-variables.md) <br>
+`jwks.json` is used to to generate the diracx authentication tokens. Please also see [how to rotate a secret](../../how-to/rotate-a-secret.md) <br>
+`podman-compose.yaml` is the container steering/manifest file. <br>
+
+### jwks.json
+
+To generate the jwks.json use a diracx container and start it with a shell (code adapted from `diracx/run_local.sh`): <br>
+`podman run --rm -ti ghcr.io/diracgrid/diracx/services:v0.0.12 /bin/bash`
+The --rm removes the container automatically after exiting.
+`python -m diracx.logic rotate-jwk --jwks-path "/tmp/jwks.json"`
+Then copy content of /tmp/jwks.json to file outside of the container to be mapped in a volume to the real thing.
+
+### diracx.env
+
+At this point you need to have created the `DiracXAuthDB` and the `jwks.json` (see above).
+Now you need two further keys:
+
+`DIRACX_SERVICE_AUTH_STATE_KEY`: Please see also [env-variables/diracx_service_auth_state_key](../../reference/env-variables.md#diracx_service_auth_state_key) and [dynamic-secrets-diracx-dynamic-secrets](../../explanations/chart-structure.md#dynamic-secrets-diracx-dynamic-secrets)
+To generate the key we follow diracx/run_local.sh: `state_key="$(head -c 32 /dev/urandom | base64)"`
+
+`DIRACX_LEGACY_EXCHANGE_HASHED_API_KEY`:
+This is adapted from ["generating a legacy exchange api key"] (connect.md) ("generating a legacy exchange api key") goes into dirac.cfg (as diracx:legacy...) and into diracx.env as a hash, but the gist of it is, the shared secret allows dirac and diracx to communicate.
+To generate the legacy key, you can use the following python snippet:
+
+```
+import secrets
+import base64
+import hashlib
+
+token = secrets.token_bytes()
+# This is the secret to include in the request by setting the
+# /DiracX/LegacyExchangeApiKey CS option in your legacy DIRAC installation (in the local -- secluded -- dirac.cfg file)
+print(f"API key is diracx:legacy:{base64.urlsafe_b64encode(token).decode()}")
+
+# This is the environment variable to set on the DiracX server
+print(f"DIRACX_LEGACY_EXCHANGE_HASHED_API_KEY={hashlib.sha256(token).hexdigest()}")
+```
+
+Below is an example of a dirac.env file. The server is called 'diractest.grid.hep.ph.ic.ac.uk'. Replace as necessary.
+
+```
+DIRACX_SERVICE_AUTH_STATE_KEY=[state_key from above]
+DIRACX_SERVICE_AUTH_TOKEN_ISSUER=["Your hostname here", e.g. "https://diractest.grid.hep.ph.ic.ac.uk"]
+DIRACX_SERVICE_AUTH_TOKEN_KEYSTORE=file:///keystore/jwks.json [The one you made earlier]
+# replace diractest.grid.hep.ph.ic.ac.uk with your hostname
+DIRACX_SERVICE_AUTH_ALLOWED_REDIRECTS='["https://diractest.grid.hep.ph.ic.ac.uk/#authentication-callback"]'
+# we are currently not using S3, nevertheless these variables need to be set
+DIRACX_SANDBOX_STORE_S3_CLIENT_KWARGS="{}"
+DIRACX_SANDBOX_STORE_BUCKET_NAME="notsetyet"
+# disable the jobs router (and hopefully S3)
+DIRACX_SERVICE_JOBS_ENABLED=false
+# we are using a local git repo on the node, this is not fully supported yet.
+# Please see: https://diracx.io/en/latest/RUN_PROD/#cs for the approved way.
+DIRACX_CONFIG_BACKEND_URL="git+file:///cs_store?revision=main"
+DIRACX_DB_URL_AUTHDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/DiracXAuthDB
+# these are the users and passwords you have chosen for the databases in DIRAC; they can be found in
+# dirac.cfg
+# host.containers.internal: For the development node our databases are on the same machine
+DIRACX_DB_URL_JOBDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/JobDB
+DIRACX_DB_URL_JOBLOGGINGDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/JobLoggingDB
+DIRACX_DB_URL_PILOTAGENTSDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/PilotAgentsDB
+DIRACX_DB_URL_SANDBOXMETADATADB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/SandboxMetadataDB
+DIRACX_DB_URL_TASKQUEUEDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/TaskQueueDB
+# opensearch related; the user and password are specific to opensearch,
+DIRACX_OS_DB_JOBPARAMETERSDB={"hosts":"YourOpensearchUsername:YourOpensearchPwd@YourOpensearchServer:9200", "use_ssl":true, "ca_certs":"/etc/opensearch-ca.pem"}
+DIRACX_OS_DB_PILOTLOGSDB={"hosts":"YourOpensearchUsername:YourOpensearchPwd@YourOpenSearchServer:9200", "use_ssl":true, "ca_certs":"/etc/opensearch-ca.pem"}
+DIRACX_LEGACY_EXCHANGE_HASHED_API_KEY=(from the output from generating the legacy key)
+```
+
+### diracx-web.env
+
+This file contains one line:
+`DIRACX_URL=https://diractest.grid.hep.ph.ic.ac.uk/api`
+
+### podman-compose.yaml
+
+Note: Please check that your ports specified in the yaml file match the ports you set up during the preparing a node for a container install procedure. The DiracX version specified must match the ones you use when you install the code. The `uvicorn` command is what actually starts diracx services. <br>
+Sources: <br>
+`ghcr.io`: from https://diracx.diracgrid.org/en/latest/RUN_PROD/#diracx-service-configuration <br>
+`command`: from diracx-charts/diracx/templates/diracx/deployment.yaml <br>
+`/opt/dirac/diracx-config:/cs_store:z,rw`: only needed because we use a local repo for the CS, a remote repo would be read directly over https. <br>
+
+```
+---
+services:
+  diracx-services:
+    ports:
+      - 127.0.0.1:8000:8000
+    image: ghcr.io/diracgrid/diracx/services:v0.0.12
+    env_file: "diracx.env"
+    command: "uvicorn --factory diracx.routers:create_app --host=0.0.0.0 --port=8000 --proxy-headers --forwarded-allow-ips=*"
+    volumes:
+      - ./entrypoint.sh:/entrypoint.sh:z,ro
+      - ./jwks.json:/keystore/jwks.json:z,ro
+      - /opt/dirac/diracx-config:/cs_store:z,rw
+      - /opt/dirac/diracx:/diracx_sources:z,ro
+      - ./opensearch-ca.pem:/etc/opensearch-ca.pem:z,ro
+  diracx-web:
+    image: ghcr.io/diracgrid/diracx-web/static:v0.1.0-a10
+    ports:
+      - 127.0.0.1:8001:8080
+    env_file: "diracx-web.env"
+```
+
+### entrypoint.sh
+
+```bash
+#!/bin/bash
+
+set -e
+echo "Welcome to the dark side"
+
+if [ -f /activate.sh ]; then
+    source /activate.sh
+else
+    eval "$(micromamba shell hook --shell=posix)" && micromamba activate base
+fi
+
+# this sets it up so that we can make changes to the code
+pip install -e /diracx_sources/diracx-core \
+            -e /diracx_sources/diracx-db \
+            -e /diracx_sources/diracx-logic \
+            -e /diracx_sources/diracx-routers \
+            -e /diracx_sources/diracx-client
+
+# initialise database and make updates if necessary
+python -m diracx.db init-sql
+
+exec "$@"
+```
+
+## Install the code
+
+```bash
+git clone https://github.com/DIRACGrid/diracx.git
+git checkout v0.0.12
+```
+
+## Tests
+
+Once the containers are installed, the following tests should be working (note: when testing, we needed to run the
+first command twice as there seemed to be a bug in DiracX. This might be fixed by the time you are trying this.)
+We ran these as root, as we were setting up our apache config as root and debugging this config as part of the process, but you can run this as any user.
+
+```
+[root@diractest ~]# curl -k https://diractest.grid.hep.ph.ic.ac.uk/.well-known/openid-configuration`
+{"issuer":"https://diractest.grid.hep.ph.ic.ac.uk","token_endpoint":"https://diractest.grid.hep.ph.ic.ac.uk/api/auth/token","userinfo_endpoint":"https://diractest.grid.hep.ph.ic.ac.uk/api/auth/userinfo","authorization_endpoint":
+[etc]
+```
+
+```
+[root@diractest ~]#  curl -k https://diractest.grid.hep.ph.ic.ac.uk/.well-known/jwks.json`
+{"keys":[{"crv":"Ed25519","x":"3_SrPXQyalji7nL4fNFh7JGBTwQBztmjnW7ogFusiPs","key_ops":["sign","verify"],"alg":"EdDSA","kid":"019a3a1b95b77b11a18bde7813b78fe2","kty":"OKP"}]}
+```
+
+To test the web container:
+
+```bash
+$ curl -k https://diractest.grid.hep.ph.ic.ac.uk/
+<!DOCTYPE html><html lang="en"><head>[etc] (i.e. returns a web site)
+```
+
+## A selection of helpful podman commands
+
+```
+podman ps -a # see all containers, dead or alive
+podman rm diracx-container_diracx-services_1 # remove dead container by name
+podman-compose -f podman-compose.yaml up -d --force-recreate # restart all containers (e.g. after update)
+podman exec -ti diracx-container_diracx-services_1 /bin/bash # to look into a running container
+podman logs -f diracx-container_diracx-services_1 # to look at the logs of a running container
+# clean up
+podman images # list all images
+podman image prune # should delete all unused images
+podman image rm [image id] # in case the automated delete doesn't work
+```

docs/admin/how-to/install-without-kubernetes/prepare-container-node.md

@@ -0,0 +1,35 @@
+# Preparing a node for container deployment
+
+!!! warning "This is for container deployment only. For kubernetes deployment please refer to [Installing kubernetes](/docs/admin/how-to/install/install-kubernetes.md)."
+
+## Podman
+
+These instructions are using the podman container tools as they are native to Rocky9. They are very similar to docker. Please check that these are installed on your system.
+
+## Configuring Apache
+
+Open port 443 for apache which serves up the connection to diracx services.
+Apache is configured in reverse proxy mode to forward external requests into the private container network.
+An example for an apache config (/etc/httpd/conf.d/diracx.conf) file is given below:
+
+```
+ProxyPreserveHost On
+ProxyPass /api http://127.0.0.1:8000/api
+ProxyPassReverse /api http://127.0.0.1:8000/api
+ProxyPass /.well-known http://127.0.0.1:8000/.well-known
+ProxyPassReverse /.well-known http://127.0.0.1:8000/.well-known
+ProxyPass / http://127.0.0.1:8001/
+ProxyPassReverse / http://127.0.0.1:8001/
+RequestHeader set "X-Forwarded-Proto" "https"
+```
+
+Note: We use 8000 for the diracx container and 8001 for the diracx-web container.
+The same ports need to the specified in the yaml files used to steer the containers.
+
+## Create the DiracXAuthDB
+
+At this point we also create the DiracXAuthDB. You can refer to the documentation [here](/docs/admin/how-to/install/installing.md#create-the-diracxauthdb)
+
+Note: The username and password will then re-appear in the `diracx.env` connection strings for the databases.
+
+You can now continue to [Convert CS](/docs/admin/how-to/install/convert-cs.md). After this please read [Installing DiracX](/docs/admin/how-to/install/installing.md) for some background information before proceeding to [Installing DiracX in a container](installing-in-a-container.md).

docs/admin/how-to/install/index.md

@@ -2,10 +2,9 @@
 
 These pages provide detailed instructions and best practices to have a running `DiracX` instance. If you start from nothing, they should be followed in order.
 
-- [Minimal Requirements](minimal-requirements.md): minimal requirements for running `DiracX`
-- [Install Kubernetes](install-kubernetes.md): how to install a minimal cluster.
+- [Minimal Requirements](/docs/admin/how-to/install/minimal-requirements.md): minimal requirements for running `DiracX`
+- [Installing in Kubernetes or in Containers](installing-in.md): how to install `DiracX`, that will do nothing to start with.
 - [Convert CS](convert-cs.md): how to generate the `DiracX` configuration from your `DIRAC` CS.
-- [Installing](installing.md): how to install `DiracX`, that will do nothing to start with.
 - [Register the Admin VO](register-the-admin-vo.md): Add the administrator VO, mandatory in `DiracX`
 - [Register a VO](register-a-vo.md): integrate a VO that you already have in `DIRAC`
 - [Embracing `DiracX`](embracing.md): last steps to redirect the traffic from `DIRAC` to `DiracX`

docs/admin/how-to/install/installing-in.md

@@ -0,0 +1,10 @@
+# Installing in Kubernetes or in Containers
+
+There are 2 ways to install DiracX, within Kubernetes or within Containers. You should pick one or the other. The recommended way is to use Kubernetes.
+
+For Kubernetes, follow these pages in order:
+
+- [Install Kubernetes](install-kubernetes.md): how to install a minimal cluster.
+- [Installing DiracX in Kubernetes](installing.md): how to install `DiracX`
+
+For using containers, instead follow [this guide](/docs/admin/how-to/install-without-kubernetes/index.md)

docs/admin/how-to/install/minimal-requirements.md

@@ -8,7 +8,7 @@ In order to run DiracX in production we recommend having:
 - **S3-compatible storage** for storing jobs' sandboxes
 - An **IdP** supporting OAuth/OIDC (e.g. [Indigo IAM](https://indigo-iam.github.io/))
 
-At the time of writing, the only supported way of running DiracX is through **[Kubernetes](https://kubernetes.io/docs/tutorials/kubernetes-basics/)**.
+At the time of writing, the only fully supported way of running DiracX is through **[Kubernetes](https://kubernetes.io/docs/tutorials/kubernetes-basics/)**. If you need to decouple your DIRAC v9 upgrade from a Kubernetes deployment, see [Installation without Kubernetes](../install-without-kubernetes/index.md).
 
 The following chapters will *NOT* cover:
 

mkdocs.yml

@@ -82,9 +82,15 @@ nav:
       - Installation:
         - admin/how-to/install/index.md
         - Minimal requirements: admin/how-to/install/minimal-requirements.md
-        - Installing kubernetes: admin/how-to/install/install-kubernetes.md
+        - Installation choices: admin/how-to/install/installing-in.md
+        - kubernetes:
+            - Installing kubernetes: admin/how-to/install/install-kubernetes.md
+            - Installing DiracX in kubernetes: admin/how-to/install/installing.md
+        - container:
+            - admin/how-to/install-without-kubernetes/index.md
+            - Preparing a container node: admin/how-to/install-without-kubernetes/prepare-container-node.md
+            - Installing DiracX in a Container: admin/how-to/install-without-kubernetes/installing-in-a-container.md
         - Convert CS: admin/how-to/install/convert-cs.md
-        - Installing DiracX: admin/how-to/install/installing.md
         - Connect DIRAC to DiracX: admin/how-to/install/connect.md
         - Register the admin VO: admin/how-to/install/register-the-admin-vo.md
         - Register a VO: admin/how-to/install/register-a-vo.md