Tutorial: Onboard a Node

Synopsis

This tutorial takes a Node from nothing to a TES service that is registered in the Cloud Registry, verified by its admins and reachable through the gateway of the ELIXIR On-Cloud. It is for the operators of compute centres, with a shorter path for data centres. Onboarding is the short checklist; this page goes through the same steps in detail, with the commands of each product.

The commands and values on this page come from the docs of each product, at the versions in the table below. For anything else, follow the linked product docs: they are the reference, and this page doesn't repeat them.

Product	Version	Role
TESK (Helm chart)	0.3.0, with tesk-api 1.2.0	TES service for Kubernetes
Poiesis	0.2.0	TES service for Kubernetes (alternative)
Funnel	0.12.3	TES service for HPC and HTC clusters (alternative)
Cloud Registry	1.0.0rc2	Registry of the services
proTES	0.6.0	Gateway in front of the TES services
Pubgrade-Sidecar	0.2.0	Automatic upgrades (optional)

TESK chart 0.3.0 is not released yet

The tutorial follows the docs of TESK chart 0.3.0, which is about to be released, and links to the latest version of the TESK docs, which describes it. Until the release, install the chart from a checkout of the TESK repository, as shown in step 2.

Under construction

The policies for joining the federation are not written down yet: who can join, what a Node has to provide, and what the admins check before they verify a service. Write to cloud-service@elixir-europe.org before you start. The steps below have not yet been tested by an operator who didn't write them: please report what doesn't work.

1. Prerequisites

You need:

• A cluster. For TESK and Poiesis, a Kubernetes or OpenShift cluster and Helm 3. TESK needs access to one namespace only (see Requirements); Poiesis needs Kubernetes 1.29 or later and a PostgreSQL database (see Prerequisites). For Funnel, an HPC or HTC cluster, e.g., with Slurm (see Funnel).
• A namespace of its own for the TES service, e.g., tesk.
• Storage. A default storage class (TESK creates a temporary volume claim for each task with inputs or outputs; ReadWriteOnce is enough), and a storage backend for the inputs and outputs of tasks, e.g., S3 or FTP (see Deploy storage).
• Ingress and HTTPS. An Ingress controller (or OpenShift Routes), a host name that points to it, and a TLS certificate, e.g., from Let's Encrypt with cert-manager. The Cloud Registry of this deployment accepts https URLs only, and the gateway uses only TES services at public addresses that answer without redirects.
• LS Login. An LS Login account, and an LS Login client that you use to get access tokens for your tests (see Register your service and Get an access token). The client must be allowed the scope eduperson_entitlement, which releases the groups of a user. For the tests, your account must be a member of the LS Login group ECP_CLN or one of its subgroups (see step 3): ask the admins.

2. Deploy a TES service

Choose the TES service that fits your cluster. What each of them supports today:

	TESK	Poiesis	Funnel
Runs on	Kubernetes, OpenShift	Kubernetes 1.29+	HPC, HTC (e.g., Slurm)
Validates LS Login tokens	yes, with gaps (see step 3)	yes (OIDC issuer and audience)	yes (OIDC issuer and audience)
Access by membership in ECP_CLN	by a claim of the user info	no, any valid token (generic-issues#21)	no, any valid token (generic-issues#20)
Service info for the registry	set in a file of the chart	fixed by the code, without environment	fixed by the code, with an environment that this registry refuses

Poiesis and Funnel can't enforce the ECP_CLN policy yet: they accept every valid token of the provider, so anyone with an LS Login account who gets such a token could run tasks on your cluster. Ask the admins before you join with one of them.

The rest of this step deploys TESK. For the alternatives, follow their own docs and continue with step 3:

• Poiesis: Deploying Poiesis and the Helm chart of version 0.2.0.
• Funnel: the Slurm guide of version 0.12.3, and the notes on Funnel.

Install TESK

The deployment guide of TESK covers the installation; this is the short path. Check out the TESK repository at the commit that this tutorial was checked against (chart 0.3.0 before its release), and work in the checkout, where the files below live:

git clone https://github.com/elixir-cloud-aai/TESK.git
cd TESK
git checkout 65db464017def4cddf8dc8571ec5161186d84517

Once chart 0.3.0 is released, install it from its release instead, after checking its attestation and signature, as described in From a release.

Write your settings into my-values.yaml, in the checkout. For an Ingress with a certificate of cert-manager (see Exposing the API; on OpenShift, use a Route instead, as described there):

hostName: tes.example.org
ingress:
  enabled: true
  className: nginx
  path: /
  # The secret that cert-manager creates; empty switches TLS off
  tlsSecretName: tesk-tls
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production

Set up the storage backend of your Node as described in Storage backends (e.g., storage: s3 and the files in charts/tesk/s3-config/).

Edit the service info, charts/tesk/service-info/service-info.yaml, before installing: at least id (unique, in reverse domain notation), name, organization, contactUrl and environment (see Service info). The registry reads these values in step 5, and this deployment accepts only the environments prod and test: the chart's default, dev, is refused.

Add the auth values of step 3 to my-values.yaml before you install. With the default, auth.mode: noauth, all endpoints are open, so anyone who reaches the Ingress could run tasks on your cluster. If you can't set them yet, keep ingress.enabled: false until you can.

Install the chart into a namespace of its own. secrets.yaml holds credentials, e.g., of FTP: leave out -f secrets.yaml if you need none.

helm upgrade --install tesk charts/tesk \
    --namespace tesk --create-namespace \
    -f my-values.yaml -f secrets.yaml

Check the installation:

helm list -n tesk
kubectl get all -n tesk
kubectl port-forward -n tesk service/tesk-api 8080:8080 &
sleep 2
curl http://localhost:8080/ga4gh/tes/v1/service-info
kill %1  # stop the port forward

The service info should show your values. Then check that it is reachable from outside, without a token:

curl https://tes.example.org/ga4gh/tes/v1/service-info

3. Enable LS Login and ECP_CLN access

The policy of the ELIXIR On-Cloud: users of a Node's TES service must be members of the LS Login group ECP_CLN or of one of its subgroups. LS Login releases group memberships as values of the claim eduperson_entitlement. The exact values are being recorded in generic-issues#46: ask the admins for them rather than guessing them.

Keep GET /service-info public. The registry can only import and sync service info that answers without a token (see Using the Cloud Registry).

TESK

With auth.mode: auth, TESK requires an access token for every endpoint except GET /service-info, which stays public by default (auth.serviceInfoPublic). Its group model doesn't fit LS Login as the chart configures it (see the gaps below), so authorise users by a claim of the user info instead (see Authorisation by a claim). Add to my-values.yaml, with the values that the admins give you:

auth:
  mode: auth
  claim:
    enabled: true
    name: eduperson_entitlement
    values:
      - <the ECP_CLN value(s) from the admins>

Users with one of the values may create tasks, which belong to them only, and list, get and cancel their own tasks. Everyone with one of the values can run any image with any command on your cluster: read the security implications in the docs of tesk-api. Set these values before the first installation (see step 2); to change them later, run its helm upgrade --install command again.

Gaps

• User info endpoint: tesk-api validates tokens at the user info endpoint of the provider, SECURITY_OAUTH2_RESOURCE_USER_INFO_URI, whose default is the endpoint of ELIXIR AAI, https://login.elixir-czech.org/oidc/userinfo (see Token validation). The endpoint of LS Login is https://login.aai.lifescience-ri.eu/oidc/userinfo. The TESK chart has no value for this setting. Whether tokens of LS Login are accepted with the default has not been verified for this tutorial.
• Group model: the chart sets the base group (auth.envBaseGroup, default ECP_CLN) and the subgroup (auth.envSubgroup), but not the prefix and suffix that tesk-api expects around group names, whose defaults are those of ELIXIR AAI. Hence the claim above.
• Status: enabling LS Login on the TESK deployments of the ELIXIR On-Cloud, and recording a configuration that works, is tracked in generic-issues#35 and generic-issues#50. Check them before you start.

Poiesis

Poiesis 0.2.0 validates OIDC tokens of one issuer, with values under auth.oidc.* of its chart: issuer and audience are required (see Authentication). The issuer of LS Login is https://login.aai.lifescience-ri.eu/oidc/; ask the admins which audience to require. Every user sees only their own tasks.

Gap

Poiesis accepts every valid token of the issuer and audience: it can't yet limit access to members of ECP_CLN. This is tracked in generic-issues#21.

Funnel

Funnel 0.12.3 validates OIDC tokens with the Server.OidcAuth section of its configuration: ServiceConfigURL (for LS Login, https://login.aai.lifescience-ri.eu/oidc/.well-known/openid-configuration) and the required RequireAudience, and TaskAccess: Owner limits users to their own tasks (see OAuth2). Ask the admins which audience to require.

Gap

Funnel accepts every valid token of the provider: it can't yet limit access to members of ECP_CLN. This is tracked in generic-issues#20.

4. Check it with a test task

Get an access token of your LS Login account, with the scope eduperson_entitlement (see Get an access token). Then create a task, from a checkout of the TESK repository (see API), and get its state, with your host name:

TES=https://tes.example.org/ga4gh/tes/v1
read -rs TOKEN   # paste the token; it is not shown or saved in the history
printf 'Authorization: Bearer %s\n' "$TOKEN" | curl -X POST "${TES}/tasks" \
    -H 'Content-Type: application/json' -H @- \
    -d @examples/success/hello.json
printf 'Authorization: Bearer %s\n' "$TOKEN" | curl "${TES}/tasks/<id>?view=BASIC" -H @-

The first request returns the ID of the new task. Repeat the second one with that ID until the state is COMPLETE. The task runs the image alpine, which the cluster must be able to pull. Poiesis and Funnel implement the same API; Deploying Poiesis has a similar example.

Also check that access is limited:

• Without a token, POST /tasks must fail (401 Unauthorized with TESK and Poiesis).
• With TESK, a token of an account without one of the values must fail with 403 Forbidden.

5. Register the service

Register your TES service in the Cloud Registry by importing its service info, and keep sync on, so that the entry follows your service. Using the Cloud Registry describes the policies of this deployment, and Using the Cloud Registry UI how to log in and register in the front end. In short:

1. Log in to the Cloud Registry with LS Login.
2. Import the service info of your service, e.g., https://tes.example.org/ga4gh/tes/v1/service-info, check the values, and register them.
3. Turn on sync.
4. If the registry answers that the API type or its version isn't allowed, ask the admins to add it: only they can add types in this deployment.

If your entry shows out-of-sync, see Out-of-sync reasons.

Gaps

• Poiesis serves fixed service info, without an environment. The registry accepts it, but the entry has no environment, and gateways that select TES services by environment skip it.
• Funnel serves fixed service info with the environment development, which this deployment doesn't allow. Its service info can't be imported or synced (invalid service info), so a Funnel entry can only be registered by hand. Ask the admins how to proceed, as entries without readable service info aren't verified here (see Verified services).

6. Get verified

Write to cloud-service@elixir-europe.org with the address of your entry, and ask for its verification. Only admins can verify services; you can't verify your own. What they check is up to this deployment, e.g., that the service accepts tokens of LS Login (see Verified services).

Verification is kept when sync updates the entry, but changing the URL or the API type removes it. See Verified services for what it means.

Under construction

The checks for the verification of a TES service are not written down yet.

7. Get added behind the gateway

proTES is the gateway in front of the Nodes' TES services: clients send their tasks to proTES, which forwards each task to one of the TES services it knows. It knows a TES service in one of two ways (see TES instances):

• Configured: its admins list your service in the configuration of proTES.
• Discovered: proTES reads the verified TES services of a Cloud Registry, so that a new Node is used once it is registered and verified, without a change of the configuration (see Discovery in a service registry).

For discovery, proTES uses a service only if all of these hold: it is verified, with its current URL and API type; it is a TES service of one of the versions that proTES accepts, and of one of the environments, if the gateway selects by environment; its URL is an https URL; and its host has public addresses only. Requests to discovered services don't follow redirects.

proTES never forwards the tokens of its users. It authenticates at TES services with credentials of its own: an access token of its own client (client_credentials, the default, also for discovered services), HTTP Basic credentials (basic, a transitional mode for TES services that don't accept tokens yet), or none (see Credentials of TES instances). Agree with the admins how the gateway authenticates at your service, and make your service accept it.

Tell the admins that your service is ready, then check that tasks sent through the gateway arrive at your service.

Under construction

The configuration of the ELIXIR On-Cloud's proTES (its address, its client, the TES services that it uses and how it authenticates at them) is tracked in generic-issues#31.

8. Optional: automatic upgrades with Pubgrade-Sidecar

Pubgrade-Sidecar rolls out new releases of a service to its Kubernetes Deployments when Pubgrade notifies it. It runs in the namespace of the service, and changes only the image tags of the Deployments that it may update. Install version 0.2.0 into the namespace of TESK, as described in Deployment:

1. Check out the release:

   git clone --depth 1 --branch v0.2.0 https://github.com/elixir-cloud-aai/Pubgrade-Sidecar.git
   cd Pubgrade-Sidecar
   

2. Create a Secret with a token of its own for your Node:

   kubectl -n tesk create secret generic pubgrade-sidecar \
     --from-literal=access-token="$(openssl rand -hex 32)"
   

3. Write the values, e.g., my-values.yaml. tesk-api is the Deployment of TESK for the release tesk:

   targetDeployments:
     - tesk-api
   accessToken:
     existingSecret: pubgrade-sidecar
   hostName: sidecar.example.org
   # On OpenShift:
   route:
     enabled: true
   # Elsewhere, instead:
   # ingress:
   #   enabled: true
   #   className: nginx
   #   tlsSecretName: sidecar-tls
   

4. Install the chart:

   helm install pubgrade-sidecar deployment/ -n tesk -f my-values.yaml
   

Gap

The central Pubgrade service of the ELIXIR On-Cloud isn't deployed yet (generic-issues#32), so nothing notifies the sidecar so far. Until then, upgrade TESK by hand, following the upgrade notes.

Data centres

A data centre that doesn't run tasks can still join:

• Storage for tasks: offer an S3 or FTP server that the TES services of the federation use for the inputs and outputs of tasks (see Deploy storage).
• Tools and workflows: TRS-Filer 0.5.0 serves them through the GA4GH TRS API. Deploy it with its Helm chart, and set its service info, at least id, name, organization, contactUrl and environment (prod or test), in a config overlay: the chart has no value for it yet (see Service info). Then register and verify it as in step 5 and step 6; ask the admins to add the TRS type if it is missing.

DRS-Filer, for data objects through the GA4GH DRS API, has no release yet, so this tutorial doesn't cover it.