Kubernetes cheat sheet for Qserv¶

Check first the official k8s cheat sheet for examples of kubectl basic commands.

Prerequisites¶

Get access to a Kubernetes cluster running Qserv, see Installation.

Interact with running pods¶

# Check a qserv instance is up and running
$ kubectl get qservs.qserv.lsst.org
NAME        AGE
qserv-dev   12h

# Get instance name
$ INSTANCE=$(kubectl get qservs.qserv.lsst.org -o=jsonpath='{.items[0].metadata.name}')
$ echo $INSTANCE
qserv-dev

# Describe a pod
$ kubectl describe pods "$INSTANCE"-worker-0

# Get the containers list for a given pod
$ kubectl get pods "$INSTANCE"-worker-0 -o jsonpath='{.spec.containers[*].name}'

# Open a shell on mariadb container on worker qserv-0
$ kubectl exec -it "$INSTANCE"-worker-0 -c mariadb bash

Access to log files¶

CZAR_POD="$INSTANCE-czar-0"

# Get the xrootd container logs on pod czar-0
kubectl logs "$CZAR_POD" -c xrootd

# Get the previous xrootd container logs on pod czar-0
kubectl logs "$CZAR_POD" -c xrootd -p

Stern provides advanced logging management features.

Update Qserv configuration¶

Update Qserv configuration by updating its related k8s configmaps.

# List configmaps
$ kubectl get configmaps -l app=qserv,instance="$INSTANCE"

# Edit configmap online, i.e. directly inside etcd, the k8s database
$ kubectl edit configmaps qserv-dev-repl-ctl-etc

# Restart the pod using the configmap, in order to take in account the new configuration
$ kubectl delete po "$INSTANCE"-repl-ctl-0

# A Qserv re-install will reset the configmap, so eventually, backup the configmap locally
$ kubectl get cm "$INSTANCE"-repl-ctl-etc -o yaml > "$INSTANCE".bck.yaml

Configure log level¶

The above technique can be used to configure logging level for xrootd and cmsd servers, replication worker and replication controller and czar proxy.

For example, for cmsd server:

kubectl edit cm qserv-cmsd-server-etc
# Then update log.cnf
# And restart Qserv worker pods
kubectl delete pods app=qserv,component=worker

Launch commands directly on Qserv nodes¶

Check if pod worker-0 can connect to replication database and dump it configuration¶

$ kubectl exec -it "$INSTANCE"-worker-0 -c repl-wrk -- mysql -h "$INSTANCE"-repl-db -u qsreplica -e "SELECT * FROM qservReplica.config;"
+------------+---------------------------------+-----------------------------+
| category   | param                           | value                       |
+------------+---------------------------------+-----------------------------+
| common     | request_buf_size_bytes          | 131072                      |
| common     | request_retry_interval_sec      | 5                           |
| controller | empty_chunks_dir                | /qserv/data/qserv           |
| controller | http_server_port                | 8080                        |
| controller | http_server_threads             | 16                          |
| controller | job_heartbeat_sec               | 0                           |
| controller | job_timeout_sec                 | 57600                       |
...

Navigation