Quickstart for Site Reliability Engineers

Familiarize yourself with K8ssandra Operator utilities and procedures for repair, upgrade, and backup/restore operations for your Apache Cassandra® database.

Before performing these post-install steps, complete at least one K8ssandra Operator cluster deployment in Kubernetes.

In this quickstart for Site Reliability Engineers (SREs), we’ll cover:

Access the Cassandra nodetool utility

Cassandra’s nodetool utility is commonly used for a variety of monitoring and management tasks. You’ll need to run nodetool on your K8ssandra cluster using the kubectl exec command, because there’s no external standalone option available.

Follow these steps to run nodetool commands.

  1. Get a list of the running K8ssandra pods using kubectl get:

    kubectl get pods -n k8ssandra-operator
    

    Output:

    NAME                                                    READY   STATUS    RESTARTS   AGE
    demo-dc1-default-stargate-deployment-7b6c9d8dcd-k65jx   1/1     Running   0          5m33s
    demo-dc1-default-sts-0                                  2/2     Running   0          10m
    demo-dc1-default-sts-1                                  2/2     Running   0          10m
    demo-dc1-default-sts-2                                  2/2     Running   0          10m
    k8ssandra-operator-7f76579f94-7s2tw                     1/1     Running   0          11m
    k8ssandra-operator-cass-operator-794f65d9f4-j9lm5       1/1     Running   0          11m
    

    The K8ssandra pod running Cassandra takes the form <k8ssandra-cluster-name>-<datacenter-name>-default-sts-<n> and, in the example above is demo-dc1-default-sts-0 which we’ll use throughout the following sections.

    Note: Although not applicable to this quick start, additional K8ssandra Operator Cassandra nodes will increment the final <n> but the rest of the name will remain the same.

  2. Run nodetool status, using the Cassandra node name demo-dc1-default-sts-0.

    Replace <k8ssandra-username> and <k8ssandra-password> with the values you retrieved in the local install topic’s Extract credentials section:

    Tip: in that topic, the first single-cluster example returned this password: ACK7dO9qpsghIme-wvfI.

    With known -u and -p credentials, you can enter a command that invokes nodetool. Here’s an example. Your credentials will be different:

    kubectl exec --stdin --tty demo-dc1-default-sts-0 -n k8ssandra-operator -c cassandra -- nodetool -u demo-superuser 
    -pw ACK7dO9qpsghIme-wvfI status
    

    Output:

    Datacenter: dc1
    ===============
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address     Load       Tokens  Owns (effective)  Host ID                               Rack
    UN  10.244.1.5  96.71 KiB  16      100.0%            4b95036b-1603-464f-bdee-b519fa28a079  default
    UN  10.244.2.4  96.62 KiB  16      100.0%            ade61d9f-90f4-464c-8e18-dd3522c2bf3c  default
    UN  10.244.3.4  96.7 KiB   16      100.0%            0f75a6fe-c91d-4c0e-9253-2235b6c9a206  default
    

    Note: All nodes should have the status UN, which stands for “Up Normal”.

  3. Other useful nodetool commands include:

    nodetool ring which outputs all the tokens in the node. Example:

    kubectl exec --stdin --tty demo-dc1-default-sts-0 -n k8ssandra-operator -c cassandra -- nodetool -u demo-superuser 
    -pw ACK7dO9qpsghIme-wvfI ring
    

    Output:

    Datacenter: dc1
    ==========
    Address      Rack        Status State   Load            Owns                Token
                                                                               9126546575375666475
    172.17.0.13  default     Up     Normal  597.42 KiB      ?                   -9138166261715795932
    172.17.0.13  default     Up     Normal  597.42 KiB      ?                   -9120920057340937901
    172.17.0.13  default     Up     Normal  597.42 KiB      ?                   -9117737800555727340
    172.17.0.13  default     Up     Normal  597.42 KiB      ?                   -9058127181143818684
    172.17.0.13  default     Up     Normal  597.42 KiB      ?                   -8998548020695455271
    ...
    

    nodetool info which provides load and uptime information:

    kubectl exec --stdin --tty demo-dc1-default-sts-0 -n k8ssandra-operator -c cassandra -- nodetool -u demo-superuser 
    -pw ACK7dO9qpsghIme-wvfI info
    

    Output:

    ID                     : dec6a537-f00c-458a-bbc0-26b173675cc7
    Gossip active          : true
    Thrift active          : true
    Native Transport active: true
    Load                   : 597.42 KiB
    Generation No          : 1614265335
    Uptime (seconds)       : 9232
    Heap Memory (MB)       : 567.72 / 1024.00
    Off Heap Memory (MB)   : 0.00
    Data Center            : dc1
    Rack                   : default
    Exceptions             : 0
    Key Cache              : entries 39, size 3.46 KiB, capacity 51 MiB, 199 hits, 240 requests, 0.829 recent hit rate, 
    14400 save period in seconds
    Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 
    0 save period in seconds
    Counter Cache          : entries 0, size 0 bytes, capacity 25 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period 
    in seconds
    Chunk Cache            : entries 6, size 384 KiB, capacity 224 MiB, 111 misses, 3472 requests, 0.968 recent hit rate, 
    NaN microseconds miss latency
    Percent Repaired       : 100.0%
    Token                  : (invoke with -T/--tokens to see all 256 tokens)
    

For details on all nodetool commands, see The nodetool utility.

Configure port forwarding

In order to access Cassandra utilities outside of the K8s cluster, you’ll need to configure port forwarding.

If you haven’t already, get a list of your K8ssandra K8s services and ports:

kubectl get services

To configure port forwarding:

  1. Open a new terminal.

  2. Run the following 3 kubectl port-forward commands in the background. Example:

    kubectl port-forward svc/k8ssandra-reaper-reaper-service 9393:8080 &
    

    Output:

    ~/
    Forwarding from 127.0.0.1:9393 -> 8080
    Forwarding from [::1]:9393 -> 8080
    

The K8ssandra Operator services are now available at:

Terminate port forwarding

To terminate a particular forwarded port:

  1. Get the process ID:

    jobs -l
    

    Output:

    [3]  + 29213 running    kubectl port-forward svc/k8ssandra-reaper-k8ssandra-reaper-service 9393:8080
    
    1. Kill the process
    kill 80940
    

    Output:

    [3]  + terminated  kubectl port-forward svc/k8ssandra-reaper-k8ssandra-reaper-service 9393:8080
    

    Note: Exiting the terminal instance will terminate all port forwarding services.

Access Reaper

Reaper is an easy interface for managing K8ssandra cluster repairs. Reaper is deployed as part of the K8ssandra Operator install.

Reaper

For details, start in the Reaper topic. Then read about the repair tasks you can perform with Reaper.

Next steps

  • Components: Dig in to each deployed component of the K8ssandra stack and see how it communicates with the others.
  • Tasks: Need to get something done? Check out the Tasks topics for a helpful collection of outcome-based solutions.
  • Reference: Explore the Custom Resource Definitions (CRDs) used by K8ssandra Operator.

We encourage developers and SREs to actively participate in the K8ssandra community.



Last modified September 9, 2022: Update v2 monitoring docs (#1479) (9da0bd6)