Kubernetes ETCDCTL

ETCD is a key-value data store for storing kubernetes state and objects.

ETCDCTL is the tool used to back up with snapshot.

ETCDCTL a command line tool for interacting with the etcd server

 Key Features: 

1. Keep the event history until compaction.
1. access to old version of keys.
2. user controlled key compaction.
2. Support range query.
1. pagination support with limit argument.
2. support consistency guarantee across multiple queries.
3. Replace TTL key with lease.
4. Support watch feature.

ETCDTOOL Installation:

1. Build binary from source code 
2. Download tool directly 

Build binary from source code:

1. Checkout the code repository
• git clone -b v3.4.16 https://github.com/etcd-io/etcd.git
2. Change directory to etcd
• cd etcd
3. Run build script
• ./build
4. The binaries are under the bin directory.
• exportPATH="$PATH:`pwd`/bin"
5. Check etcd version
• etcd --version

Download ETCD tool directly:

1. kubectl exec -it etcd-docker-desktop -n kube-system -- /bin/bash -c 'ETCDCTL_API=3 /usr/local/bin/etcd --version' | head
2. export RELEASE='3.4.3'
3. wget https://github.com/etcd-io/etcd/releases/download/v{RELEASE}/etcd-v{RELEASE}-linux-amd64.tar.gz
4. cd etcd-v{RELEASE}-linux-amd64
5. sudo cp etcdctl /usr/local/bin

ETCD pod running on kube-system namespace:

$kubectl get namespace
kube-system         Active      11d

$kubectl get pods -n kube-system

etcd-docker-desktop                      1/1     Running         8 (15m ago)     11d

Check out etcd configuration information.

 $kubectl describe pod etcd-docker-desktop -n kube-system

Configuration for etcd comes from the static pod manifest

$sudo more /etc/kubernetes/manifests/etcd.yaml

--data-dir=/var/lib/etcd  # is the path where etcd data is stored inside a pod.

Take ETCD Backup using ETCDCTL tool:

Set which api to user to fetch data from etcd and provide certificates and endpoints so that cluster authenticate etcdctl tool.

$export ETCDCTL_API=3 etcdctl --endpoints=https:/127.0.0.1:2379/ --cacert = /etc/kubernetes/pki/etcd/ca.cert  --cert = /etc/kubernetes/pki/etcd/server.crt --key = /etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/data-backup.db

To Validate whether backup is successful or not

export ETCDCTL_API=3 etcdctl --write-out=table snapshot status /var/lib/data-backup.db

Restoring etcd 

1. Delete corrupted etcd distributed db under /var/lib/etcd
2. Stop etcd pod 
3. Move backup etcd db under /var/lib/etcd
4. Kubelet will restart etcd pod.

Restore command 

export ETCDCTL_API=3 etcdctl snapshot save <path to backup etcd db>

Practical :

Create secret :  

$kubectl create secret generic test-secret --from-literal=username='dayananda' --from-literal=password='password'

Get secrets:

$kubectl get secrets

test-secrets secret stored in etcd datastore.

Define a variable for the endpoint to etcd:

$ENDPOINT="https://127.0.0.1:2379

Verify whether we are connecting to a right cluster using member list:

$sudo ETCDCTL_API=3 etcdctl --endpoints=$ENDPOINT --cacert=/etc/kubernetes/pki/etcd/ca.cert  --cert=/etc/kubernetes/pki/etcd/server.crt --key = /etc/kubernetes/pki/etcd/srever.key member list

Backup etcd data:

$sudo ETCDCTL_API=3 etcdctl --endpoints=$ENDPOINT --cacert=/etc/kubernetes/pki/etcd/ca.cert --cert=/etc/kubernetes/pki/etcd/server.crt --key = /etc/kubernetes/pki/etcd/srever.key snapsot save /var/lib/etcd/data-backup.db

Check back up is valid or not:

$sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status /var/lib/etcd/data-backup.db

Delete the secret which we have created earlier and restore it back using etcdctl tool:

$kubectl delete secret test-secret

Run restore command which will restore test-secret 

$sudo ETCDCTL_API=3 etcd snapshot restore /var/lib/etcd/data-backup.db

default.etcd copy will be created with restore copy

stop etcd containier, move deafualt.etcd to /var/lib/etcd/

sudo mv ./default.etcd /var/lib/etcd/

wait for etcd to get restarted.

Other way to restore is :

Restore to a specific directory using --data-dir 

$sudo ETCDCTL_API=3 etcdctl snapshot restore /var/lib/data-backup.db --data-dir=/var/lib/etcd-restore

Update the pod manifest to point to /var/lib/etcd-restore

--data-dir=/var/lib/etcd-restore , mountPath under VolumeMounts and path under volumes

Some useful ETCD commands:

Command to set the value of key foo to bar:

$etcdctl put foo bar

OK

Command to set the value of key foo1 to bar1 for 10s.

$etcdctl put foo1 bar1 --lease=1234abcd

Command to get foo values

$etcd get foo

Command to print only key's value

$etcd get foo --print-value-only

Command to get all the keys which are prefixed with foo

$etcdctl get --prefix foo

Command to get keys revision of 4

$etcdctl get --prefix --rev=4

Command to delete key foo

$etcdctl del foo

Command to monitor foo

$etcdctl watch foo

Grant a lease with 60 second TTL

$etcdctl lease grant 60

lease 32695410dcc0ca06 granted with TTL(60s)

Attach key foo to lease 32695410dcc0ca06

$etcdctl put --lease=32695410dcc0ca06 foo bar

OK

Command to revoke the lease which in turn deletes all the associated keys. 

$etcdctl lease revoke 32695410dcc0ca06

lease 32695410dcc0ca06 revoked

$ etcdctl get foo

#empty response since foo is deleted due to lease revocation.

Happy Learning😀