kubernetes, persistent storage

Easy Way to Resize Kuberntes Persistent Volumes

3 min read

Last update:

The previous post about Rook got great attention from the community. But, I also got a lot of questions about persistence on Kubernetes in general. Among many questions, there were a few about resizing persistent volume claims or PVCs. If you are using on-premises storage solutions like Ceph (Rook) this is not the big issue. The part of the reason is that when you create a volume in Ceph, let's say a 100GB, this space is not actually allocated. But, if you are using AWS you will pay for that unused space even if you don't need it at the moment. Why is this so hard? The storage is elastic in the cloud, so resize is easy, but the problem is that resize should happen on the file system level also. Manual resize is possible, but it isn't convenient.

Previous blog post

Kubernetes Community to the Rescue

It seems that it would be great if we could grow persistent volume claims in size when needed. Actually, this is available from Kubernetes v1.8 and it is an alpha feature. In v1.9 resize is possible with those storage provisioners:

  • gcePersistentDisk
  • awsElasticBlockStore
  • OpenStack Cinder
  • glusterfs
  • rbd

Unfortunately, this is still not available for Rook. To be able to use this feature you need to allow expanding PVCs by setting ExpandPersistentVolumes feature gate to true. Also, enable PersistentVolumeClaimResize admission plugin if you want to prevent resizing of all claims unless you enable resize in storage class. How to update feature gates and admission plugins depends on how you deployed Kubernetes cluster, but in general, add the following parameters to kube-apiserver process --feature-gates=ExpandPersistentVolumes=true,... and --admission-control=PersistentVolumeClaimResize,..., and also add the same feature gate to kube-controller-manager.

For the purpose of this blog post, I deployed a Kubernetes cluster using Heptio AWS quickstart guide which deploys Kubernetes with kubeadm. Instructions in this blog post should work for any cluster deployed with kubeadm v1.9. After the stack is up and running you need to update manifests files kube-apiserver.yaml and kube-controller-manager.yaml:

sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --admission-control=PersistentVolumeClaimResize,Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota
    - --feature-gates=ExpandPersistentVolumes=true

sudo vim /etc/kubernetes/manifests/kube-controller-manager.yaml

spec:
  containers:
  - command:
    - kube-controller-manager
    - --feature-gates=ExpandPersistentVolumes=true

Then you need to restart kubelet to apply those changes:

sudo systemctl restart kubelet

After enabling changed configurations you need to set allowVolumeExpansion for storage class if you want to enable expansion. I will create the new gp2-resize storage class:

⚡ cat <<EOF | kubectl create -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: gp2-resize
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
allowVolumeExpansion: true
EOF

And let's create the simple PVC:

⚡ cat <<EOF | kubectl create -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: gp2-resize
EOF

⚡ kubectl get pvc myclaim
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myclaim   Bound     pvc-d15808da-2575-11e8-a133-02cfcfdc585a   8Gi        RWO            gp2-resize     36s

In this case, PVC is not used by any Pod. I can go ahead and make it larger, from 8GB to 12GB:

⚡ kubectl edit pvc myclaim

spec:
  resources:
    requests:
      storage: 12Gi

This will trigger the expansion of this PVC. Supported file systems are ext3, ext4, and XFS. Please note that file system resize will not happen unless you recreate your Pods. This is valid if the PVC is in use.

NOTE: Online resize support should be available in Kubernetes v1.11, and you won't need to recreate resources for file system resize.

Wait a few minutes and check for the updated EBS size:

⚡ aws --region $REGION ec2 describe-volumes --filters Name=tag-key,Values="kubernetes.io/created-for/pvc/name" Name=tag-value,Values="myclaim" --query 'Volumes[0].Size'
12

However, PVC size in Kubernetes was not updated to reflect the new size:

⚡ kubectl get pvc myclaim -o=jsonpath="{.status.capacity.storage}"
8Gi

Maybe I'm missing something or this is a bug. After resize you will have to wait at least 6 hours before another modification per EBS volume.

Summary

With features like online volumes resize, stateful apps will be easier to use on Kubernetes. While not that straightforward to use at the moment, I'm sure that resize will become better over time. After all, this is an alpha feature. Stay tuned for the next one.

You may want to read