Storage Configuration

This guide covers persistent storage configuration in Krutrim Kubernetes Service using Container Storage Interface (CSI) drivers.

Understanding Storage in Kubernetes

Kubernetes provides several storage abstractions:

• Volumes: Temporary storage tied to pod lifecycle

• PersistentVolumes (PV): Cluster-level storage resources

• PersistentVolumeClaims (PVC): User requests for storage

• StorageClass: Dynamic provisioning configuration

CSI Add-on Overview

The Container Storage Interface (CSI) driver enables dynamic storage provisioning in your cluster.

What is CSI?

CSI is a standard interface for exposing block and file storage systems to containerized workloads on Kubernetes.

Benefits:

• ✅ Dynamic volume provisioning

• ✅ Automatic volume creation

• ✅ Volume snapshots

• ✅ Volume expansion

• ✅ Integration with cloud storage

Critical: CSI Add-on Requirement

Default storage class is ONLY available if CSI Node Plugin add-on is enabled.

Without CSI:

• ❌ No default storage class

• ❌ Cannot create PersistentVolumeClaims

• ❌ No dynamic volume provisioning

With CSI:

• ✅ Default storage class available

• ✅ Can create PersistentVolumeClaims

• ✅ Dynamic volume provisioning works

Installing CSI Add-on

Install the CSI add-on to enable persistent storage in your cluster.

Configuring Persistent Storage

Prerequisites

Before configuring storage, ensure:

• ✅ CSI Node Plugin Installed:

  • CSI Node Plugin add-on: ACTIVE

Verify Installation

# Check CSI pods are running
kubectl get pods -n kube-system | grep csi

# Expected output:
csi-cinder-nodeplugin-xxxxx     3/3   Running   0   3m
csi-cinder-nodeplugin-yyyyy     3/3   Running   0   3m

# Verify storage class is available
kubectl get storageclass

# Expected output:
NAME                    PROVISIONER                 RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION
csi-cinder-sc-qos-delete (default)   cinder.csi.openstack.org    Delete          Immediate   true

Understanding Storage Classes

Available Storage Classes

Once the CSI add-on is installed, a default storage class is automatically configured for your cluster.

Default Storage Class

After CSI installation, the default storage class is automatically created:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: csi-cinder-sc-qos-delete
parameters:
  availability: nova
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: Immediate

Storage Class Parameters

Provisioner: cinder.csi.openstack.org

• Uses OpenStack Cinder for block storage

• Provides persistent volumes backed by cloud storage

Reclaim Policy: Delete

• Volumes are deleted when PVC is deleted

• Data is permanently removed

• Use caution with production data

Volume Binding Mode: Immediate

• Volume is created immediately when PVC is created

• Does not wait for pod to be scheduled

• Volume may be created in different zone than pod

Allow Volume Expansion: true

• Volumes can be resized after creation

• Requires PVC and pod restart

Using Persistent Storage

Creating a PersistentVolumeClaim

Basic PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  # storageClassName: krutrim-csi  # Optional, uses default if omitted

Apply PVC:

kubectl apply -f pvc.yaml

Check PVC Status:

kubectl get pvc my-app-data

# Expected states:

# Pending → Binding in progress

# Bound → Volume created and bound

Access Modes

ReadWriteOnce (RWO):

• Volume can be mounted read-write by a single node

• Most common mode

• Supported by default storage class

ReadOnlyMany (ROX):

• Volume can be mounted read-only by multiple nodes

• Less common

• Check if supported

ReadWriteMany (RWX):

• Volume can be mounted read-write by multiple nodes

• Requires special storage types (not block storage)

• Not supported by default Cinder CSI

Using PVC in Pods

Pod with Volume Mount

apiVersion: v1
kind: Pod
metadata:
  name: app-with-storage
spec:
  containers:
  - name: app
    image: nginx:latest
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-app-data

StatefulSet with Volume Claim Template

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: database
spec:
  serviceName: database
  replicas: 3
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
      - name: postgres
        image: postgres:14
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 50Gi

Benefits of VolumeClaimTemplates:

• Each pod gets its own PVC

• PVC name includes pod name (data-database-0, data-database-1, etc.)

• Automatic creation and management

Common Storage Scenarios

Scenario 1: Web Application with User Uploads

# PVC for uploads
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: uploads-storage
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
---
# Deployment using the PVC
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: production
spec:
  replicas: 1  # Note: RWO limits to single replica
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: app
        image: my-web-app:latest
        volumeMounts:
        - name: uploads
          mountPath: /app/uploads
      volumes:
      - name: uploads
        persistentVolumeClaim:
          claimName: uploads-storage

Scenario 2: Database with Persistent Storage

# PVC for database
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: database
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
---
# StatefulSet for database
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: database
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:14
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        ports:
        - containerPort: 5432
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: postgres-data

Scenario 3: Shared Configuration Files

# ConfigMap for config files (not PVC)
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  app.conf: |
    server {
      listen 80;
      server_name localhost;
    }
---
# Pod using ConfigMap
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: nginx:latest
    volumeMounts:
    - name: config
      mountPath: /etc/nginx/conf.d
  volumes:
  - name: config
    configMap:
      name: app-config

Note: Use ConfigMap for configuration, PVC for data

Scenario 4: Multiple Containers Sharing Data

# Single PVC shared by containers in same pod
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: multi-container-app
spec:
  containers:
  - name: writer
    image: writer-app:latest
    volumeMounts:
    - name: shared
      mountPath: /data
  - name: reader
    image: reader-app:latest
    volumeMounts:
    - name: shared
      mountPath: /data
      readOnly: true
  volumes:
  - name: shared
    persistentVolumeClaim:
      claimName: shared-data

Managing Storage

Viewing Storage Resources

# List storage classes
kubectl get storageclass

# List PersistentVolumes
kubectl get pv

# List PersistentVolumeClaims
kubectl get pvc -A

# Describe PVC for details
kubectl describe pvc my-app-data

# Check PVC events
kubectl get events --field-selector involvedObject.name=my-app-data

Expanding Volumes

The default storage class supports volume expansion.

Step: Edit PVC

kubectl edit pvc my-app-data

Step: Increase Size (example)

spec:
  resources:
    requests:
      storage: 20Gi  # Increased from 10Gi

Step: Restart Pod

# Delete pod to trigger remount (for Deployment)
kubectl delete pod <pod-name>

# For StatefulSet
kubectl rollout restart statefulset <statefulset-name>

Verification:

# Check PVC size
kubectl get pvc my-app-data

# Check from inside pod
kubectl exec <pod-name> -- df -h /data

Important notes:

• Can only increase size, not decrease

• Pod must be restarted for filesystem resize

• Some storage backends may take time to expand

Deleting Storage

Delete PVC:

kubectl delete pvc my-app-data

What happens:

• PVC is deleted

• PV is deleted (due to Delete reclaim policy)

• Underlying storage is removed

• Data is permanently lost

Warning: Deleting PVC deletes data permanently!

Protecting Important Data:

• Option 1: Change Reclaim Policy

# Change PV reclaim policy to Retain
kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

• Option 2: Backup Before Deletion

# Create snapshot or backup data before deleting

• Option 3: Don't Delete PVC

  • Keep PVC even if not in use

  • PVC doesn't cost money, storage does

  • Reattach to new pods when needed

Storage Best Practices

1

Size Appropriately

• Start with reasonable size

• Plan for growth (can expand later)

• Monitor usage regularly

2

Use Appropriate Access Modes

• ReadWriteOnce for most applications

• ReadOnlyMany for shared read-only data

• Understand access mode limitations

3

Plan for Data Persistence

• Important data: Use PVCs

• Temporary data: Use emptyDir

• Configuration: Use ConfigMaps/Secrets

4

Monitor Storage Usage

# Check PVC usage
kubectl exec <pod-name> -- df -h

# Monitor regularly
# Set up alerts for high usage

5

Backup Critical Data

• Regular backups of databases

• Export important data

• Test restore procedures

6

Use StatefulSets for Stateful Apps

• Databases, queues, caches

• Automatic PVC management

• Stable network identities

Anti-Patterns / Don'ts

1

Don't Use ReadWriteOnce for Multi-Replica Apps

# ❌ Bad: Multiple replicas with RWO
replicas: 3
volumes:
- persistentVolumeClaim:
    claimName: shared-data  # RWO - only one can mount

# ✅ Good: Use StatefulSet with volumeClaimTemplates
# Or use single replica with RWO

2

Don't Delete PVC Without Backup

• Always backup important data first

• Verify backups are restorable

• Document data recovery procedures

3

Don't Overprovision Storage

• Start reasonable, expand as needed

• Storage costs money

• Monitor actual usage

4

Don't Ignore Disk Full Errors

• Monitor disk usage

• Set up alerts at 80% usage

• Expand or clean up before full

5

Don't Store Secrets in Volumes

• Use Kubernetes Secrets

• Use proper secret management

• Don't write passwords to persistent storage

Troubleshooting Storage

PVC Stuck in Pending

Symptoms:

• PVC status remains Pending

• No PV created

Possible Causes:

1. CSI add-on not installed

2. Storage provisioning in progress

3. Storage quota exceeded

Solution:

# Check CSI pods
kubectl get pods -n kube-system | grep csi

# Check PVC events
kubectl describe pvc <pvc-name>

# Check if storage provisioning completed
kubectl get pvc <pvc-name>

# Verify storage class exists
kubectl get storageclass

Pod Cannot Mount Volume

Symptoms:

• Pod in ContainerCreating state

• Events show volume mount errors

Possible Causes:

1. PVC not bound

2. Node doesn't have CSI driver

3. Volume in use by another pod (RWO)

Solution:

# Check PVC status
kubectl get pvc

# Check pod events
kubectl describe pod <pod-name>

# Check CSI node plugin on node
kubectl get pods -n kube-system -l app=csi-cinder-nodeplugin -o wide

# If RWO, ensure only one pod uses volume
kubectl get pods -o wide | grep <pvc-name>

Volume Out of Space

Symptoms:

• Application errors writing to disk

• Pod logs show "no space left on device"

Solution:

# Check current usage
kubectl exec <pod-name> -- df -h /data

# Expand volume (see Expanding Volumes section)
kubectl edit pvc <pvc-name>

# Increase storage size

# Restart pod

# Or clean up data
kubectl exec <pod-name> -- rm -rf /data/old-files

Storage Class Not Found

Symptoms:

• PVC pending with "no storage class found"

• kubectl get storageclass shows nothing

Solution:

# Install CSI Node Plugin add-on

# Verify installation
kubectl get storageclass

# Check CSI pods
kubectl get pods -n kube-system | grep csi

Storage Sizing Guidelines

Application Types

Small Applications / Development:

storage: 10-20Gi

Medium Applications / Production:

storage: 50-100Gi

Databases:

# Small database
storage: 100Gi

# Medium database
storage: 200-500Gi

# Large database
storage: 1Ti+

File Storage / Media:

# Image hosting
storage: 200Gi-1Ti

# Video storage
storage: 1Ti+

Monitoring / Logging:

# Prometheus (30 days retention)
storage: 100-200Gi

# Elasticsearch / Logging
storage: 500Gi-2Ti

Additional Resources

• Installing Add-ons - Install CSI add-ons

• Best Practices - Storage optimization tips

• Troubleshooting Guide - Common storage issues