Persistent Storage for Kubernetes
Deploy Portworx on Red Hat OpenShift Container Platform
Overview
In today’s digital era, the value of containerized application is unquestionable. It’s a paradigm shift in application development and it’s a move in adopting DevOps and cloud native architecture for successful digital transformation. Due to this, Kubernetes (K8) has become a mainstream and de-facto standard for container orchestration. Kubernetes has lot of moving parts, however, in this article, I’ll be covering container storage solution.
Workloads deployed in containers and orchestrated via K8 are either stateless or stateful. By default, K8 workloads are stateless. Stateless application don’t persist, which means it uses temporary storage provided within K8 and destroys once the application or pod is terminated. That’s why we call containers are ephemeral in nature, data associated with containers can be lost once the container is terminated or accidentally crashed. Furthermore, data can’t be shared among other containers either.
For stateful application, persistent storage is the first “must have” requirement. Kubernetes supports various persistent storage solutions that help addressing this problem and support stateful workloads in a containerized environment. Kubernetes introduces the concept of Persistent Volumes, which exist independently of containers, survive even after containers shut down, and can be requested and consumed by containerized workloads.
In this particular write up, I’ll be covering Portworx, which is a scalable persistent storage platform for Kubernetes. Portworx is available in almost all major Kubernetes platforms such as Amazon EKS, Azure AKS, Google GKE, to name a few. In this document, I’ll be talking about how to implement Portworx in Red Hat OpenShift Container Platform.
Logical Topology
Below figure shows how worker nodes in OpenShift platform is configured with respect to disk allocated to be used in forming a Portworx cluster. It is also important to note that, in this setup, I’ll be using internal built-in key value database (KVDB). This requires minimum of three nodes to form key-value store (kvdb) cluster and nodes should be labelled as px-metadata-node=true. Optionally you can use label px/enabled=false if you do not want any specific node to be a part of Portworx cluster.
If you plan to use your own etcd cluster, please refer to Portworx documentation. This document does not cover Portworx deployment using external key-value store.
Prerequisites
Following are the prerequisite to setup Portworx in OpenShift
· Your cluster must be running OpenShift 4 or higher.
· Minimum of three nodes are required
· Each node must meet the following minimum requirements for Portworx
Install Portworx Operator in OpenShift
Portworx operator is an easy way to deploy Portworx in OpenShift. It manages the overall lifecycle of Portworx cluster. With that, we can install, configure, update, and remove Portworx.
Below diagram shows the components that makes up Portworx platform.
Login to OpenShift web console and navigate to OperatorHub. Search for Portworx as shown below.
In this setup, I will be installing Portworx Essentials. It’s a free version with limited capability such as up to 5 nodes, 5 TB of capacity, and 500 volumes. It is good enough for PoC purposes. However, for production grade setup, Portworx enterprise can be used. You can also give enterprise version a try with 30 days evaluation period.
Now install Portworx essentials in kube-system namespace.
This step would provision operator Pod and wait until its state changes to “running” and installed operators shows “InstallSucceeded” as shown in the below figure.
You can navigate to Pods in kube-system project to verify if the operator Pod is created and running.
Setup Storage Nodes
It is recommended to use raw disk, no format and unmounted. In case, if you encounter issues such as KVDB is not creating. Try the following.
Try to zero out and see if that help. Make sure you uninstall Portworx with wipe option in yaml file for storage cluster mentioned in the uninstall Portworx section in this article.
# dd if=/dev/zero of=/dev/sdc bs=512 count=1Label Nodes
Make sure to label nodes if you are using internal KVDB (Key Value Data Base). Since, here we are using built-in internal KVDB in this setup
Use the following command to label OpenShift worker nodes to run built in KVDB.
# kubectl label nodes worker0.ocp4.sjc02.lab.cisco.com px/metadata-node=true# kubectl label nodes worker1.ocp4.sjc02.lab.cisco.com px/metadata-node=true# kubectl label nodes worker2.ocp4.sjc02.lab.cisco.com px/metadata-node=true
Generate Portworx specs in PX-Central
To install Portworx in OpenShift, we must first generate StorageCluster spec that will be used to deploy storage cluster
Generate the storagecluster spec using Portworx spec generator tool. It is available at https://docs.portworx.com/portworx-install-with-kubernetes/openshift/operator/2-deploy-px/#
Since we are deploying Portworx Essentials, select the same in the following screen as shown below.
We will be deploying using Portworx Operator. Check “Use the Portworx Operator”
On the Storage screen
Select “On Premises“
Select Manually specify disk. In this case, we will be dedicating one disk e.g. /dev/sdc for Portworks. You can specify more disks by clicking + sign. For simplicity, we are using “Auto create journal device” and skipping “KVDB device”. You can specify dedicated KVDB device such as /dev/sdd or so to separate KVDB I/O with storage I/O
On the network screen, keep all defaults.
On customize screen, select OpenShift 4+ as shown below.
Click Finish. Download the spec file by providing a name. this will give you .yaml file for provisioning storage cluster.
Download the spec file. Spec file will look like as below:
# SOURCE: https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=&b=true&s=%2Fdev%2Fsdc&j=auto&c=px-cluster&osft=true&stork=true&lh=true&st=k8s
kind: StorageCluster
apiVersion: core.libopenstorage.org/v1
metadata:
name: px-cluster
namespace: kube-system
annotations:
portworx.io/install-source: "https://install.portworx.com/?operator=true&mc=false&kbver=&oem=esse&user=&b=true&s=%2Fdev%2Fsdc&j=auto&c=px-cluster&osft=true&stork=true&lh=true&st=k8s"
portworx.io/is-openshift: "true"
portworx.io/misc-args: "--oem esse"
spec:
image: portworx/oci-monitor:2.6.1.3
imagePullPolicy: Always
kvdb:
internal: true
storage:
devices:
- /dev/sdc
journalDevice: auto
secretsProvider: k8s
stork:
enabled: true
args:
webhook-controller: "false"
userInterface:
enabled: true
autopilot:
enabled: true
providers:
- name: default
type: prometheus
params:
url: http://prometheus:9090
---
apiVersion: v1
kind: Secret
metadata:
name: px-essential
namespace: kube-system
data:
px-essen-user-id:
px-osb-endpoint: Note: Above yaml file is for reference purpose only. As is copy and paste of this yaml spec file will not work.
Create a secret for Portworx Essentials
If you’re running a Portworx Essentials cluster, then create the following secret with your Essential Entitlement ID:
You can get the Entitlement ID from your registered Portworx account
It will look similar to this: df12e1234–1234–12ab-a1c3-c124e123c1234 (This is for sample)
kubectl -n kube-system create secret generic px-essential \— from-literal=px-essen-user-id=YOUR_ESSENTIAL_ENTITLEMENT_ID \— from-literal=px-osb-endpoint='https://pxessentials.portworx.com/osb/billing/v1/register'
Verify it using OCP console as below
Create Storage Cluster
Apply the spec using the following command
# oc apply -f px-spec.yamlOr it can be installed directly from portworx.com. this is generated for you at the time of creating spec for storage cluster
# kubectl apply -f ‘https://install.portworx.com/2.6?operator=true&mc=false&kbver=&oem=esse&user=d123e123-1234-12eb-a1c1-c12e123c1234&b=true&s=%2Fdev%2Fsdc&j=auto&c=px-cluster&osft=true&stork=true&lh=true&st=k8s'After running the command. Pods will start getting provisioned in kube-system namespace as shown below. Make sure all Pods eventually gets into “Running” status and Readiness should show “Ready”
Verify that Portworx has deployed successfully by navigating to the Storage Cluster tab of the Installed Operators page. Once Portworx has fully deployed, the status will show as Online.
Validate Install
Validate by running the following to see if you have it installed correctly and successfully. Verify that all Pods in kube-system namespace are up and running.
[root@rhel1 ocp-install]# oc get pods -n kube-system
Check the status of the cluster Pods. Since we used 4 worker nodes to participate in forming Portworx cluster, you will have four cluster Pods. You can verify each one of them by running the following command.
oc exec px-cluster-35526b65–8e30–446d-af4f-be66f82e745d-25x4v -n kube-system — /opt/pwx/bin/pxctl status
Run the following command to verify the status of the disk allocated for Portworx.
[root@rhel1 ocp-install]# for i in {worker0,worker1,worker2,worker3}; do ssh core@$i.ocp4.sjc02.lab.cisco.com lsblk /dev/sdc; done;
Internal kvdb is sharing the storage device /dev/sdc2 to store its data. Remember, we did not select the separate dedicated device for kvdb in our spec generation. We are seeing two partition /dev/sdc1 and /dev/sdc2. Journaling device is /dev/sdc1
Uninstall Portworx
During install, you might encounter issues, and in some cases, you might want to uninstall and start fresh again. Uninstall is few clicks for Portworx, however, if the steps outlined below are not taken, you may not have it uninstall completely and creates more problems in fresh install.
For uninstall add the following in storage cluster yaml file. Go to Installed Operator, click StorageCluster and edit YAML. The following line should be in your YAML file. If it is not there, add it, and save the YAML file.
spec:
deleteStrategy:
type: UninstallAndWipe
In Installed Operators. Click Storage Cluster
Click Delete StorageCluster as shown below
Unlink the cluster from your profile, otherwise, new cluster install will have license expiration issue. Visit your profile in Portworx.com and click unlink.
Provision Volumes
Create Storage Classes
In RHOCP console click “Storage Classes”, then click “Create Storage Class” button
Click Edit YAML.
Copy and paste the followingkind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: portworx-sc
provisioner: kubernetes.io/portworx-volume
parameters:
repl: "1"
Click create.
Storage class can also be created by saving the above yaml file and run the following
kubectl create -f portworx-sc.yaml
verify if the storage class has been created by running the following
[root@rhel1 ocp-install]# kubectl describe storageclass portworx-sc
Name: portworx-sc
IsDefaultClass: No
Annotations: <none>
Provisioner: kubernetes.io/portworx-volume
Parameters: repl=1
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
[root@rhel1 ocp-install]#Create Persistent Volume Claim
Use the following yaml file and save it as portworx-volume-pvcsc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvcsc001
annotations:
volume.beta.kubernetes.io/storage-class: portworx-sc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2GiRun the following command
kubectl create -f portworx-volume-pvcsc.yamlYou can create the same in the GUI by clicking Persistent Volume Claim and copy and paste the above YAML.
Run the following command to validate
[root@rhel1 ocp-install]# kubectl describe pvc pvcsc001 -n kube-system
Name: pvcsc001
Namespace: kube-system
StorageClass: portworx-sc
Status: Bound
Volume: pvc-00fb7cd2-638d-4f3b-a8bd-226bdaeffa97
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-class: portworx-sc
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/portworx-volume
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 79s persistentvolume-controller Successfully provisioned volume pvc-00fb7cd2-638d-4f3b-a8bd-226bdaeffa97 using kubernetes.io/portworx-volume
[root@rhel1 ocp-install]#
[root@rhel1 ocp-install]#Create Pod which uses Persistent Volume Claim with storage class
apiVersion: v1
kind: Pod
metadata:
name: pvpod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/test-webserver
volumeMounts:
- name: test-volume
mountPath: /test-portworx-volume
volumes:
- name: test-volume
persistentVolumeClaim:
claimName: pvcsc001Create POD by running the following command
kubectl create -f portworx-volume-pvcscpod.yamlWe can create the same Pod in RHOCP console as below.
Click Workload → Pods → Create Pod as show below
Copy and paste the above mentioned yaml file content
click Create. This would provision Pod with volume mounts from persistent storage
Conclusion
Kubernetes persistent volumes are user-provisioned storage volumes assigned to a Kubernetes cluster. Persistent volumes’ life-cycle is independent from any pod using it. Thus, persistent volumes are perfect for use cases in which you need to retain data regardless of the unpredictable life process of Kubernetes pods.
Without persistent volumes, maintaining services as common as a database would be impossible. Whenever a pod gets replaced, the data gained during the life-cycle of that pod would be lost. However, thanks to persistent volumes, data is contained in a consistent state.
