Kubernetes deployment#


Install Xorbits on the machine where you plan to run the kubernetes deploy code. Refer to installation document.


Make sure a K8s cluster is properly installed on your machine(s), and enable the ingress service.

For example, if you use Minikube, start a cluster and enable ingress like this:

$ minikube start
$ minikube addons enable ingress

Follow minikube documentation to verify whether ingress is enabled correctly.

For MacOS with docker driver, docker-mac-net-connect is needed due to its limitation:

# Install via Homebrew
$ brew install chipmk/tap/docker-mac-net-connect

# Run the service and register it to launch at boot
# Notice that this command must be executed with sudo
$ sudo brew services start chipmk/tap/docker-mac-net-connect

Then deploy Xorbits cluster, for example:

from kubernetes import config
from xorbits.deploy.kubernetes import new_cluster
cluster = new_cluster(config.new_client_from_config(), worker_cpu=1, worker_mem='4g')

Please make sure kubectl uses your kubernetes cluster context.

You should be able to see Xorbits endpoint http://<ingress_service_ip>:80 is ready! soon, and you can access the web UI of your Xorbits cluster using the endpoint.

new_cluster api refers to xorbits.deploy.kubernetes.client.new_cluster().

To verify the cluster:

import xorbits.pandas as pd
print(pd.DataFrame({'a': [1,2,3,4]}).sum())

Docker Image#

By default, the image tagged by xprobe/xorbits:<xorbits version> on our Dockerhub is used in the kubernetes deployment. Each released version of Xorbits has its image, distinguished by the <xorbits version>.


Since v0.1.2, each release image of xorbits supports python 3.7, 3.8, 3.9 and 3.10, with -py<python_version> as the suffix of the image tag.

For example, xprobe/xorbits:v0.1.2-py3.10 means the image is built on python 3.10.

By default, the image tagged by xprobe/xorbits:<xorbits version> still exists, and it is built on python 3.9.

Since v0.2.0, Xorbits automatically selects the deployment image according to your local python version by default. For example, if your local python version is 3.9, Xorbits uses the image tagged by xprobe/xorbits:<xorbits version>-py3.9 during deployment.

Since v0.2.1, Xorbits image has removed support for python 3.7 and introduced support for python 3.11. The image tagged by xprobe/xorbits:<xorbits version> is built on python 3.10.

If you need to build an image from source, the related Dockerfiles exists at this position for reference. You can follow the Docker document to build your own Xorbits image.

After you build your own image, push it to a image repository accessible by your K8s cluster, e.g. your own DockerHub namespace.

Finally, specify your own image during the deployment process through the image option of the xorbits.deploy.kubernetes.client.new_cluster() api.

Install Python Packages#

Refer DockerFile for the python packages included in the Xorbits image. If you want to install additional python packages in your Xorbits K8s cluster, use pip and conda options of the xorbits.deploy.kubernetes.client.new_cluster() api.

Please make sure your K8s cluster can access the corresponding channel of conda or PyPi, when using pip and conda options.

JuiceFS on Kubernetes#

Xorbits is now able to integrate JuiceFS, a distributed POSIX file system that can be easily integrated with Kubernetes to provide persistent storage, as one of the storage backend.



Install Xorbits on the machine where you plan to deploy Kubernetes with JuiceFS. Refer to installation document.

Metadata Storage#

JuiceFS decouples data and metadata. Many databases are supported. See How to Set Up Metadata Engine and choose an appropriate metadata storage.

In our example here, we select Redis as our metadata storage.

Follow Configuring Redis using a ConfigMap and create a pod inside default namespace.

You should set its maxmemory as 50mb since 2mb in the example is too small.

Make sure redis pod is running:

$ kubectl get po redis
redis   1/1     Running   0           6d6h

Check redis pod’s IP address. In this example, IP for redis is

$ kubectl get po redis -o custom-columns=IP:.status.podIP --no-headers

You can also check how much CPU and memory resources redis pod gets by running

$ kubectl get po redis

and check the corresponding fields.


$ pip install kubernetes

Follow previous Kubernetes section to initialize a K8s cluster on your machine.

Install kubectl, a command-line tool for interacting with Kubernetes clusters and verify its installation.

$ kubectl version --client
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7

JuiceFS Installation#

We will walk you through the process of installing JuiceFS on a Kubernetes cluster, enabling you to leverage its features and benefits. There are three ways to use JuiceFS on K8S Use JuiceFS on Kubernetes.

But our implementation in k8s must rely on CSI since CSI provides better portability, enhanced isolation, and more advanced features.

Reference Page: JuiceFS CSI Driver

JuiceFS CSI Driver#

Installation with Helm#

Reference Page: JuiceFS Installation with Helm

Firstly, Install Helm

Secondly, download the Helm chart for JuiceFS CSI Driver

$ helm repo add juicefs https://juicedata.github.io/charts/
$ helm repo update
$ helm fetch --untar juicefs/juicefs-csi-driver
$ cd juicefs-csi-driver
# Installation configurations is included in values.yaml, review this file and modify to your needs
$ cat values.yaml

You should be careful with limits and requests of cpu and memory. Change according to your system settings. Here we give you the minimal configuration.

    cpu: 100m
    memory: 50Mi
    cpu: 100m
    memory: 50Mi

Thirdly, execute below commands to deploy JuiceFS CSI Driver:

$ helm install juicefs-csi-driver juicefs/juicefs-csi-driver -n kube-system -f ./values.yaml`

Fourthly, verify installation

$ kubectl -n kube-system get pods -l app.kubernetes.io/name=juicefs-csi-driver
NAME                       READY   STATUS    RESTARTS   AGE
juicefs-csi-controller-0   3/3     Running   0          22m
juicefs-csi-node-v9tzb     3/3     Running   0          14m

If you want to delete JuiceFS CSI Driver in the future, you can run:

$ helm list -n kube-system
NAME                NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
juicefs-csi-driver  kube-system     1               2023-05-31 03:12:16.717087425 +0000 UTC deployed        juicefs-csi-driver-0.15.1       0.19.0

$ helm uninstall juicefs-csi-driver -n kube-system
Create and use PV#

If you want to directly use JuiceFS on K8S, you can skip this Create and use PV section because in Xorbits, new_cluster function would create secret, pv, and pvc for you.

If you want to understand how the mounting works and the meaning of each parameter in the configurations, you can walk through this section.

JuiceFS leverages persistent volumes to store data.

Reference Page: Create and use PV

We would create several YAML files. Validate their formats on YAML validator before usage.

First, create Kubernetes Secret:

$ vim secret.yaml

Write the following into the yaml file:

apiVersion: v1
kind: Secret
  name: juicefs-secret
type: Opaque
  name: jfs
  metaurl: redis:// # Replace with your own metadata storage URL
  storage: file # Check out full supported list on `Set Up Object Storage <https://juicefs.com/docs/community/how_to_setup_object_storage/>`_.
  bucket: /var # Bucket URL. Read `Set Up Object Storage <https://juicefs.com/docs/community/how_to_setup_object_storage/>`_ to learn how to setup different object storage.

In our case, we do not need access-key and secret-key. Add if you need object storage credentials.

Secondly, create Persistent Volume and Persistent Volume Claim with static provisioning

Read Usage to learn the difference between static and dynamic provisioning.

$ vim static_provisioning.yaml

Write the following into the yaml file:

apiVersion: v1
kind: PersistentVolume
  name: juicefs-pv
    juicefs-name: juicefs-fs # Works as a match-label for selector
  # For now, JuiceFS CSI Driver doesn't support setting storage capacity for static PV. Fill in any valid string is fine.
    storage: 10Pi
  volumeMode: Filesystem
  mountOptions: ["subdir=/data/subdir"]  # Mount in sub directory to achieve data isolation. See https://juicefs.com/docs/csi/guide/pv/#create-storage-class for more references.
    - ReadWriteMany # accessModes is restricted to ReadWriteMany because it's the most suitable mode for our system. See https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes for more reference.
  persistentVolumeReclaimPolicy: Retain # persistentVolumeReclaimPolicy is restricted to Retain for Static provisioning. See https://juicefs.com/docs/csi/guide/resource-optimization/#reclaim-policy for more references.
    # A CSIDriver named csi.juicefs.com is created during installation
    driver: csi.juicefs.com
    # volumeHandle needs to be unique within the cluster, simply using the PV name is recommended
    volumeHandle: juicefs-pv
    fsType: juicefs
    # Reference the volume credentials (Secret) created in previous step
    # If you need to use different credentials, or even use different JuiceFS volumes, you'll need to create different volume credentials
      name: juicefs-secret
      namespace: xorbits-ns-cc53e351744f4394b20180a0dafd8b91 # change the namespace to your Xorbits namespace
apiVersion: v1
kind: PersistentVolumeClaim
  name: juicefs-pvc
  namespace: xorbits-ns-cc53e351744f4394b20180a0dafd8b91 # change the namespace to your Xorbits namespace
    - ReadWriteMany
  volumeMode: Filesystem
  # Must use an empty string as storageClassName
  # Meaning that this PV will not use any StorageClass, instead will use the PV specified by selector
  storageClassName: ""
  # For now, JuiceFS CSI Driver doesn't support setting storage capacity for static PV. Fill in any valid string that's lower than the PV capacity.
      storage: 10Pi
      juicefs-name: juicefs-fs

Thirdly, apply Secret, PV, and PVC to your namespace and verify:

Create your namespace (or Xorbits namespace) and run the following:

$ kubectl apply -f secret.yaml -n {your_namespace}
$ kubectl apply -f static_provisioning -n {your_namespace}
$ kubectl get pv
NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                       STORAGECLASS   REASON   AGE
juicefs-pv    10Pi       RWX            Retain           Bound    xorbits-ns-cc53e351744f4394b20180a0dafd8b91/juicefs-pvc                             17h

$ kubectl get pvc --all-namespaces
NAMESPACE                                     NAME           STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
xorbits-ns-cc53e351744f4394b20180a0dafd8b91   juicefs-pvc    Bound    juicefs-pv    10Pi       RWX                           17h

Fourthly, create a pod

$ vim pod.yaml

Write the following into the yaml file:

apiVersion: v1
kind: Pod
  name: juicefs-app
  namespace: xorbits-ns-cc53e351744f4394b20180a0dafd8b91 # Replace with your namespace
  - args:
    - -c
    - while true; do echo $(date -u) >> /juicefs-data/out.txt; sleep 5; done
    - /bin/sh
    image: centos
    name: app
    - mountPath: /juicefs-data
      name: data
        cpu: 10m
  - name: data
      claimName: juicefs-pvc

After pod is up and running, you’ll see out.txt being created by the container inside the JuiceFS mount point /juicefs-data.

Congratulations! You have successfully set up JuiceFS on Kubernetes by yourself.

Deploy Cluster#

Before you deploy, go to our DockerHub and get the latest docker image.

Check the image by running:

$ docker image ls
REPOSITORY                              TAG                                      IMAGE ID       CREATED         SIZE
xprobe/xorbits                          v0.4.0-py3.10                            4166e1b0676d   5 days ago      3.38GB

Deploy Xorbits cluster, for example:

from kubernetes import config
from xorbits.deploy.kubernetes import new_cluster

cluster = new_cluster(config.new_client_from_config(), worker_num=1, worker_cpu=1, worker_mem='1g', supervisor_cpu=1, supervisor_mem='1g', image='xprobe/xorbits:v0.4.0-py3.10', external_storage='juicefs',external_storage_config={"metadata_url": "redis://","bucket": "/var", "mountPath": "/juicefs-data"},)

Currently, only juicefs is supported as one of our storage backend. When you want to switch from shared memory to JuiceFS, You must specify external_storage='juicefs' explicitly when you initialize a new cluster.

JuiceFS has corresponding parameters which you should specify in a dictionary named external_storage_config.

You must explicitly specify connection URL metadata_url, in our case redis:// is the IP address of the Redis server, and 6379 is the default port number on which the Redis server is listening. 1 represents the Redis database number.

Specify bucket URL with bucket or use its default value /var if you do not want to change the directory for bucket. See Set Up Object Storage to set up different object storage.

Specify mount path with mountPath or use its default value /juicefs-data.

After several minutes, you would see Xorbits endpoint http://<ingress_service_ip>:80 is ready!

Verify the cluster by running a simple task.

import xorbits
import xorbits.pandas as pd
pd.DataFrame({'a': [1,2,3,4]}).sum()

If the cluster is working, the output should be 10.

Verify the storage#

In our example, we mount JuiceFS storage data in /juicefs-data, which is also the default path.

Firstly, get the namespace that starts with xorbits and get its pods.

$ kubectl get namespaces
NAME                                          STATUS   AGE
default                                       Active   38d
kube-node-lease                               Active   38d
kube-public                                   Active   38d
kube-system                                   Active   38d
xorbits-ns-cc53e351744f4394b20180a0dafd8b91   Active   4m5s

$ kubectl get po -n xorbits-ns-cc53e351744f4394b20180a0dafd8b91
NAME                                 READY   STATUS             RESTARTS   AGE
xorbitssupervisor-84754bf5f4-dcstd   1/1     Running            0          80s
xorbitsworker-5b9b976767-sfpkk       1/1     Running            0          80s

Then, execute an interactive shell (bash) inside a pod which belongs to the Xorbits namespace. You can verify either supervisor pod or worker pod, or both.

$ kubectl exec -it xorbitssupervisor-84754bf5f4-dcstd -n xorbits-ns-cc53e351744f4394b20180a0dafd8b91 -- /bin/bash

Check if data is stored in /juicefs-data.

You should see a similar hex string like 9c3e069a-70d9-4874-bad6-d608979746a0, meaning that data inside JuiceFS is successfully mounted!

$ cd ..
$ cd juicefs-data
$ ls
$ cat 9c3e069a-70d9-4874-bad6-d608979746a0

You should see the serialized output of the simple task which may not be human-readable. It should contain pandas, meaning that it matches our simple task!

Manage the Xorbits cluster & Debug#

You can get Xorbits namespace, check the status of Xorbits pods, and check Xorbits UI by following Detailed tutorial: Deploying and Running Xorbits on Amazon EKS..

If everything works fine, now you can easily scale up and down the storage resources by adding or deleting pods inside the namespace.