Kubernetes deployment

Prerequisites

Install Xorbits on the machine where you plan to run the kubernetes deploy code. Refer to installation document.

Kubernetes

Make sure a K8s cluster is properly installed on your machine(s), and enable the ingress service.

For example, if you use Minikube, start a cluster and enable ingress like this:

$ minikube start
$ minikube addons enable ingress

Follow minikube documentation to verify whether ingress is enabled correctly.

For MacOS with docker driver, docker-mac-net-connect is needed due to its limitation:

# Install via Homebrew
$ brew install chipmk/tap/docker-mac-net-connect

# Run the service and register it to launch at boot
# Notice that this command must be executed with sudo
$ sudo brew services start chipmk/tap/docker-mac-net-connect

Then deploy Xorbits cluster, for example:

from kubernetes import config
from xorbits.deploy.kubernetes import new_cluster
cluster = new_cluster(config.new_client_from_config(), worker_cpu=1, worker_mem='4g')

Please make sure kubectl uses your kubernetes cluster context.

You should be able to see Xorbits endpoint http://<ingress_service_ip>:80 is ready! soon, and you can access the web UI of your Xorbits cluster using the endpoint.

new_cluster api refers to xorbits.deploy.kubernetes.client.new_cluster().

To verify the cluster:

import xorbits.pandas as pd
print(pd.DataFrame({'a': [1,2,3,4]}).sum())

Docker Image

By default, the image tagged by xprobe/xorbits:<xorbits version> on our Dockerhub is used in the kubernetes deployment. Each released version of Xorbits has its image, distinguished by the <xorbits version>.

Note

Since v0.1.2, each release image of xorbits supports python 3.7, 3.8, 3.9 and 3.10, with -py<python_version> as the suffix of the image tag.

For example, xprobe/xorbits:v0.1.2-py3.10 means the image is built on python 3.10.

By default, the image tagged by xprobe/xorbits:<xorbits version> still exists, and it is built on python 3.9.

Since v0.2.0, Xorbits automatically selects the deployment image according to your local python version by default. For example, if your local python version is 3.9, Xorbits uses the image tagged by xprobe/xorbits:<xorbits version>-py3.9 during deployment.

If you need to build an image from source, the related Dockerfiles exists at this position for reference. You can follow the Docker document to build your own Xorbits image.

After you build your own image, push it to a image repository accessible by your K8s cluster, e.g. your own DockerHub namespace.

Finally, specify your own image during the deployment process through the image option of the xorbits.deploy.kubernetes.client.new_cluster() api.

Install Python Packages

Refer DockerFile for the python packages included in the Xorbits image. If you want to install additional python packages in your Xorbits K8s cluster, use pip and conda options of the xorbits.deploy.kubernetes.client.new_cluster() api.

Please make sure your K8s cluster can access the corresponding channel of conda or PyPi, when using pip and conda options.