Kubernetes deployment#
Prerequisites#
Install Xorbits on the machine where you plan to run the kubernetes deploy code. Refer to installation document.
Kubernetes#
Make sure a K8s cluster is properly installed on your machine(s), and enable the ingress service.
For example, if you use Minikube, start a cluster and enable ingress like this:
$ minikube start
$ minikube addons enable ingress
Follow minikube documentation to verify whether ingress is enabled correctly.
For MacOS with docker driver, docker-mac-net-connect is needed due to its limitation:
# Install via Homebrew
$ brew install chipmk/tap/docker-mac-net-connect
# Run the service and register it to launch at boot
# Notice that this command must be executed with sudo
$ sudo brew services start chipmk/tap/docker-mac-net-connect
Then deploy Xorbits cluster, for example:
from kubernetes import config
from xorbits.deploy.kubernetes import new_cluster
cluster = new_cluster(config.new_client_from_config(), worker_cpu=1, worker_mem='4g')
Please make sure kubectl
uses your kubernetes cluster context.
You should be able to see Xorbits endpoint http://<ingress_service_ip>:80 is ready!
soon, and
you can access the web UI of your Xorbits cluster using the endpoint.
new_cluster
api refers to xorbits.deploy.kubernetes.client.new_cluster()
.
To verify the cluster:
import xorbits.pandas as pd
print(pd.DataFrame({'a': [1,2,3,4]}).sum())
Docker Image#
By default, the image tagged by xprobe/xorbits:<xorbits version>
on our Dockerhub
is used in the kubernetes deployment. Each released version of Xorbits has its image, distinguished by the <xorbits version>
.
Note
Since v0.1.2
, each release image of xorbits supports python 3.7
, 3.8
, 3.9
and 3.10
,
with -py<python_version>
as the suffix of the image tag.
For example, xprobe/xorbits:v0.1.2-py3.10
means the image is built on python 3.10
.
By default, the image tagged by xprobe/xorbits:<xorbits version>
still exists, and it is built on python 3.9
.
Since v0.2.0
, Xorbits automatically selects the deployment image according to your local python version by default.
For example, if your local python version is 3.9
, Xorbits uses the image tagged by xprobe/xorbits:<xorbits version>-py3.9
during deployment.
If you need to build an image from source, the related Dockerfiles exists at this position for reference. You can follow the Docker document to build your own Xorbits image.
After you build your own image, push it to a image repository accessible by your K8s cluster, e.g. your own DockerHub namespace.
Finally, specify your own image during the deployment process through the image
option of the xorbits.deploy.kubernetes.client.new_cluster()
api.
Install Python Packages#
Refer DockerFile for the python packages included in the Xorbits image.
If you want to install additional python packages in your Xorbits K8s cluster, use pip
and conda
options of the xorbits.deploy.kubernetes.client.new_cluster()
api.
Please make sure your K8s cluster can access the corresponding channel of conda or PyPi, when using pip
and conda
options.