Cluster deployment#

If you have some machines either on the cloud or not, and you want to deploy Xorbits via command lines, you can refer to this documentation.


First, ensure Xorbits is correctly installed on each machine, if not, see installation document.

Starting Supervisor#

Among the machines, pick at least one as the supervisor which ships with a web service as well, starting supervisor via command:

xorbits-supervisor -H <supervisor_ip> -p <supervisor_port> -w <web_port>

Or using python -m:

python -m xorbits.supervisor -H <supervisor_ip> -p <supervisor_port> -w <web_port>

<web_port> is used for creating a web service, which is used for:

  1. Provide a web UI to show cluster status and detailed information of submitted tasks.

  2. Accept connections from clients when users call xorbits.init(“<supervisor_ip>:<web_port>”) to connect to a cluster.

Starting Workers#

The rest of the machines can be started as workers via command:

xorbits-worker -H <worker_ip> -p <worker_port> -s <supervisor_ip>:<supervisor_port>

Or using python -m:

python -m xorbits.worker -H <worker_ip> -p <worker_port> -s <supervisor_ip>:<supervisor_port>

Connecting to Created Cluster#

Now, you can connect to the supervisor from anywhere that can run Python code.

import xorbits

Replace the <supervisor_ip> with the supervisor IP that you just specified and <supervisor_web_port> with the supervisor web port.

Xorbits Web UI#

You can open a web browser and type http://<supervisor_ip>:<supervisor_web_port> to open Xorbits Web UI to look up resource usage of workers and execution progress of submitted tasks.

Command Line Options#

Common Options#

Common Command line options are listed below.




Service IP binding, by default


Port of the service. If absent, a randomized port will be used


Path to service configuration file. Absent when use default configuration.


List of supervisor endpoints, separated by commas. Useful for workers to spot supervisors, or when you want to run more than one supervisor


Log level, can be debug, info, warning, error


Log format, can be Python logging format


Python logging configuration file, logging.conf by default


Whether to use uvloop to accelerate, auto by default

Extra Options for Supervisors#




Port of web service in supervisor

Extra Options for Workers#




Number of CPU cores to use. If absent, the value will be the available number of cores


Number of IO processes for network operations. 1 by default


Index of CUDA devices to use. If not specified, all devices will be used. Specifying an empty string will ignore all devices

Custom configuration#

Default configuration can be modified by specifying a -f flag. Provide the path of your yml file which contains custom configuration along with -f flag.

For example#

If the user want to modify transfer_block_size` and node_timeout, specify -f your-config.yml.


"@inherits": "@default"
    transfer_block_size: 10 * 1024 ** 2
node_timeout: 1200


For instance, if you want to start a Xorbits cluster with one supervisor and two workers, you can run commands below (memory and CPU tunings are omitted):

On Supervisor (

xorbits-supervisor -H -p 7001 -w 7005

On Worker 1 (

xorbits-worker -H -p 7003 -s

On Worker 2 (

xorbits-worker -H -p 7004 -s

On a client which is able to connect to supervisor, you can run the Python code below to verify:

import xorbits
import xorbits.numpy as np

print(np.random.rand(100, 100).mean())