Ray · distributed computing

Also known as: Ray RLlib · Anyscale Ray

On this page 7

Ray is an open-source distributed computing framework that lets ordinary Python code run in parallel across many cores, GPUs and machines through a small set of primitives: remote tasks, stateful actors and a shared object store. Among AI frameworks & tools, it occupies the layer below model libraries such as PyTorch or TensorFlow: it does not define neural networks, it schedules and coordinates the processes that train, tune and serve them.

Ray began as a research project at the UC Berkeley RISELab and is now developed by Anyscale and a large open-source community, which is why the names Anyscale Ray and Ray RLlib appear as synonyms. Its main appeal in artificial intelligence is that one runtime covers data preprocessing, distributed training, hyperparameter search, reinforcement learning and model serving, so teams avoid stitching together separate cluster systems for each stage.

How the Ray runtime distributes work

Ray distributes work by turning decorated Python functions and classes into units the cluster scheduler can place anywhere. Calling a function marked with @ray.remote returns immediately with an ObjectRef, a future that points to a result that will exist somewhere in the cluster. Code calls ray.get only when it actually needs the value, which allows thousands of tasks to run concurrently.

A Ray cluster consists of one head node and any number of worker nodes. The head node hosts the Global Control Service (GCS), which stores cluster metadata such as node membership, actor locations and job state. Every node runs a raylet, a local daemon that combines a scheduler and an object manager, and a pool of worker processes that execute the code.

The following table summarizes the core components and the role each one plays at runtime.

ComponentWhere it runsResponsibility
TaskAny worker processStateless function call executed remotely
ActorDedicated worker processStateful object whose methods run in order
Object storeShared memory on each nodeHolds immutable results for zero-copy reads
RayletEvery nodeLocal scheduling and object transfer
GCSHead nodeCluster metadata and fault-tolerance records
DriverHead node or clientRuns the main program that submits work
Core components of a Ray cluster

Ray uses a bottom-up, decentralized scheduling model: a raylet first tries to run a task locally and forwards it to another node only when local resources are insufficient. Ownership of each object belongs to the worker that created its reference, and this ownership-based reference counting decides when results can be freed or must be reconstructed after a failure.

Tasks, actors and objects in practice

Choosing between a task and an actor is the first design decision in any Ray program, and it depends on whether the computation needs to keep state. Tasks suit embarrassingly parallel work such as transforming file shards, while actors suit components that hold a loaded model, a simulator or a connection pool for a long time.

PrimitiveBest suited forMain caution
Remote taskIndependent batch transformsVery small tasks add scheduling overhead
ActorModel replicas, simulators, parameter holdersA single actor processes calls serially by default
Async actorI/O-bound services and request handlingBlocking code inside stalls the event loop
ObjectRef passingSharing large arrays between stepsHolding references keeps memory pinned
Placement groupGang scheduling of GPU workersReserved bundles stay idle if unused
When to use each Ray primitive

Resource requests are declared explicitly, for example num_cpus=2 or num_gpus=1, and Ray treats them as logical scheduling hints rather than hard isolation. A task that claims one CPU can still spawn many threads, so accurate declarations are essential for predictable packing. Custom resources let operators label nodes, for instance to pin work to machines with a specific accelerator or local dataset.

Large objects are written once into the node’s shared-memory object store and read by co-located workers without copying, which matters for NumPy arrays and Arrow tables. When memory fills, Ray spills objects to local disk or external storage, which preserves correctness but can slow a pipeline sharply.

The Ray AI libraries

On top of the core runtime, Ray ships a set of domain libraries for the machine learning lifecycle, often called the Ray AI libraries. Each library uses the same cluster and object store, so outputs of one stage can flow into the next without exporting files between systems.

LibraryPurposeTypical integration
Ray DataStreaming preprocessing and batch inferenceParquet, images, Hugging Face datasets
Ray TrainDistributed data-parallel trainingPyTorch, PyTorch Lightning, XGBoost
Ray TuneHyperparameter search and schedulingOptuna, HyperOpt, early stopping
Ray ServeOnline model serving and compositionFastAPI, LLM inference engines
RLlibScalable reinforcement learningGymnasium environments, multi-agent setups
Ray libraries and their purpose

Ray Train does not replace the training framework: it launches and coordinates the workers and sets up process groups so that PyTorch distributed or DeepSpeed can do the gradient synchronization. Ray Tune adds trial schedulers such as ASHA and population-based training, which stop weak configurations early and reallocate their resources to promising ones.

How Ray compares with other distributed computing options

Ray is best understood as a general-purpose task and actor runtime, which places it between low-level communication libraries and high-level data engines. The comparison below shows where its trade-offs differ from tools that teams often evaluate alongside it.

SystemCore abstractionStrongest fit
RayTasks, actors, object storeHeterogeneous AI pipelines in Python
Apache SparkPartitioned datasets and SQLLarge tabular ETL and analytics
DaskTask graphs over arrays and dataframesScaling pandas and NumPy workflows
PyTorch distributedCollective communication groupsSynchronous gradient exchange
Kubernetes jobsContainers and podsCoarse-grained batch scheduling
Ray compared with related systems

In production, these tools are frequently combined rather than substituted. A common pattern runs Ray on Kubernetes through the KubeRay operator, uses Spark for upstream warehouse extraction and lets Ray handle the GPU-heavy training and inference stages where fine-grained scheduling pays off.

Limitations and common mistakes with Ray

Ray removes much of the boilerplate of distributed programming, but it does not remove the physics of networks and memory. Most production incidents trace back to a small set of recurring patterns.

  • Tasks that are too small: when each call does only microseconds of work, scheduling and serialization overhead dominate; batching items into larger tasks restores efficiency.
  • Calling ray.get in a loop: fetching results one by one serializes the program; submitting all tasks first and collecting them together keeps the cluster busy.
  • Object store pressure: holding many references prevents eviction, triggers spilling and can end in out-of-memory errors on worker nodes.
  • Head node as a single point of failure: without GCS fault tolerance backed by external storage such as Redis, losing the head node stops the cluster.
  • Version drift: the driver and all nodes must run matching Ray and Python versions, or serialization fails in confusing ways.

Fault tolerance is real but not automatic for every case. Ray can retry failed tasks and rebuild lost objects from lineage, yet actor state is lost on failure unless the application checkpoints it explicitly or uses library-level checkpointing in Ray Train and Ray Tune.

In simple terms

Ray is a coordinator for computer work. Imagine a large kitchen with many cooks. Instead of one chef doing every step, a head chef hands out jobs, keeps track of who is busy and places finished dishes on a shared counter where anyone can pick them up.

Ray plays that head-chef role for computers. A programmer writes ordinary Python, marks which parts can run separately, and Ray spreads those parts across many processors or machines. It also remembers where every result is stored. The key takeaway is that Ray makes one program act like many without the programmer managing each machine by hand.

A practical example

A machine learning team at an online retailer needs to fine-tune a product-recommendation model every night on about two terabytes of click logs. Previously, a single eight-GPU server took roughly ten hours, which regularly ran into business hours.

The team moves the pipeline to a Ray cluster on Kubernetes with four GPU nodes and several CPU-only nodes. Ray Data streams and preprocesses the logs on the CPU nodes while Ray Train runs data-parallel PyTorch on 32 GPUs, and Ray Tune tests twelve learning-rate and batch-size combinations with early stopping. Training results are checkpointed to object storage after each epoch.

The nightly job now finishes in under three hours, failed spot instances are replaced without restarting the run, and the best configuration is deployed through Ray Serve the next morning.