Ray is an open-source distributed computing framework that lets ordinary Python code run in parallel across many cores, GPUs and machines through a small set of primitives: remote tasks, stateful actors and a shared object store. Among AI frameworks & tools, it occupies the layer below model libraries such as PyTorch or TensorFlow: it does not define neural networks, it schedules and coordinates the processes that train, tune and serve them.
Ray began as a research project at the UC Berkeley RISELab and is now developed by Anyscale and a large open-source community, which is why the names Anyscale Ray and Ray RLlib appear as synonyms. Its main appeal in artificial intelligence is that one runtime covers data preprocessing, distributed training, hyperparameter search, reinforcement learning and model serving, so teams avoid stitching together separate cluster systems for each stage.
How the Ray runtime distributes work
Ray distributes work by turning decorated Python functions and classes into units the cluster scheduler can place anywhere. Calling a function marked with @ray.remote returns immediately with an ObjectRef, a future that points to a result that will exist somewhere in the cluster. Code calls ray.get only when it actually needs the value, which allows thousands of tasks to run concurrently.
A Ray cluster consists of one head node and any number of worker nodes. The head node hosts the Global Control Service (GCS), which stores cluster metadata such as node membership, actor locations and job state. Every node runs a raylet, a local daemon that combines a scheduler and an object manager, and a pool of worker processes that execute the code.
The following table summarizes the core components and the role each one plays at runtime.
| Component | Where it runs | Responsibility |
|---|---|---|
| Task | Any worker process | Stateless function call executed remotely |
| Actor | Dedicated worker process | Stateful object whose methods run in order |
| Object store | Shared memory on each node | Holds immutable results for zero-copy reads |
| Raylet | Every node | Local scheduling and object transfer |
| GCS | Head node | Cluster metadata and fault-tolerance records |
| Driver | Head node or client | Runs the main program that submits work |
Ray uses a bottom-up, decentralized scheduling model: a raylet first tries to run a task locally and forwards it to another node only when local resources are insufficient. Ownership of each object belongs to the worker that created its reference, and this ownership-based reference counting decides when results can be freed or must be reconstructed after a failure.
Tasks, actors and objects in practice
Choosing between a task and an actor is the first design decision in any Ray program, and it depends on whether the computation needs to keep state. Tasks suit embarrassingly parallel work such as transforming file shards, while actors suit components that hold a loaded model, a simulator or a connection pool for a long time.
| Primitive | Best suited for | Main caution |
|---|---|---|
| Remote task | Independent batch transforms | Very small tasks add scheduling overhead |
| Actor | Model replicas, simulators, parameter holders | A single actor processes calls serially by default |
| Async actor | I/O-bound services and request handling | Blocking code inside stalls the event loop |
| ObjectRef passing | Sharing large arrays between steps | Holding references keeps memory pinned |
| Placement group | Gang scheduling of GPU workers | Reserved bundles stay idle if unused |
Resource requests are declared explicitly, for example num_cpus=2 or num_gpus=1, and Ray treats them as logical scheduling hints rather than hard isolation. A task that claims one CPU can still spawn many threads, so accurate declarations are essential for predictable packing. Custom resources let operators label nodes, for instance to pin work to machines with a specific accelerator or local dataset.
Large objects are written once into the node’s shared-memory object store and read by co-located workers without copying, which matters for NumPy arrays and Arrow tables. When memory fills, Ray spills objects to local disk or external storage, which preserves correctness but can slow a pipeline sharply.
The Ray AI libraries
On top of the core runtime, Ray ships a set of domain libraries for the machine learning lifecycle, often called the Ray AI libraries. Each library uses the same cluster and object store, so outputs of one stage can flow into the next without exporting files between systems.
| Library | Purpose | Typical integration |
|---|---|---|
| Ray Data | Streaming preprocessing and batch inference | Parquet, images, Hugging Face datasets |
| Ray Train | Distributed data-parallel training | PyTorch, PyTorch Lightning, XGBoost |
| Ray Tune | Hyperparameter search and scheduling | Optuna, HyperOpt, early stopping |
| Ray Serve | Online model serving and composition | FastAPI, LLM inference engines |
| RLlib | Scalable reinforcement learning | Gymnasium environments, multi-agent setups |
Ray Train does not replace the training framework: it launches and coordinates the workers and sets up process groups so that PyTorch distributed or DeepSpeed can do the gradient synchronization. Ray Tune adds trial schedulers such as ASHA and population-based training, which stop weak configurations early and reallocate their resources to promising ones.
How Ray compares with other distributed computing options
Ray is best understood as a general-purpose task and actor runtime, which places it between low-level communication libraries and high-level data engines. The comparison below shows where its trade-offs differ from tools that teams often evaluate alongside it.
| System | Core abstraction | Strongest fit |
|---|---|---|
| Ray | Tasks, actors, object store | Heterogeneous AI pipelines in Python |
| Apache Spark | Partitioned datasets and SQL | Large tabular ETL and analytics |
| Dask | Task graphs over arrays and dataframes | Scaling pandas and NumPy workflows |
| PyTorch distributed | Collective communication groups | Synchronous gradient exchange |
| Kubernetes jobs | Containers and pods | Coarse-grained batch scheduling |
In production, these tools are frequently combined rather than substituted. A common pattern runs Ray on Kubernetes through the KubeRay operator, uses Spark for upstream warehouse extraction and lets Ray handle the GPU-heavy training and inference stages where fine-grained scheduling pays off.
Limitations and common mistakes with Ray
Ray removes much of the boilerplate of distributed programming, but it does not remove the physics of networks and memory. Most production incidents trace back to a small set of recurring patterns.
- Tasks that are too small: when each call does only microseconds of work, scheduling and serialization overhead dominate; batching items into larger tasks restores efficiency.
- Calling ray.get in a loop: fetching results one by one serializes the program; submitting all tasks first and collecting them together keeps the cluster busy.
- Object store pressure: holding many references prevents eviction, triggers spilling and can end in out-of-memory errors on worker nodes.
- Head node as a single point of failure: without GCS fault tolerance backed by external storage such as Redis, losing the head node stops the cluster.
- Version drift: the driver and all nodes must run matching Ray and Python versions, or serialization fails in confusing ways.
Fault tolerance is real but not automatic for every case. Ray can retry failed tasks and rebuild lost objects from lineage, yet actor state is lost on failure unless the application checkpoints it explicitly or uses library-level checkpointing in Ray Train and Ray Tune.
In simple terms
Ray is a coordinator for computer work. Imagine a large kitchen with many cooks. Instead of one chef doing every step, a head chef hands out jobs, keeps track of who is busy and places finished dishes on a shared counter where anyone can pick them up.
Ray plays that head-chef role for computers. A programmer writes ordinary Python, marks which parts can run separately, and Ray spreads those parts across many processors or machines. It also remembers where every result is stored. The key takeaway is that Ray makes one program act like many without the programmer managing each machine by hand.
A practical example
A machine learning team at an online retailer needs to fine-tune a product-recommendation model every night on about two terabytes of click logs. Previously, a single eight-GPU server took roughly ten hours, which regularly ran into business hours.
The team moves the pipeline to a Ray cluster on Kubernetes with four GPU nodes and several CPU-only nodes. Ray Data streams and preprocesses the logs on the CPU nodes while Ray Train runs data-parallel PyTorch on 32 GPUs, and Ray Tune tests twelve learning-rate and batch-size combinations with early stopping. Training results are checkpointed to object storage after each epoch.
The nightly job now finishes in under three hours, failed spot instances are replaced without restarting the run, and the best configuration is deployed through Ray Serve the next morning.