7 terms
An AI gateway is a proxy layer between applications and AI model providers that centralizes routing, access control, cost tracking, caching and failover.
Distributed training splits the training of a machine learning model across many GPUs or nodes so that larger models and datasets can be trained faster.
GPU inference is the execution of trained AI models on graphics processors to generate predictions or text quickly, at scale and at acceptable cost.
An inference server is software that loads trained AI models and answers prediction requests over a network, managing batching, hardware and scaling.
Model routing sends each AI request to the most suitable model, balancing answer quality, latency and cost across a pool of language models in production.
Model serving is the practice of running trained machine learning models in production so applications can request predictions reliably, quickly and at scale.
Prompt caching reuses the computed attention state of a repeated prompt prefix, cutting latency and input cost when serving large language models.