S3-compatible storage

On this page 7

S3-compatible storage is any object storage service or software that implements the Amazon S3 API, so that applications, SDKs and command-line tools written for S3 can read and write data on it by changing only the endpoint and credentials. Cloud storage vendors, managed hosting providers and on-premises platforms adopt this interface because the S3 API has become the de facto standard for object access over HTTP.

Compatibility is a spectrum rather than a binary property. A system may support the core object operations perfectly while omitting versioning, lifecycle rules or fine-grained policies, which is why practitioners treat the label as a starting point for testing rather than a guarantee.

How S3 compatibility works

An S3-compatible system works by accepting the same HTTP requests and returning the same responses as Amazon S3. Clients send REST calls such as PUT, GET, HEAD, DELETE and LIST against a bucket and object key, and the server answers with S3-formatted XML bodies, headers and error codes.

Authentication relies on Signature Version 4 (SigV4), an HMAC-based scheme in which the client signs each request with a secret key, the region name and a timestamp. A compatible server must reproduce the signature calculation exactly, including canonical header ordering and payload hashing, or every request fails with a signature mismatch.

Two addressing styles exist. Virtual-hosted style places the bucket in the hostname (bucket.endpoint.example), while path style places it in the URL path (endpoint.example/bucket). Many self-hosted systems default to path style because it does not require wildcard DNS and wildcard TLS certificates, and clients usually expose a setting to force one or the other.

The table below lists the API areas that determine how compatible an implementation really is.

API areaWhat it coversTypical support level
Core object operationsPut, get, head, delete, list objectsNearly universal
Multipart uploadParallel parts for large objectsCommon
Presigned URLsTime-limited signed linksCommon
Object versioningMultiple versions per keyFrequent, sometimes partial
Lifecycle rulesExpiration and tier transitionsVariable
Bucket policies and ACLsAccess control documentsVariable, often simplified
Object LockWORM retention and legal holdLimited
Event notificationsWebhooks or queues on object changesVendor-specific
API areas that define the depth of compatibility

Types of S3-compatible storage

S3-compatible storage comes in three broad deployment models: public cloud services from providers other than AWS, self-hosted software that turns servers into an object store, and hardware appliances sold with an S3 front end. Each model shifts responsibility for durability, scaling and operations differently.

Self-hosted platforms commonly use erasure coding to spread data and parity fragments across disks and nodes. A layout of 8 data and 4 parity fragments, for example, survives the loss of any 4 fragments while consuming 1.5 times the raw capacity, compared with 3 times for triple replication.

Deployment modelWho operates itMain advantageMain burden
Alternative cloud providerThe providerLower or no egress feesFeature gaps versus AWS
Self-hosted softwareYour teamFull control and data localityHardware, upgrades, on-call
Storage applianceYour team with vendor supportIntegrated hardware and supportCapital cost and vendor lock-in
Gateway or proxy layerYour teamS3 interface over existing storageAdded latency and translation limits
Deployment models and their trade-offs

A gateway layer deserves special mention: it translates S3 calls into another backend, such as a file system or a different cloud API. This approach is useful for migration but tends to lose semantics that the backend lacks, such as atomic versioning.

S3-compatible storage is a form of object storage defined by its interface, not by its internal architecture. It differs from native cloud object services that use their own APIs and from file or block storage that present entirely different access models.

Storage typeAccess methodRelationship to the S3 API
Amazon S3S3 REST APIThe reference implementation
S3-compatible storageS3 REST APIReimplementation of the interface
Azure Blob StorageAzure Blob REST APIDifferent native API
Google Cloud StorageJSON API and XML APIXML API is largely S3-interoperable
Cloud file storageNFS or SMBHierarchical files, not objects
How S3-compatible storage relates to neighboring concepts

The distinction matters for portability. Code written against the S3 API can often move between compatible providers without rewriting, whereas code using the Azure or Google native SDKs requires an abstraction layer or a rewrite to change platforms.

Choosing and configuring an S3-compatible service

Selecting an S3-compatible service starts with mapping the exact API calls your workloads use and verifying each one against the target system. Backup software, data lakes and media pipelines often rely on different subsets, so a generic compatibility claim is not sufficient.

WorkloadRecommended choiceReason
Backup with immutabilityService with verified Object LockRansomware protection depends on WORM retention
Public content deliveryProvider with low egress costDownload traffic dominates the bill
Analytics data lakePlatform with fast listing and range readsQuery engines issue many small requests
Edge or air-gapped siteSelf-hosted softwareNo dependency on external connectivity
Development and CI testsLightweight local emulatorFast, disposable, no cloud charges
Decision guide for common workloads

On the client side, configuration usually involves four settings: the endpoint URL, the region string, the access key pair and the addressing style. Some SDKs also send integrity checksums by default in newer versions, and older compatible servers may reject those headers until the client is told to disable them.

Limitations and common mistakes

The most frequent problem with S3-compatible storage is assuming identical behavior where only syntax matches. Two systems can accept the same request yet differ in consistency guarantees, listing order, maximum object size or error handling.

  • Consistency differences: Amazon S3 offers strong read-after-write consistency, while some implementations provide only eventual consistency for listings.
  • Silent feature gaps: an unsupported lifecycle rule may be accepted and stored but never executed, leaving data that was meant to expire.
  • Multipart limits: part counts, minimum part sizes and maximum object sizes may differ from the 10,000-part and 5 MiB conventions.
  • Storage class mapping: class names such as GLACIER may be accepted but mapped to a single tier, which changes cost and retrieval behavior.
  • Signature edge cases: special characters in keys can break SigV4 canonicalization on less mature servers.

The safest practice is a compatibility test suite run before migration, covering every operation the application performs, plus a restore test that proves data can actually be read back at the required speed.

Explained without jargon

Think of the S3 API as a standard plug shape for storing files online. Amazon designed the plug, and it became so popular that many other companies built sockets that accept it. S3-compatible storage is any of those other sockets.

The benefit is freedom of choice: software built for Amazon can often switch to another provider by changing an address and a password. The catch is that not every socket supports every feature, so it is wise to test the features you rely on before moving important data.

A typical use case

A regional hospital network keeps nightly backups of about 40 TB and must hold them for seven years. Paying public cloud egress fees for restore tests was too expensive, so the infrastructure team deployed a self-hosted S3-compatible cluster of six servers using erasure coding with 4 data and 2 parity fragments in its own data center.

The team pointed its existing backup software at the new endpoint, enabled path-style addressing and turned on Object Lock in compliance mode with a 30-day immutability window. Before go-live, engineers ran a test suite that revealed one lifecycle rule was silently ignored, and they replaced it with a scheduled cleanup job.

The result was predictable costs, restore tests without egress charges and backups that ransomware could not delete during the retention window.