UTF-8

On this page 7

UTF-8 is a variable-length character encoding that represents every Unicode code point as a sequence of one to four bytes, while keeping the first 128 values identical to ASCII. In software interoperability, it is the de facto common language for text: files, databases, web pages, APIs and operating systems that agree on UTF-8 can exchange text in any script without corruption or lossy conversion.

The encoding was designed in 1992 by Ken Thompson and Rob Pike and is specified in RFC 3629 and the Unicode Standard. Today it carries the overwhelming majority of web content and is the mandated encoding for formats such as JSON, which makes it the safe default whenever two systems must read the same text.

How UTF-8 encodes Unicode code points

UTF-8 converts each Unicode code point into a byte sequence whose length depends on the size of the number. The leading bits of the first byte announce how many bytes follow, and every continuation byte begins with the bit pattern 10, so a decoder always knows where a character starts and ends.

The following table shows the four byte patterns, the code point ranges they cover and typical characters in each range.

BytesCode point rangeBit patternTypical characters
1 byteU+0000 to U+007F0xxxxxxxASCII letters, digits, punctuation
2 bytesU+0080 to U+07FF110xxxxx 10xxxxxxLatin accents, Greek, Cyrillic, Arabic, Hebrew
3 bytesU+0800 to U+FFFF1110xxxx 10xxxxxx 10xxxxxxChinese, Japanese, Korean, Indic scripts
4 bytesU+10000 to U+10FFFF11110xxx plus three continuation bytesEmoji, historic scripts, rare ideographs
UTF-8 byte patterns by code point range

This design gives UTF-8 several properties that matter for interoperability. It is self-synchronizing, because a reader dropped into the middle of a stream can find the next character boundary by skipping continuation bytes. It contains no zero bytes except for the actual NUL character, so C-style string functions keep working. Byte-wise sorting of UTF-8 strings also matches code point order, which simplifies indexing.

Why UTF-8 became the interoperability default

UTF-8 won because it solved the legacy code page problem without breaking existing ASCII software. Before its adoption, each region used its own encoding, such as ISO-8859-1, Windows-1252, Shift_JIS or KOI8-R, and a file moved between systems often turned into unreadable symbols. A single universal encoding removed the need to guess or negotiate the encoding for every exchange.

Standards bodies reinforced this choice. RFC 8259 requires UTF-8 for JSON exchanged between systems, the WHATWG Encoding Standard treats it as the default for the web, HTML5 recommends it for all new documents, and modern programming languages such as Go and Rust use it as their native string representation. Most Linux distributions and macOS use UTF-8 locales by default, and Windows has added a UTF-8 code page option for system-wide use.

The table below summarizes where UTF-8 is required, recommended or merely common across typical software layers.

LayerUTF-8 statusPractical consequence
JSON and REST APIsRequired for open exchangeOther encodings break parsers
HTML and the webDefault and recommendedDeclare it in the meta charset tag
XMLDefault when no declaration existsDocuments without a prolog are read as UTF-8
DatabasesConfigurable per server or columnChoose utf8mb4 in MySQL for full coverage
Operating systemsDefault on Linux and macOSWindows tools may still assume a code page
UTF-8 status across common software layers

UTF-8 compared with other character encodings

UTF-8 is one of several Unicode encoding forms, and it competes with older single-byte encodings in legacy systems. Each choice trades storage size, processing simplicity and compatibility differently, so the right comparison depends on the text and the environment.

The comparison below covers the encodings that software engineers most often meet during format conversion and data import.

EncodingUnit sizeUnicode coverageASCII compatibleMain weakness
UTF-81 to 4 bytesFullYesLarger for East Asian text than UTF-16
UTF-162 or 4 bytesFullNoByte order issues and surrogate pairs
UTF-324 bytes fixedFullNoUses four times the space of ASCII
ISO-8859-11 byteWestern European onlyYesCannot represent most scripts
Windows-12521 byteWestern European onlyYesOften mislabeled as ISO-8859-1
Comparison of common character encodings

UTF-16 remains the internal string format of Windows APIs, Java and JavaScript, so many systems convert at their boundaries. A common rule is to keep UTF-16 or another representation inside the process but to use UTF-8 on the wire and on disk, where compatibility counts more than in-memory convenience.

Configuring and detecting UTF-8 in practice

Correct UTF-8 handling depends on declaring the encoding explicitly at every boundary rather than relying on platform defaults. Software reads bytes, and those bytes become text only when the reader applies the same encoding the writer used.

  • Send the HTTP header Content-Type: text/html; charset=utf-8 and repeat the declaration inside the document.
  • Open files with an explicit encoding argument, for example encoding="utf-8" in Python, instead of the locale default.
  • Configure databases, connection strings and client libraries with the same UTF-8 setting end to end.
  • Validate incoming data and reject or replace invalid sequences at the system boundary.

The byte order mark (BOM), the sequence EF BB BF, is optional in UTF-8 and generally discouraged. Some Windows tools add it to signal the encoding, yet Unix shells, CSV importers and JSON parsers may treat it as unexpected content, which produces stray characters in the first field or failed script execution.

Common UTF-8 errors and how to fix them

Most UTF-8 problems arise from an encoding mismatch between writer and reader, not from the encoding itself. The visible result, often called mojibake, is garbled text such as “café” instead of “café”.

The following table maps frequent symptoms to their causes and corrective actions.

SymptomLikely causeFix
Accented letters become two symbolsUTF-8 bytes read as Windows-1252Declare and read the file as UTF-8
Question marks replace charactersLossy conversion to a single-byte encodingKeep UTF-8 through the whole pipeline
Emoji fail to save in MySQLLegacy utf8 charset limited to 3 bytesMigrate columns to utf8mb4
Replacement character U+FFFD appearsInvalid or truncated byte sequencesFix the source and validate input
Strange characters at file startUnexpected byte order markSave the file as UTF-8 without BOM
UTF-8 symptoms, causes and fixes

Developers should also remember that byte length differs from character length. Truncating a UTF-8 string at a fixed byte count can cut a multibyte character in half, and a database column sized in bytes may reject text that looks short on screen. Security filters must operate on validated, normalized text, because overlong or malformed sequences have historically been used to bypass naive checks.

The simple explanation

Computers store letters as numbers, and an encoding is the rulebook that says which number means which letter. UTF-8 is the one rulebook almost everyone now shares. It can write every letter, symbol and emoji in the world, from English and Greek to Chinese and Hindi.

Simple English letters take only one byte, while more complex characters take two, three or four. When two programs both use UTF-8, text moves between them unchanged. When they use different rulebooks, words turn into nonsense symbols. The key takeaway is simple: choose UTF-8 and say so clearly.

How it is used in practice

A mid-sized online retailer migrates its product catalog from an old inventory system to a new e-commerce platform. The legacy system exports CSV files in Windows-1252, and the first test import shows product names such as “Müller Kaffeemaschine” and missing Polish and Czech letters. Customer reviews containing emoji fail to save entirely.

The integration team converts the export to UTF-8 without a BOM, sets the import tool to read UTF-8 explicitly, and migrates the MySQL tables from the legacy utf8 charset to utf8mb4. They add a validation step that rejects rows containing invalid byte sequences and logs them for review.

After the change, the import of about 48,000 product records completes with no garbled names, and reviews with emoji display correctly in all storefront languages. The new platform’s JSON API now exchanges text with the warehouse and marketplace partners without any per-partner encoding rules.