What is the difference between Unicode and UTF-8?

Code points first, bytes second

Unicode defines a code space from U+0000 to U+10FFFF, room for 1,114,112 code points split into 17 planes of 65,536 each. More than 150,000 of them are assigned to characters, covering modern and historic scripts, symbols and emoji. A code point is written as U+ followed by at least four hex digits.

A code point is only a number. To save or transmit text, it has to be turned into bytes, and there are several ways to do that: UTF-8, UTF-16 and UTF-32 are all encodings of the same Unicode characters.

How UTF-8 picks 1, 2, 3 or 4 bytes

UTF-8, specified in RFC 3629, is a variable-width encoding. The leading bits of the first byte say how many bytes follow, and every continuation byte starts with the bits 10. That makes the stream self-synchronizing: a decoder that lands in the middle of a character can skip forward to the next start byte.

Worked example: "é" is U+00E9, binary 000 1110 1001. It needs the 2-byte pattern. The top 5 bits 00011 go into 110xxxxx to give 11000011 (C3), and the low 6 bits 101001 go into 10xxxxxx to give 10101001 (A9).

Because the first 128 code points encode to the same single bytes as ASCII, any valid ASCII file is already valid UTF-8. That backward compatibility is the main reason UTF-8 became the default for the web, where it is used by about 98% of websites.

UTF-8 vs UTF-16 vs UTF-32

UTF-16 stores characters above U+FFFF as a surrogate pair of two 16-bit units. That is why "😀".length is 2 in JavaScript: the language counts UTF-16 units, not characters. Iterating with [..."😀"].length returns 1.

For English text UTF-8 is the most compact. For text that is mostly Chinese, Japanese or Korean, UTF-16 can be smaller, since those characters take 3 bytes in UTF-8 but 2 in UTF-16.

The byte order mark (BOM)

A BOM is the character U+FEFF placed at the very start of a file. In UTF-16 and UTF-32 it tells the reader whether the bytes are big-endian or little-endian. In UTF-8 it encodes as EF BB BF and serves only as a signature, since UTF-8 has no byte order.

The Unicode FAQ says a UTF-8 BOM is neither required nor recommended. Some Windows editors add it anyway, and it then breaks shell scripts (the #! line is no longer first), CSV headers and JSON parsers that do not expect it. If a file shows  at the top, that is a UTF-8 BOM read as Windows-1252.

Why text turns into é or ���

Garbled text, called mojibake, appears when bytes are decoded with a different encoding than the one used to write them:

• é instead of é: UTF-8 bytes C3 A9 read as Windows-1252 or Latin-1, which maps C3 to Ã and A9 to ©.
• � (U+FFFD, the replacement character): the decoder hit bytes that are not valid UTF-8, often Latin-1 text read as UTF-8.
• Question marks in a database: the column or connection used a charset that cannot hold the character, so it was replaced on write and is gone.

The fix is to declare the encoding at every boundary: <meta charset="utf-8"> in HTML, Content-Type: text/html; charset=utf-8 in HTTP headers, and UTF-8 for database columns and connections. In MySQL, use utf8mb4; the older utf8 charset stores at most 3 bytes per character and rejects emoji.