Unicode Tutorial - Part 6: Visualization of the UTF-16 Encoding Algorithm

---

UTF-16 Encoding Algorithm

---

Below is an interactive visualization of the encoding algorithm that takes as input a Unicode code point in hex notation and converts it to a byte string according to the UTF-16, UTF-16BE, and UTF-16LE encoding schemes.

Step 1:	Enter a code point between 0 and 10FFFF in hex notation
0x

Step 2:	Subtract 0x10000 from the code point, yielding:

0x

Step 3:	The code point must now be smaller than or equal to 0xFFFFF and its binary representation must fit within 20 bits. We prepare for the next step by splitting it into two halves of 10 bits each:

Binary

Step 4:	Insert the 6 bits 110110 in front of the left half and insert the 6 bits 110111 in front of the right half:

Binary

Step 5:	The 16-bit left half becomes the high surrogate and the 16-bit right half becomes the low surrogate. Here is the surrogate pair in hex notation:

Step 6:	Finally, the surrogate pair can be serialized as a sequence of bytes (in hex notation) using any one of the following UTF-16 encoding schemes:

UTF-16BE:
UTF-16LE:
UTF-16:	(UTF-16/no BOM)
	(UTF-16/BE BOM)
	(UTF-16/LE BOM)

Step 2:	Finally, the code unit can be serialized as a sequence of bytes (in hex notation) using any one of the following UTF-16 encoding schemes:

UTF-16BE:
UTF-16LE:
UTF-16:	(UTF-16/no BOM)
	(UTF-16/BE BOM)
	(UTF-16/LE BOM)

While the visualization above works for all code points, it is interesting to try specific code points to better understand the encoding of supplementary characters, that is, those that are outside the Basic Multilingual Plane (BMP).

For example, the smallest supplementary character has code point 0x10000. which reduces to 0x0 after the subtraction in Step 2.  Later in Step 4, inserting the bits 110110 in front of the first half (of all zeros) yields the value 0xD800, which is the smallest possible value for a high surrogate.

Similarly, inserting the bits 110111 in front of the second half (of all zeros) yields the value 0xDC00, which is the smallest possible value for a low surrogate.

Therefore, in UTF-16 encoding, the smallest supplementary character gets assigned the smallest possible (component-wise) surrogate pair.

At the other end of the spectrum, the character with the largest possible code point, namely 0x10FFFF, gets assigned the largest possible (component-wise) surrogate pair, namely:

 < 0xDBFF, 0xDFFF >

More generally, there is a one-to-one mapping between the set of supplementary character code points and the set of all surrogate pairs.

In the next post, we will discuss the inverse mapping, that is, from byte sequences to Unicode character sequences.

---

The algorithm visualized in this post is described in plain text in this RFC.