Tutorial · 6

Character RNN

No images, no dataset file — a few dozen words typed into the page. A recurrent network reads a word one letter at a time and guesses which language it came from, which is the smallest useful sequence problem there is.

1 · The data, and one-hot letters

A network cannot read a letter; it reads numbers. One-hot encoding gives each letter its own axis, so "no relationship between letters" is stated rather than learned. Real systems use embeddings — the same idea with the axes squeezed down.

2 · A recurrent cell

nn.RNNCell takes one letter and the running hidden state, and returns the next hidden state. Loop it over the word yourself — that loop is the recurrence, and seeing it written out is the point of doing it at this size.

3 · Train it

One word at a time — no batching, because words differ in length and padding them is a complication this size does not need. The loss is noisy for the same reason; the curve still finds its way down.

4 · Ask it about names it has never seen

Made-up words, and a few real ones that were not in the list. It has 45 examples to go on, so it is learning letter patterns — -ini, -mann, short and vowel-light — not names.