tree 9de41768671de4d59c51169e215fd01e5b29d8ca
parent f09a5b9fd9678a8adb705740192037a4f3757046
author Ariya Hidayat <ariya.hidayat@gmail.com> 1488150371 -0800
committer Ariya Hidayat <ariya.hidayat@gmail.com> 1488382960 -0800

Pure tokenizer: simplify the choice of obtaining a new token

Fix #1768
Closes gh-1774