New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

不支持指定special_token吗？ #6

Open

zipzou opened this issue Oct 15, 2023 · 1 comment

zipzou commented Oct 15, 2023 •

edited by bojone

Loading

比如将\n处理为<n>

Owner

bojone commented Oct 17, 2023

个人认为这不是一个tokenizer该做的事情，tokenizer的作用就是对原始字节串进行切分，这种特殊需求可以自行包装一层。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment