fix perplexity after c-api refactor (#390)
* preallocate a buffer of fitting size for tokenization (utils.cpp) * don't create a new std::string (especially here, where it's usually large)pull/403/head master-56e659a
parent
40ea807a97
commit
56e659a0b2
Loading…
Reference in New Issue