Skip to content

Tarik Billa

  • Web Development
    • html
    • vue.js
    • laravel
    • css
    • javascript
    • jquery
    • node.js
    • php
    • asp.net
  • Programming
    • python
    • java
    • c
    • c++
    • c#
  • git
  • android

huggingface-tokenizers

Suppress HuggingFace logging warning: “Setting `pad_token_id` to `eos_token_id`:{eos_token_id} for open-end generation.”

April 2, 2024 by Tarik
Categories huggingface-transformers Tags huggingface-tokenizers, huggingface-transformers Leave a comment

Transformers v4.x: Convert slow tokenizer to fast tokenizer

April 2, 2024 by Tarik
Categories python Tags huggingface-tokenizers, huggingface-transformers, nlp, python Leave a comment

ValueError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]] – Tokenizing BERT / Distilbert Error

November 23, 2023 by Tarik
Categories tokenize Tags bert-language-model, distilbert, huggingface-tokenizers, huggingface-transformers, tokenize Leave a comment

How to disable TOKENIZERS_PARALLELISM=(true | false) warning?

November 20, 2023 by Tarik
Categories python Tags huggingface-tokenizers, huggingface-transformers, python, pytorch Leave a comment

Tarik Billa

Software Engineer
tarikbilla@gmail.com
+8801884414000
  • Reuse a hash in YAMLApril 17, 2024
  • Dockerfile: how to redirect the output of a RUN command to a variable?April 16, 2024
  • How to cd to a directory with spaces in the directory name?April 16, 2024
  • Maximum MIME type length when storing the type in a databaseApril 16, 2024
  • What is the difference between Unit, Integration, Regression and Acceptance Testing?April 16, 2024
© 2026 Tarik Billa