# Eli5.map :tokenizer not defined

**URL:** <https://discuss.huggingface.co/t/eli5-map-tokenizer-not-defined/52536>\
**Category:** Beginners\
**Created:** [August 28, 2023, 6:24pm UTC](https://discuss.huggingface.co/t/eli5-map-tokenizer-not-defined/52536 "2023-08-28T18:24:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kobosh](https://avatars.discourse-cdn.com/v4/letter/k/58f4c7/32.png) [@kobosh](https://discuss.huggingface.co/u/kobosh)\
**Post date:** [August 28, 2023, 6:24pm UTC](https://discuss.huggingface.co/t/eli5-map-tokenizer-not-defined/52536/1 "2023-08-28T18:24:08Z")

</div>

I am trying this [tutotial](https://huggingface.co/docs/transformers/main/tasks/masked_language_modeling).  
preprocess function runs fine when I test it with portion of eli5[train]. However it runs I think out of memory when provided whole dataset. Now when I run eli5.map I get name tokenizer is not defined. Could it be memory issue? I modified the preprocess function to return only 10 items from example like this: return tokenizer([" ".join(x) for x in examples[“answers.text”][0:10]]) but still get the same error

---

<div class="post-metadata">

**Author:** ![kobosh](https://avatars.discourse-cdn.com/v4/letter/k/58f4c7/32.png) [@kobosh](https://discuss.huggingface.co/u/kobosh)\
**Post date:** [August 28, 2023, 7:22pm UTC](https://discuss.huggingface.co/t/eli5-map-tokenizer-not-defined/52536/2 "2023-08-28T19:22:39Z")

</div>

found solution [here](https://discuss.huggingface.co/t/tokenizer-is-not-defined/39231)