What is Tokenization in NLP?
What is Tokenization in NLP?
We have demonstrated the possibility of using ML and NLP technologies to extract information from a textual description (and generate equipment profiles based on it). The generated profiles were compared with the results of manual mapping
Contents
Tokenization
Tokenization is the process of breaking up a text document into individual words called tokens.
As you can see on doctranslator website, the sentence is broken down into words (tokens).
The Natural Language Toolkit (NLTK Library) is a popular open source library package used for all kinds of NLP tasks. In this article, we will use the NLTK library in all stages of text preprocessing.
Removing meaningless words
Stop words are commonly used words that do not add any additional information to the text. Words like "the", "is", "a" have no meaning and only add noise to the data.
The NLTK library has a built-in stop word list that you can use to remove stop words from text. However, this is not a universal stop word list for any business, we can also create our own set of stop words depending on the volume.
The NLTK library has a predefined stopword list. We can add or remove stop words from this list or use it depending on the specific task.