Large Language Models Are Not One-Size-Fits-All for NLP
Using a jackhammer to crack a nut... That's kind of what we're doing when we use a large language model for a basic NLP task... What about good old TF-IDF?
We've gotten into the habit of using large language models for all sorts of tasks. We have to admit, they've become easy to access via APIs, interfaces, cloud provider services or even locally. But this ease of use shouldn't lead to laziness, especially when it comes to being more efficient and less demanding on computing resources.
Let's consider the use case where we need to select keywords from a text. For example, to classify a document or to automatically suggest tags to a user for SEO.
There is a proven NLP method that allows us to do this fairly easily. It's called TF-IDF.
Suppose we have a corpus of documents. The statistical TF-IDF method assumes that important words are probably not the ones that are abundant across all documents, because those are common.
On the flip side, words that are frequent in only a few documents are likely the most important.
For a given word, in a given document, the score assigned is:
The frequency of the word's appearance in the document (TF - Term Frequency).
Weighted by the ratio of the total number of documents over the number of documents where the term appears (IDF - Inverse Document Frequency).
I suggest a little code example on Github using the first paragraph of the iPhone Wikipedia page. I "cheat" a bit by using each sentence as a document, clean up the words a bit, do some lemmatization, and apply tf-idf using bigrams (consecutive word pairs) as well.
The result isn't perfect but this low-cost method can largely suffice in some cases. Plus, it can be refined later with more advanced techniques.
Of course tf-idf has its limitations, it's a purely statistical method and thus doesn't take semantics into account. It requires a certain number of documents and words for the result to be relevant, etc... But NLP shouldn't be reduced to just large language models!