NLP: Text Summarisation with Python

Here's a simple Python method based on the Natural Language Toolkit for extractive text summarisation in natural language processing.

- Dr Dipankar Ray

In natural language processing (NLP), frequency-based summarisation is a straightforward extractive text summarisation technique that selects sentences based on the frequency of important words in the text. The approach is based on the assumption that frequently occurring words represent the core themes of the text. Let's discuss a simplified algorithm using this approach.

Steps in frequency-based summarisation Preprocessing:

• Tokenization: Split the text into sentences and words.

• Stop word removal: Remove common words like 'and', 'the', or 'is' that do not contribute to meaning.

• Stemming: Reduce words to their base forms.

Word frequency calculation:

• Count the occurrences of each word in the text.

• Normalise frequencies if needed, e.g., by dividing by the total number of words.

Sentence scoring:

• Assign scores to sentences based on the cumulative frequency of the words they contain.

• Sentences with more frequent words score higher.

Sentence selection:

• Rank sentences by their scores.

• Select the top n sentences (based on a predefined ratio or word count) to form the summary.

Natural Language Toolkit (NLTK) package-based text processing uses this package with all the required modules. The following modules have been used here.

from nltk.corpus import stopwords

from nltk.tokenize import word_tokenize, sent_tokenize

from nltk.stem import PorterStemmer

Tokenization

In natural language processing, tokenization divides a string into a list of tokens. Tokens are useful when finding valuable patterns; tokenization also replaces sensitive data components with non-sensitive ones.