Text similarity

Text

Similarity

LLM

Introduction

Text similarity is essential to data science, powering duplicate detection, content clustering, and search systems. The challenge is identifying when different text strings convey the same meaning.

Raw text often contains inconsistent capitalization, special characters, and formatting. Python’s re module provides an efficient way to normalize these variations.

Regular expressions identify and replace unwanted patterns, including symbols, whitespace, and formatting inconsistencies, creating cleaner text for downstream analysis.

Dataset

Load dataset

we use simple generated data as follow:

df = pd.DataFrame([
    "iPhone® 14 Pro Max",
    "IPHONE 14 pro max",
    "Apple iPhone 14 Pro Max 256GB",
    "iPhone14ProMax",
    "i-Phone 14 Pro Max",
    "Samsung Galaxy S23 Ultra",
    "SAMSUNG Galaxy S23 Ultra 5G",
    "Galaxy S23 Ultra (512GB)",
    "Samsung S23 Ultra",
    "wireless headphones",
    "bluetooth earbuds",
    "Sony WH-1000XM4 Headphones",
    "WH-1000XM4 Wireless Headphones",
], columns=['product'])

before we use the data above, we could transform the text into well defined format such as: lower case, spacing, and remove non-words.

df['cleaned_text'] = df.map(text.text_manipulation, types=['lower_case','non_word','spacing'])

This give as:

• Removing special characters and symbols
• Standardizing case and formatting
• Cleaning messy product names
• Preparing text for similarity analysis

Function above give us well defined format text like below:

With our test data established, we can build a comprehensive preprocessing function to handle these variations.

Main Function

similarity_text(main_data, col_name, types='cosine')

This function requires the following parameters:

• main_data (dataframe): Data input
• col_name (string): column name
• types (string): type of text similarity that we want to use

difflib: Python’s Built-in Sequence Matching

difflib is a Python built-in module that provides similarity ratios. It analyzes character sequences to calculate similarity scores between text strings.

text.similarity_text(df[['cleaned_text']], col_name='cleaned_text', types='difflib')

The limitation for this module:

• Sensitivity to word order.
• Poor handling of extra content.

Use difflib when:

• Character-Level Similarity
• Learning text similarity concepts
• Working with small datasets (<1000 records)
• External dependencies not allowed
• Simple typo detection is sufficient

RapidFuzz: High-Performance Fuzzy String Matching

RapidFuzz is a high-performance fuzzy string matching library with C++ optimization. It addresses word reordering and complex text variations that difflib cannot handle effectively.

text.similarity_text(df[['cleaned_text']], col_name='cleaned_text', types='fuzz')

The limitation for this module:

• It excels at text-level variations but cannot understand meaning. When products serve identical purposes using different terminology, we need semantic understanding rather than pattern matching.

Use RapidFuzz when:

• Production Fuzzy Matching
• Processing thousands of records
• Need fast approximate matching
• Handling abbreviations and variations
• Text-level similarity is sufficient

Sentence Transformers: AI-Powered Semantic Similarity

Surface-level text matching misses semantic relationships. Sentence Transformers, a library built on transformer neural networks, can understand that “wireless headphones” and “bluetooth earbuds” serve identical purposes by analyzing meaning rather than just character patterns.

text.similarity_text(df[['cleaned_text']], col_name='cleaned_text', types='transformers')

Sentence Transformers unlocks semantic understanding at computational cost. The decision depends on whether conceptual relationships provide sufficient business value to justify resource overhead.

Use Sentence Transformers when:

• Semantic Understanding
• Conceptual relationships matter
• “wireless headphones” should match “bluetooth earbuds”
• Building recommendation systems
• Multilingual content similarity
• Compute resources are available

JF jaro winkler similarity

text.similarity_text(df.head()[['cleaned_text']], col_name='cleaned_text', types='jf_winkler')

Final Thoughts

When facing complex challenges, start with the most basic solution first, identify where it fails through testing, then strategically upgrade the failing component.

Build complexity incrementally based on real limitations, not anticipated ones.