Building a Recommender System for Mental Health Resources with Scikit-learn's TF-IDF

In an era of information overload, finding the right mental health resource when you need it most can be a challenge. Imagine feeling overwhelmed and typing "I'm feeling stressed and anxious about my work" into a wellness app. Instead of a generic list, the app intelligently suggests a 5-minute breathing exercise for anxiety, an article on managing workplace stress, and a guided meditation for focus. This is the power of a recommender system.

In this tutorial, we'll build a simple yet effective content-based recommender system using Python and Scikit-learn. We will create a system that takes a user's journal entry or mood description and recommends relevant mental health resources. This project is a fantastic introduction to the world of Natural Language Processing (NLP) and recommender systems, showcasing how we can use text data to provide personalized and helpful suggestions.

What we'll build/learn:

• How to preprocess text data for machine learning.
• The intuition behind Term Frequency-Inverse Document Frequency (TF-IDF).
• How to use TfidfVectorizer from Scikit-learn to transform text into feature vectors.
• The concept of cosine similarity and how to use it to find similar items.
• How to tie everything together to build a functional recommender system.

Prerequisites:

• Basic understanding of Python.
• Familiarity with pandas for data manipulation.
• Python 3.x installed on your system.
• Jupyter Notebook or your favorite IDE.

Why this matters to developers: Building recommender systems is a highly sought-after skill. This project not only teaches you the fundamentals of content-based filtering but also demonstrates how you can apply these skills to the rapidly growing field of health tech, creating applications that can make a real difference.

The core challenge is to bridge the gap between a user's unstructured text input (how they are feeling) and a structured database of mental health resources. How can a machine understand the meaning or intent behind "I can't focus on my project" and connect it to a resource about "mindfulness for productivity"?

This is where content-based filtering comes in. Instead of relying on user ratings (like in collaborative filtering), we will analyze the content of the resources themselves. By creating a "profile" for each resource based on its description, we can then compare a user's input to these profiles and find the best match.

The main tools we'll use are:

• TF-IDF (Term Frequency-Inverse Document Frequency): A statistical measure that evaluates how relevant a word is to a document in a collection of documents. This helps us identify the keywords that best describe each resource.
• Cosine Similarity: A metric used to measure the similarity between two non-zero vectors. In our case, it will measure the similarity between the user's input vector and each resource's vector. A higher score means a better match.

Before we start coding, let's set up our environment and create the dataset.

Required Libraries:

• pandas: For creating and managing our dataset.
• scikit-learn: For TF-IDF vectorization and calculating cosine similarity.

You can install these libraries using pip:

pip install pandas scikit-learn

Creating Our Dataset:

For this tutorial, we'll create a small, representative dataset of mental health resources. In a real-world application, this would come from a database. Let's create a file named mental_health_resources.csv with the following content:

id,type,title,description
1,Meditation,"5-Minute Mindfulness Meditation","A short guided meditation to bring awareness to the present moment. Ideal for beginners looking to reduce stress and improve focus. Helps calm an anxious mind and find peace in your day. mindfulness, stress, anxiety, focus"
2,Breathing Exercise,"Box Breathing for Anxiety","A simple and powerful breathing technique to calm your nervous system. Inhale for 4 counts, hold for 4, exhale for 4, and hold for 4. Excellent for immediate anxiety and stress relief. anxiety, stress, calm, breathing"
3,Article,"How to Manage Workplace Stress","Practical tips and strategies for dealing with stress at work. Learn to set boundaries, manage your workload, and improve your work-life balance. stress, work, productivity, balance"
4,Meditation,"Guided Meditation for Sleep","A calming meditation to help you fall asleep faster and have a more restful night. Let go of the day's worries and drift into a deep sleep. sleep, anxiety, relaxation"
5,Article,"Understanding Cognitive Distortions","Learn about common negative thought patterns that contribute to anxiety and depression. This article helps you identify and challenge these distortions. anxiety, depression, cognitive behavioral therapy"
6,Breathing Exercise,"4-7-8 Breathing for Relaxation","A breathing exercise developed by Dr. Andrew Weil. It is a natural tranquilizer for the nervous system. Inhale for 4, hold for 7, and exhale for 8 seconds. Promotes deep relaxation and can help with sleep. sleep, relaxation, breathing, calm"
7,Article,"The Benefits of Journaling for Mental Health","Discover how writing down your thoughts and feelings can improve your mental well-being. A great tool for processing emotions and reducing stress. journaling, stress, self-reflection"
8,Meditation,"Meditation for Boosting Focus and Productivity","A guided session to help you sharpen your concentration and be more effective in your work. Clear mental clutter and enhance your cognitive performance. focus, productivity, work, mindfulness"

This CSV file contains a unique id, the type of resource, a title, and a description. The descriptions include keywords that our TF-IDF model will pick up on.

First, let's load our data into a pandas DataFrame and get it ready for our recommender.

What we're doing

We'll load the mental_health_resources.csv file and inspect its contents to ensure everything is in order.

Implementation

Create a new Python script or Jupyter Notebook and add the following code:

# src/recommender.py
import pandas as pd

# Load the dataset
try:
    df = pd.read_csv('mental_health_resources.csv')
    print("Dataset loaded successfully!")
    print(df.head())
except FileNotFoundError:
    print("Error: 'mental_health_resources.csv' not found. Please make sure the file is in the correct directory.")
    exit()

# For our content-based recommender, we will focus on the 'description'
print("\nResource Descriptions:")
print(df['description'])

How it works

This code snippet uses pandas to read our CSV file into a DataFrame, which is a tabular data structure. We then print the first few rows (df.head()) to verify it's loaded correctly. The most important column for our recommender is the description, as it contains the text we will analyze.

Now for the core of our recommender: converting the text descriptions into numerical vectors that our machine learning model can understand.

What we're doing

We will use TfidfVectorizer from Scikit-learn to transform the text in the description column into a TF-IDF matrix. This matrix will represent the importance of each word in each resource's description.

Implementation

# src/recommender.py (continued)
from sklearn.feature_extraction.text import TfidfVectorizer

# Initialize the TfidfVectorizer
# stop_words='english' removes common English words like 'the', 'a', 'in'
# that don't add much meaning.
tfidf_vectorizer = TfidfVectorizer(stop_words='english')

# Fit and transform the description text
# This learns the vocabulary and inverse document frequency from our descriptions
# and then transforms them into a TF-IDF matrix.
tfidf_matrix = tfidf_vectorizer.fit_transform(df['description'])

# Let's look at the shape of our TF-IDF matrix
# It should have 8 rows (for our 8 resources) and a certain number of columns
# (representing the unique words in our vocabulary).
print("\nShape of TF-IDF Matrix:")
print(tfidf_matrix.shape)

How it works

1. Initialization: We create an instance of TfidfVectorizer. stop_words='english' is a helpful parameter that filters out common English words that don't carry much semantic weight (e.g., "and", "the", "is").
2. fit_transform: This two-step process first learns the vocabulary from all the descriptions (fit) and then converts each description into a vector where each element is the TF-IDF score for a word in the vocabulary (transform).
3. The tfidf_matrix: This is a sparse matrix where each row corresponds to a resource and each column corresponds to a unique word in our dataset's vocabulary. The value at matrix[i, j] is the TF-IDF score of the j-th word in the i-th resource.

With our resources represented as vectors, we can now find the similarity between a user's input and each resource.

What we're doing

We will use cosine_similarity from Scikit-learn to calculate the similarity between vectors. We'll create a function that takes a user's query, transforms it using our fitted TfidfVectorizer, and then computes the cosine similarity between the query vector and all the resource vectors in our tfidf_matrix.

Implementation

# src/recommender.py (continued)
from sklearn.metrics.pairwise import cosine_similarity

def get_recommendations(user_input, top_n=3):
    """
    Generates recommendations based on user input.
    """
    # 1. Transform the user's input using the same TF-IDF vectorizer
    user_tfidf = tfidf_vectorizer.transform([user_input])

    # 2. Calculate the cosine similarity between the user's input and all resources
    # This will result in a matrix of shape (1, 8), where each value is the similarity
    # score between the user input and a resource.
    cosine_similarities = cosine_similarity(user_tfidf, tfidf_matrix).flatten()

    # 3. Get the indices of the most similar resources
    # We use argsort to get the indices that would sort the array, then reverse it
    # and take the top_n.
    related_docs_indices = cosine_similarities.argsort()[::-1][:top_n]

    # 4. Return the recommended resources
    recommendations = df.iloc[related_docs_indices]
    
    return recommendations

# --- Let's test it! ---
user_query = "I'm feeling anxious about work and can't sleep"
print(f"\nUser Query: '{user_query}'")
recommended_resources = get_recommendations(user_query)

print("\nRecommended Resources:")
print(recommended_resources[['title', 'type', 'description']])

How it works

1. Transform User Input: We use tfidf_vectorizer.transform() (note: not fit_transform) to convert the user's query into a TF-IDF vector. We use transform because we want to use the vocabulary and IDF weights learned from our existing dataset, not create new ones.
2. Compute Similarity: cosine_similarity takes the user's vector and our resource matrix and calculates the similarity scores. We use .flatten() to convert the resulting matrix into a simple array of scores.
3. Get Top Recommendations: argsort() gives us the indices of the similarity scores in ascending order. We reverse this with [::-1] to get descending order and then select the top n indices.
4. Return Results: We use df.iloc to select the rows from our original DataFrame that correspond to the recommended indices.

Expected Output:

User Query: 'I'm feeling anxious about work and can't sleep'

Recommended Resources:
                                        title               type                                        description
2               How to Manage Workplace Stress            Article  Practical tips and strategies for dealing with...
3                  Guided Meditation for Sleep         Meditation  A calming meditation to help you fall asleep f...
1                 Box Breathing for Anxiety  Breathing Exercise  A simple and powerful breathing technique to c...
```As you can see, the recommender correctly identified keywords like "anxious", "work", and "sleep" and suggested the most relevant resources!

## Putting It All Together

Here is the complete, runnable script:

```python
# src/mental_health_recommender.py

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

def load_data(filepath):
    """Loads the mental health resources dataset."""
    try:
        df = pd.read_csv(filepath)
        print("Dataset loaded successfully!")
        return df
    except FileNotFoundError:
        print(f"Error: '{filepath}' not found. Please check the file path.")
        return None

def create_recommender_components(df):
    """Creates the TF-IDF vectorizer and matrix."""
    tfidf_vectorizer = TfidfVectorizer(stop_words='english')
    tfidf_matrix = tfidf_vectorizer.fit_transform(df['description'])
    return tfidf_vectorizer, tfidf_matrix

def get_recommendations(user_input, df, tfidf_vectorizer, tfidf_matrix, top_n=3):
    """Generates and returns top N recommendations."""
    if df is None:
        return "Dataset not loaded. Cannot provide recommendations."

    user_tfidf = tfidf_vectorizer.transform([user_input])
    cosine_similarities = cosine_similarity(user_tfidf, tfidf_matrix).flatten()
    related_docs_indices = cosine_similarities.argsort()[::-1][:top_n]
    
    recommendations = df.iloc[related_docs_indices]
    return recommendations

if __name__ == '__main__':
    # Define the path to your dataset
    DATA_PATH = 'mental_health_resources.csv'
    
    # Load data and create recommender components
    resources_df = load_data(DATA_PATH)
    
    if resources_df is not None:
        vectorizer, matrix = create_recommender_components(resources_df)

        # Example usage of the recommender system
        user_query_1 = "I am feeling stressed and overwhelmed by my job"
        print(f"\n--- Recommendations for: '{user_query_1}' ---")
        recommendations_1 = get_recommendations(user_query_1, resources_df, vectorizer, matrix)
        print(recommendations_1[['title', 'type']])

        print("-" * 50)

        user_query_2 = "I want to be more mindful and improve my focus"
        print(f"\n--- Recommendations for: '{user_query_2}' ---")
        recommendations_2 = get_recommendations(user_query_2, resources_df, vectorizer, matrix)
        print(recommendations_2[['title', 'type']])

Even for a simple recommender, if you were to deploy this, security and privacy are paramount, especially in health tech.

• Data Privacy: User inputs could be sensitive. Ensure that any stored journal entries or queries are anonymized and encrypted. Never log personally identifiable information (PII) alongside sensitive health data.
• Input Sanitization: Sanitize all user inputs to prevent injection attacks if this system were to interact with a database or be exposed via a web API.
• Model Bias: Be aware that the recommendations are only as good as the data they are trained on. A limited set of resources could lead to biased or repetitive recommendations. Ensure your resource database is diverse and inclusive.

While TF-IDF and cosine similarity are excellent for getting started, other techniques exist:

• Word Embeddings (like Word2Vec, GloVe): These models capture the semantic meaning of words. "Stressed" and "anxious" would be represented by similar vectors, even if the words themselves are different. This can lead to more nuanced recommendations.
• Hybrid Filtering: A more advanced approach combines content-based methods with collaborative filtering (using other users' behavior) to provide more robust recommendations.
• BERT and Transformers: State-of-the-art language models that understand context deeply and can provide even more accurate similarity matching.

Congratulations! ✨ You've successfully built a content-based recommender system for mental health resources. We've taken raw text, transformed it into a meaningful numerical format using TF-IDF, and used cosine similarity to match user needs with relevant, helpful content.

This project is a stepping stone. You can expand it by adding more resources, experimenting with different text preprocessing techniques, or exploring the alternative approaches mentioned above. The skills you've learned here are foundational to many applications in NLP and machine learning.

• Official Documentation:
  • Scikit-learn TfidfVectorizer documentation
  • Scikit-learn cosine_similarity documentation
• Further Reading:
  • TF-IDF Explained