Detecting Poor Sleep: Anomaly Detection in Health Data with Python & Scikit-learn

Ever woken up feeling tired despite getting a full night's sleep? The key to understanding why might be hidden in the data from your sleep tracker. Wearable devices provide a wealth of information, like heart rate and movement, that can reveal insights into your sleep quality. In this tutorial, we'll explore how to use Python and the powerful Scikit-learn library to perform anomaly detection on sleep data. By identifying unusual patterns, we can uncover nights of poor sleep and start to understand the factors that might be affecting our rest.

We'll be working with a real-world, open-source dataset of sleep information to make this a practical, hands-on experience. This is a great project for anyone interested in data science, machine learning, and health tech. Whether you're a seasoned developer or just starting, you'll walk away with a functional understanding of how to analyze time-series data and a cool project for your portfolio.

Prerequisites:

• Basic understanding of Python and pandas.
• Familiarity with machine learning concepts.
• Python 3.x, pandas, Scikit-learn, and Matplotlib installed.

Sleep is a complex physiological process, and what constitutes a "good" night's sleep can vary. However, significant deviations from your normal sleep patterns can be a sign of underlying issues. These deviations, or anomalies, could be sudden spikes in heart rate, excessive movement, or unusual sleep stage transitions.

Detecting these anomalies can be challenging due to the nature of time-series data. Sleep data is sequential, and the order of events matters. We need a method that can identify "the odd one out" in the context of your typical sleep patterns. This is where unsupervised anomaly detection algorithms, like Isolation Forest, come in.

Before we dive in, make sure you have the necessary libraries installed. You can do this with a single pip command:

pip install pandas scikit-learn matplotlib

We'll be using an open-source dataset from PhysioNet that contains sleep data from an Apple Watch, including heart rate and motion. For the purpose of this tutorial, we'll use a simplified, pre-processed version of this data.

First, let's load our sleep data and prepare it for analysis. Real-world data is often messy, so data cleaning is a crucial first step.

What we're doing

We will load the sleep data from a CSV file into a pandas DataFrame, inspect it for any missing values, and convert the timestamp column to a datetime object for easier manipulation.

Implementation

# src/data_preprocessing.py
import pandas as pd

# Load the dataset
try:
    df = pd.read_csv('sleep_data.csv')
except FileNotFoundError:
    print("Error: 'sleep_data.csv' not found. Please ensure the dataset is in the correct directory.")
    exit()


# Display the first few rows
print("Original Data:")
print(df.head())

# --- Data Cleaning ---

# Check for missing values
print("\nMissing values before cleaning:")
print(df.isnull().sum())

# Handle missing values (e.g., forward fill)
df.ffill(inplace=True)

# Convert timestamp to datetime objects
df['timestamp'] = pd.to_datetime(df['timestamp'])

# Set timestamp as the index
df.set_index('timestamp', inplace=True)

print("\nCleaned Data:")
print(df.head())
print(f"\nData shape after cleaning: {df.shape}")

How it works

We start by importing the pandas library, a go-to for data manipulation in Python. We load our sleep_data.csv file into a DataFrame. Then, we check for any missing values, which could disrupt our analysis. A simple way to handle this is using ffill(), which propagates the last valid observation forward. Finally, we convert the timestamp column to a proper datetime object and set it as the index of our DataFrame, which is essential for time-series analysis.

Common pitfalls

• Incorrect file path: Ensure your CSV file is in the same directory as your script or provide the correct path.
• Various data issues: Be prepared for other data quality problems like duplicates or incorrect data types, which might require additional cleaning steps.

Now that our data is clean, we need to create features that will help our anomaly detection model identify unusual patterns. This process is called feature engineering and is a critical part of any machine learning project.

What we're doing

We'll create new features from our existing data, such as rolling averages and standard deviations of heart rate and movement. These will help the model understand the context of each data point.

Implementation

# src/feature_engineering.py
import pandas as pd

# (Assuming df is our cleaned DataFrame from the previous step)
# For a standalone script, you would reload and clean the data first.

# --- Feature Engineering ---

# Calculate rolling statistics
window_size = 10 # 10-minute window
df['heart_rate_rolling_mean'] = df['heart_rate'].rolling(window=window_size).mean()
df['heart_rate_rolling_std'] = df['heart_rate'].rolling(window=window_size).std()
df['movement_rolling_mean'] = df['movement'].rolling(window=window_size).mean()

# Drop rows with NaN values created by rolling window
df.dropna(inplace=True)

print("\nData with new features:")
print(df.head())

How it works

We're creating new features that capture the recent trend and volatility of the heart rate and movement data. A rolling mean provides a smoothed version of the time series, while the standard deviation gives us a sense of how much the data is fluctuating. A 10-minute window is a good starting point, but you can experiment with different sizes. After creating these new features, we drop any rows with NaN values, which are a result of the initial rolling window calculations.

Common pitfalls

• Choosing the right window size: The window size can significantly impact your results. A small window might be too sensitive to noise, while a large window might smooth out important patterns. It's often a good idea to experiment with different window sizes.

With our features ready, it's time to apply the Isolation Forest algorithm. This is an unsupervised learning algorithm that's particularly effective for anomaly detection.

What we're doing

We'll train an Isolation Forest model on our engineered features. The model will then predict which data points are anomalies.

Implementation

# src/anomaly_detection.py
from sklearn.ensemble import IsolationForest
import numpy as np

# (Assuming df is our DataFrame with engineered features)

# --- Anomaly Detection ---

# Select features for the model
features = ['heart_rate', 'movement', 'heart_rate_rolling_mean', 'heart_rate_rolling_std', 'movement_rolling_mean']
X = df[features]

# Initialize and fit the Isolation Forest model
model = IsolationForest(n_estimators=100, contamination='auto', random_state=42)
model.fit(X)

# Predict anomalies (-1 for anomalies, 1 for inliers)
df['anomaly'] = model.predict(X)

# Get anomaly scores
df['anomaly_score'] = model.decision_function(X)

print("\nData with anomaly predictions:")
print(df.head())

# Display the anomalies
anomalies = df[df['anomaly'] == -1]
print("\nDetected Anomalies:")
print(anomalies)

How it works

Isolation Forest works by randomly selecting a feature and then randomly selecting a split value for that feature. The idea is that anomalies are "few and different," so they are easier to "isolate" from the rest of the data. The contamination parameter tells the model the expected proportion of outliers in the dataset. By setting it to 'auto', we let the algorithm decide the threshold. The model's predict() method returns -1 for anomalies and 1 for normal data points.

Common pitfalls

• Choosing the contamination value: This can be tricky. If you have some domain knowledge about how often anomalies occur, you can set it to a specific value. Otherwise, 'auto' is a reasonable choice.

Now that we've identified the anomalies, let's visualize them to get a better understanding of what's happening on those nights of poor sleep.

What we're doing

We'll use Matplotlib to create a plot that shows the heart rate over time, with the detected anomalies highlighted.

Implementation

# src/visualization.py
import matplotlib.pyplot as plt
import seaborn as sns

# (Assuming df is our DataFrame with anomaly predictions)

# --- Visualization ---

plt.style.use('seaborn-v0_8-whitegrid')
fig, ax = plt.subplots(figsize=(15, 6))

# Plot normal heart rate data
ax.plot(df.index, df['heart_rate'], color='blue', label='Normal Heart Rate')

# Highlight anomalies
ax.scatter(anomalies.index, anomalies['heart_rate'], color='red', label='Anomaly', s=50)

ax.set_title('Sleep Heart Rate with Anomaly Detection')
ax.set_xlabel('Time')
ax.set_ylabel('Heart Rate (bpm)')
ax.legend()
plt.show()

How it works

This code generates a time-series plot of the heart rate. The normal data points are shown as a blue line, and the anomalies we detected are highlighted as red dots. This visualization makes it easy to see when the unusual heart rate events occurred during the night.

In this tutorial, we've walked through the process of detecting anomalies in sleep data using Python and Scikit-learn. We started with raw data, cleaned it, engineered new features, and applied the Isolation Forest algorithm to identify unusual patterns. By visualizing the results, we can clearly see the moments during sleep that might be indicative of poor rest.

This is just the beginning of what you can do with sleep data. You could explore other anomaly detection algorithms, incorporate more data sources (like sleep stages), or even build a dashboard to track your sleep quality over time.

• Scikit-learn's Isolation Forest Documentation: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html
• PhysioNet - Apple Watch Sleep Data: https://physionet.org/content/sleep-accel/1.0.0/
• A deep dive into the Isolation Forest algorithm: https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf