Stock Correlation, Clusters and Pairs¶

Live-basket correlation heatmap, rolling correlation vs Nifty, and why a tight pair is not a hedge — the useful part of the Kaggle market-analysis notebooks.

Part 32 of 36 in the ServLoci algo/options trading notebook series — full index in notebooks/README.md.

Setup — no broker account needed¶

!pip install -q yfinance

import yfinance as yf
import pandas as pd
import numpy as np

# No broker account, no API key, no ServLoci setup needed for this chapter —
# yfinance reads public end-of-day data from Yahoo Finance. Indian tickers take
# an ".NS" suffix (NSE) or ".BO" (BSE); index tickers are prefixed with "^"
# (^NSEI = Nifty 50, ^BSESN = Sensex, ^GSPC = S&P 500).
NSE_TICKER = "RELIANCE.NS"
US_TICKER = "AAPL"
INDEX_TICKER = "^NSEI"

print("yfinance", yf.__version__)

Correlation is a fact about this window, not a law¶

Kaggle stock-analysis notebooks almost all draw the same picture: download a handful of names, plot a seaborn heatmap of return correlations, and declare that "these stocks move together." That picture is useful — and routinely over-read.

This chapter does the useful part on a live NSE + US basket, then the part those notebooks skip: rolling correlation (it is not stable), a pair that looks "hedgeable" until a crash week, and why a 0.85 correlation is not two independent bets.

Sources this chapter is in conversation with (read them, don't paste them):

Stock Market Analysis Using Python — AAPL/GOOG/AMZN/MSFT tour, daily returns, heatmap.
Stocks Analysis by Regression — scatter matrix + correlation as a prelude to regression.
Yahoo Finance: Optimized Portfolio & CAPM — heatmap then jumps to CAPM / weights (notebook 34).
Stock Performance with Yahoo Finance — long-history yfinance pull and relative performance.

None of those is a pair-trading system. Neither is this.

!pip install -q matplotlib
import matplotlib.pyplot as plt

BASKET = {
    "RELIANCE.NS": "Reliance", "TCS.NS": "TCS", "HDFCBANK.NS": "HDFC Bank",
    "INFY.NS": "Infosys", "ICICIBANK.NS": "ICICI Bank", "BHARTIARTL.NS": "Airtel",
    "SBIN.NS": "SBI", "ITC.NS": "ITC", "LT.NS": "L&T", "^NSEI": "Nifty 50",
    "AAPL": "Apple", "MSFT": "Microsoft",
}

raw = yf.download(
    list(BASKET), period="3y", interval="1d",
    auto_adjust=True, progress=False, group_by="ticker", threads=True,
)

def close_of(symbol):
    if isinstance(raw.columns, pd.MultiIndex):
        if symbol not in raw.columns.get_level_values(0):
            return pd.Series(dtype=float, name=symbol)
        frame = raw[symbol]
        col = "Close" if "Close" in frame.columns else frame.columns[0]
        return frame[col].rename(symbol)
    return raw[symbol].rename(symbol) if symbol in raw.columns else raw["Close"].rename(symbol)

prices = pd.concat([close_of(s) for s in BASKET], axis=1).dropna(how="all")
prices.columns = [BASKET[c] for c in prices.columns]
returns = prices.pct_change(fill_method=None).dropna(how="any")
print("aligned sessions:", len(returns), returns.index.min().date(), "->", returns.index.max().date())
print("names:", list(returns.columns))
returns.tail(3)

Full-sample correlation heatmap¶

This is the chart every Kaggle market notebook leads with. Read it as "how much of the daily move was shared over this whole 3-year window," not "these two names are the same bet tomorrow." Banks will cluster. Infosys and TCS will cluster. Apple and Microsoft will cluster. ITC often looks like the diversifier — until it doesn't.

corr = returns.corr()
fig, ax = plt.subplots(figsize=(8.5, 7))
im = ax.imshow(corr.values, cmap="coolwarm", vmin=-1, vmax=1)
ax.set_xticks(range(len(corr.columns)))
ax.set_yticks(range(len(corr.columns)))
ax.set_xticklabels(corr.columns, rotation=45, ha="right", fontsize=8)
ax.set_yticklabels(corr.columns, fontsize=8)
for i in range(len(corr)):
    for j in range(len(corr)):
        ax.text(j, i, f"{corr.values[i, j]:.2f}", ha="center", va="center", fontsize=7,
                color="white" if abs(corr.values[i, j]) > 0.65 else "#1a1a1a")
fig.colorbar(im, ax=ax, fraction=0.046)
ax.set_title("Daily-return correlation, full sample")
fig.tight_layout()
plt.show()
print("highest off-diagonal pairs:")
pairs = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool)).stack().sort_values(ascending=False)
print(pairs.head(8).to_string())
print("\nlowest pairs (closest to diversifiers):")
print(pairs.tail(5).to_string())

Rolling correlation — the heatmap is a freeze-frame¶

A 60-session rolling correlation of two "tight" names against the index usually drifts. Crisis weeks it jumps toward 1: everything falls together, which is exactly when you wanted the diversifier. That is why a portfolio built on a single full-sample matrix (notebook 34) looks safer than it is.

window = 60
nifty = returns["Nifty 50"] if "Nifty 50" in returns.columns else returns.iloc[:, 0]
roll = pd.DataFrame({
    name: returns[name].rolling(window).corr(nifty)
    for name in returns.columns if name != nifty.name
})
fig, ax = plt.subplots(figsize=(11, 4))
for col in roll.columns:
    ax.plot(roll.index, roll[col], lw=1, label=col)
ax.axhline(0, color="#999", lw=0.6)
ax.set_title(f"{window}-session rolling correlation vs {nifty.name}")
ax.legend(loc="upper left", fontsize=7, ncol=3, frameon=False)
ax.set_ylim(-0.2, 1.05)
ax.grid(True, alpha=0.25)
fig.tight_layout()
plt.show()
print(roll.tail(1).T.rename(columns={roll.index[-1]: "latest_vs_index"}))

A pair is not a hedge¶

Pick the tightest NSE pair from the matrix. Their spread of log prices looks mean-reverting on a quiet chart and then steps when one name has a stock-specific week (results, a block deal, a sector news item the other name does not share). The Kaggle move is to regress one on the other and call the residual a "spread." Residual ≠ tradable hedge: beta is estimated in-sample, the legs have different borrow/impact, and the relationship breaks on the week you need it.

# Tightest pair that is not the index and not the same listing twice.
pair_names = [a for a, b in pairs.index if a != "Nifty 50" and b != "Nifty 50"]
left, right = pairs.index[0]
# pairs.index entries are (row, col) labels; prefer two Indian names if present.
for a, b in pairs.index:
    if a != "Nifty 50" and b != "Nifty 50" and a != b:
        left, right = a, b
        break
print(f"inspecting pair: {left} vs {right}  (full-sample corr {corr.loc[left, right]:.2f})")
spread = np.log(prices[left]) - np.log(prices[right])
z = (spread - spread.rolling(60).mean()) / spread.rolling(60).std()
fig, axes = plt.subplots(2, 1, figsize=(11, 5), sharex=True)
axes[0].plot(prices.index, prices[left] / prices[left].iloc[0], label=left)
axes[0].plot(prices.index, prices[right] / prices[right].iloc[0], label=right)
axes[0].legend(frameon=False)
axes[0].set_title("Rebased prices (start = 1)")
axes[1].plot(z.index, z, color="#0f766e", lw=1)
axes[1].axhline(2, color="#b45309", ls="--", lw=0.8)
axes[1].axhline(-2, color="#b45309", ls="--", lw=0.8)
axes[1].set_title("60-session z-score of log spread — looks tradable, is not a system")
axes[1].grid(True, alpha=0.25)
fig.tight_layout()
plt.show()
print("z-score now:", float(z.iloc[-1]) if pd.notna(z.iloc[-1]) else "warming up")

What to take into notebook 34¶

Names that cluster on the heatmap are overlapping bets. Equal-weighting five bank stocks is not five-name diversification.
Rolling correlation vs the index is the honest "how much systematic risk is this name carrying right now."
A z-score on a spread is a research plot. Turning it into a pair trade needs borrow, beta stability, and a stop for when the residual is a company event, not noise.

Next: notebook 33 — the other Kaggle genre, "I predicted the stock with LSTM," rewritten so the naive baseline has to lose first.

« Previous: Capstone: Build Your Own Strategy End to End
Next: Return Prediction Baselines (Beat Naive First) »

Try the concepts above interactively: Options Strategy Builder · Docs · Get your static IP