Day 6 · Vector anisotropy and collapse (No.5, No.6)

[onestardao]

Day 6 · Vector anisotropy and collapse (No.5, No.6)

Symptom
embeddings bunch into a skinny cone. neighbors look the same for every query. recall tanks even with a decent retriever.

ProblemMap

• No.5 Semantic ≠ Embedding
• No.6 Logic Collapse & Recovery
  Reference map: WFGY ProblemMap

---

What this usually looks like

• cosine sims sit high for almost everything, top-k lists barely change across queries
• nearest neighbors are dominated by global terms or boilerplate
• recall@k on a heldout set falls off a cliff after a re-ingest or a model swap
• FAISS IVF/PQ shows busy lists with poor re-rank separation, HNSW shows shallow fronts

---

60-sec quick test

1. sample 5k vectors from your store
2. compute per-dimension variance and the explained variance ratio from PCA(50)
3. plot the first 2 PCs and check cone shape

Rules of thumb

• red flag if the first PC explains > 0.70 of variance, or if var(dim) has Gini > 0.6
• also bad if cosine(self, centroid) median > 0.55 across the sample

---

Diagnose checklist

• embedding model change without re-whitening
• mixed normalization states (some L2, some not)
• truncation or unicode normalization bugs during text → embedding
• over-aggressive stopword removal, or duplicate boilerplate in documents
• FAISS metric set to IP while vectors were already L2-normalized for cosine
• shard trained with one domain, then queried with another

---

Minimal fix

Goal keep geometry isotropic enough for cosine to mean something.

1. Mean-center + L2 normalize all vectors again

2. Whiten small-rank

   • fit PCA on a random 50k subset, keep r so that cumulative EVR in [0.90, 0.98]
   • transform, then L2 normalize again

3. Rebuild the index with a metric that matches the vector state

   • cosine → use L2 with normalized vectors
   • inner-product → use IP and do not renormalize twice

4. Trash and re-ingest any mixed-state shards. no partial patches.

If time is tight, at least run steps 1, 3, 4. You will usually see recall recover immediately.

---

Hard fixes when minimal is not enough

• Domain de-dup and boilerplate mask before embedding
  keep per-doc tf-idf masks or a learned salience to damp headers, nav, legalese
• Subspace drop
  if PC1..PCk are topic or style axes, drop k where EVR spikes, renorm
• Temperature upsampling for rare intents
  balance the corpus so neighbors reflect intent, not frequency
• Metric sanity
  cosine with normalized vectors is usually safer than IP on mixed magnitudes
• FAISS index hygiene
  re-train IVF/PQ codebooks after geometry change. do not reuse old centroids

---

WFGY guardrails you can enable

• BBMC residual check to spot geometry drift and flag re-whiten when ‖B‖ grows
• BBPF multi-path retrieval to avoid single-cone collapse during query expansion
• BBCR bridge step when the chain stalls on near-duplicates
• BBAM attention damping to stop one-token hijacks in long answers

If you are replying in forums, label it as No.5 or No.6 and suggest the 60-sec repro first.
Then point to a minimal fix. Avoid touching their infra beyond re-embed + rebuild.

---

Tiny scripts you can paste

variance + cone check

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import normalize

X = np.load("sample_embeddings.npy")        # shape [N, d]
X = X - X.mean(axis=0, keepdims=True)
X = normalize(X, norm="l2", axis=1)

p = PCA(n_components=min(50, X.shape[1])).fit(X)
evr = p.explained_variance_ratio_
print("PC1 EVR:", evr[0], "PC1-5 cum:", evr[:5].sum())

centroid = X.mean(axis=0, keepdims=True)
cos = (X @ centroid.T).ravel()
print("median cos to centroid:", float(np.median(cos)))

whiten + renorm + save

from sklearn.decomposition import PCA
from sklearn.preprocessing import normalize
import numpy as np, joblib

X = np.load("all_embeddings.npy")
mu = X.mean(0, keepdims=True)
Xc = X - mu
p = PCA(n_components=0.95, svd_solver="full").fit(Xc)
Z = p.transform(Xc)
Z = normalize(Z, norm="l2", axis=1)

joblib.dump({"mu": mu, "pca": p}, "whitener.pkl")
np.save("embeddings_whitened.npy", Z)

FAISS rebuild sketch

import faiss, numpy as np

Z = np.load("embeddings_whitened.npy").astype("float32")  # L2-normalized
d = Z.shape[1]
index = faiss.IndexHNSWFlat(d, 32)                        # cosine via L2
index.hnsw.efConstruction = 200
faiss.normalize_L2(Z)
index.add(Z)
faiss.write_index(index, "hnsw_cosine.faiss")

---

Acceptance checks before you call it fixed

• PC1 EVR ≤ 0.35 and cum EVR for 1..5 ≤ 0.70
• median cos to centroid ≤ 0.35 after renorm
• neighbor overlap rate across 20 random queries ≤ 0.35 at k=20
• recall@k on a heldout set improves, and the top-k list varies with the query

---

Related docs

• Problem map index: WFGY ProblemMap
• Vector store pitfalls and metrics: vectorstore-metrics-and-faiss-pitfalls.md

https://github.com/onestardao/WFGY/blob/main/ProblemMap/article/README.md

---

TL;DR

cone geometry means your space collapsed. re-center, whiten, renorm, rebuild. then re-check PC1 EVR and neighbor overlap. if the chain still stalls, tag it No.6 and enable a bridge step with BBCR.

[cj-elevate]

This is an unbelievable wealth of resources. I feel like I tripped and fell into a literal gold mine. This information will serve me well in my efforts and I cannot thank you enough for taking the time to compile it into one location. I want you to know, I see what you did here and I thank you for it. Your work is appreciated.

[onestardao]

You are really welcome, and you can share our repo to your friends. ^_______________^ I will wirte it until Day 16

[onestardao]

also remind you we will have Global Fix Map very soon , every platform, every framework, you can find it 100% in my new MAP (will be launched around Sep 1)